This Guy Recorded His Child Constantly for a Year to Understand Word Learning (and Twitter)

Some parents keep track of their kids’ growth by making pencil marks on a wall.

But If you’re a researcher at the MIT Media Lab, you capture and analyze 200 gigabytes of toddler data daily using high-tech recording devices, computer synthesis, 3-D visualizations and a staff of researchers. Then you show it all off in a TED talk.

For over a year, Deb Roy, former director of the Media Lab’s Cognitive Machines Group, turned his apartment in Cambridge into a miniature version of “The Truman Show,” cast his toddler as the Jim Carrey character, and recorded everything in order to better understand early childhood language learning.

It wasn’t a walk in the playground: to process 16 months of video from 14 cameras and microphones (that’s about 90,000 hours of footage total), he couldn’t rely on existing software or research lackeys alone.

So with fifty thousand dollars in seed funding from the National Science Foundation and the open checkbook of the Media Lab, Roy developed a software suite called Blitzscribe that was able to find speech in the recordings and break it down into easily transcribed sound bites, making it easier for humans to process the data. The software can also take into account how a word was said – its “prosody” – and who said it. What Dad could do in ten hours, computer plus researcher could process in two. (Note to self: ask interns to download this app or whatever it is now. – Ed.)

The technology — which Roy demonstrated in a presentation I saw last week at the TED conference (yet to be placed online) — have proven fascinating for developmental linguists. “Current samples that the field works with – typically an hour of recorded speech a week – are one to two orders of magnitude too small for our scientific purposes,” Professor Steven Pinker of Harvard University told BBC News in 2009. (The project, dubbed The Speechome Project, ended back in 2008.)

Next, by mapping the language to the footage throughout the house, Roy and his team were able to to create fly-throughs and “time worms” (remember Donnie Darko?) that illustrated how certain words were uttered where. “Ball” would happen most near the door; caretakers would hear “water” most clearly in the kitchen.

Another remarkable finding: caregivers’ speech adjusts in order to usher a child’s new words into their vocabulary. That is, adults’ sentences shorten and dwell on a word that a child is learning, but lengthen and become more complex as a child becomes more comfortable with that word. The language of teacher and student dance around and influence each other.

While the ultimate goal at the start of the project, back in 2006, was to assist robots in their own language acquisition, Roy is now using his research for something completely different: marketing, natch.

His startup Bluefin Labs applies high-tech analytics not to childhood or robot learning, but to process the way people discuss televised events on social media like Twitter, helping to comprehend how people "engage" with shows or car commercials.

Now kids: can you say “no money down”?

Thanks to

Fast Company

for some video.