Deb Roy

Twitter provides a space for both social and broadcast communication; a tweet by a celebrity can reach millions of users, some of whom may …

This project is aimed at building a machine learning pipeline that will discover and predict links between the visible structure of village…

Over the last two decades, digital technologies have flattened old hierarchies in the news business and opened the conversation to a multit…

In this project we examine the social media and traditional media's response to the Boston Marathon bombings from the moment of the explosi…

The Human Speechome Project is an effort to observe and computationally model the longitudinal language development of a single child at an…

Collection and analysis of longitudinal observational data of child behavior in natural, ecologically valid, non-laboratory settings holds …

BlitzScribe is a new approach to speech transcription driven by the demands of today's massive multimedia corpora. High-quality annotations…

HouseFly combines audio-video recordings from multiple cameras and microphones to generate an interactive, 3D reconstruction of recorded ev…

Language is inextricably linked to the activities and events that make up our daily lives. For a child learning language, everyday activiti…

This project presents a multimodal speech recognition system for real world scene description tasks. Given a visual scene, the system dynam…

We are adapting the video data collection and analysis technology derived from the Human Speechome Project for the retail sector through re…

Varied important problems can be solved using surprisingly approximate arithmetic. We've designed a co-processor for such arithmetic that …

People are surprisingly predictable. We use real-time video analysis to extract patterns of behavior from crowds browsing demos in our lab …

What is the relationship between the input children hear and the words they acquire? We are investigating the role of variables such as inp…

This project attempts to address the practical problems involved with extracting behavioral information from large, multi-camera video corp…

Understanding spatial language is a challenging problem that requires the ability to map between language and real-world situations. We are…

Concrete Financial Sim aims to anticipate probable outcomes of different decisions across time. Life consistently presents choices that req…

The Smart Cities and Cognitive Machines groups have teamed up with the Center for Future Banking to design a concept banking store in the B…

HeadLock is a semi-automated system for head pose annotation that explores how human-computer interfaces can be combined with computer visi…

Trisk is a humanoid robot that integrates speech input, visual perception, and active touch in order to interact with humans and its enviro…

The quantity and availability of video content is soaring due to the combination of television networks and the Internet. The aim of this p…

This project focuses on new methods for analyzing, visualizing, and interacting with large text and speech collections. We are exploring tr…

We are developing knowledge representations and natural language processing algorithms that connect spatial language to physical environmen…

In this project we explore forms of representing that emphasize notions such as discovery, embodiment, and fluidity. We are currently work…

About a decade ago, University of Edinburgh and its partners put together a collection of transcribed human/human dialogues in which both p…

We approach the problem of how machines, and humans, can learn words that describe actions. We put forth that such words are grounded not i…

An intelligent robot must build an understanding of its environment with minimal human mediation. We investigate the characterization of ob…

We are developing a spoken-language interface for interacting with mobile robots. In our approach, language understanding is treated as a p…

Imagine the full power of a theater lighting system, complete with an expert human lighting controller, at your disposal to light your home…

Over two million speech-disabled people in the US alone rely on Augmentative and Alternative Communication (AAC) devices to help express th…

In this project we demonstrate an application of our work in understanding natural language about spatial scenes. All 3-D modeling applicat…

The automated processing of natural language by computer has become a paramount concern in nearly every field of human endeavor. The amount…

We are developing a multimodal processing system called Fuse to explore the effects of visual context on the performance of speech recognit…

The Bishop project explores the subtleties of human language when talking about spatial scenes. In particular, we investigate the various d…

How would you translate the word "samosa" for someone who has never experienced this food item? One way would be to map the word onto names…

The meanings of simple verbs such as "pick up" convey more information than meets the eye. In order to pick up an object, a person much rea…

We are interested in building systems which can learn to connect natural language to objects and actions in video. In our current experimen…

Language is grounded in experience. Unlike dictionaries which define words in terms of other words, humans understand many basic words in t…