This project examines how the expression granted by new musical interfaces can be harnessed to create positive changes in health and wellbeing. We are conducting experiments to measure EEG dynamics and physical movements performed by participants who are using software designed to invite physical and musical expression of the basic emotions. The present demonstration of this system incorporates an expressive gesture sonification system using a Leap Motion device, paired with an ambient music engine controlled by EEG-based affective indices. Our intention is to better understand affective engagement, by creating both a new musical interface to invite it, and a method to measure and monitor it. We are exploring the use of this device and protocol in therapeutic settings in which mood recognition and regulation are a primary goal.

Technologies to automatically recognize stress are extremely important to prevent chronic psychological stress and pathophysiological risks associated with it. The introduction of comfortable and wearable biosensors has created new opportunities to measure stress in real-life environments, but there is often great variability in how people experience stress and how they express it physiologically. In this project, we modify the loss function of Support Vector Machines to encode a person's tendency to feel more or less stressed, and give more importance to the training samples of the most similar subjects. These changes are validated in a case study where skin conductance was monitored in nine call center employees during one week of their regular work. Employees working in this type of setting usually handle high volumes of calls every day, and they frequently interact with angry and frustrated customers that lead to high stress levels.

Deep and slow breathing techniques are components of various relaxation methods and have been used in treatment of many psychiatric and somatic disorders, as well as to improve mental function and attentiveness in healthy individuals. Many adults spend more than eight hours a day in front of screens, suggesting a need for an on-screen system to guide the user toward healthier breathing habits without requiring interruption. In this project, we explore the design and implementation of unobtrusive systems to promote healthier breathing habits.

Electrodermal Activity (EDA) is a physiological indicator of stress and strong emotion. While an increasing number of wearable devices can collect EDA, analyzing the data to obtain reliable estimates of stress and emotion remains a difficult problem. We have built a graphical tool that allows anyone to upload their EDA data and analyze it. Using a highly accurate machine learning algorithm, we can automatically detect noise within the data. We can also detect skin conductance responses, which are spikes in the signal indicating a "fight or flight" response. Users can visualize these results and download files containing features calculated on the data to be used in their own analysis. Those interested in machine learning can also view and label their data to train a machine learning classifier. We are currently adding active learning, so the site can intelligently select the fewest possible samples for the user to label.

The wide availability of low-cost, wearable, biophysiological sensors enables us to measure how the environment and our experiences impact our physiology. This creates a new challenge: in order to interpret the collected longitudinal data, we require the matching contextual information as well. Collecting weeks, months, and years of continuous biophysiological data makes it unfeasible to rely solely on our memory for providing the contextual information. Many view maintaining journals as burdensome, which may result in low compliance levels and unusable data. We present an architecture and implementation of a system for the acquisition, processing, and visualization of biophysiological signals and contextual information.

Conversations between two individuals--whether between doctor and patient, mental health therapist and client, or between two people romantically involved with each other--are complex. Each participant contributes to the conversation using her or his own "lens." This project involves advanced probabilistic graphical models to statistically extract and model these dual lenses across large datasets of real-world conversations, with applications that can improve crisis and psychotherapy counseling and patient-cardiologist consultations. We're working with top psychologists, cardiologists, and crisis counseling centers in the United States.

Physiological arousal is an important part of occupational therapy for children with autism and ADHD, but therapists do not have a way to objectively measure how therapy affects arousal. We hypothesize that when children participate in guided activities within an occupational therapy setting, informative changes in electrodermal activity (EDA) can be detected using iCalm. iCalm is a small, wireless sensor that measures EDA and motion, worn on the wrist or above the ankle. Statistical analysis describing how equipment affects EDA was inconclusive, suggesting that many factors play a role in how a child's EDA changes. Case studies provided examples of how occupational therapy affected children's EDA. This is the first study of the effects of occupational therapy's in situ activities using continuous physiologic measures. The results suggest that careful case study analyses of the relation between therapeutic activities and physiological arousal may inform clinical practice.

We are conducting EEG studies to identify the musical features and musical interaction patterns that universally impact measures of arousal. We hypothesize that we can induce states of high and low arousal using electrodermal activity (EDA) biofeedback, and that these states will produce correlated differences in concurrently recorded skin conductance and EEG data, establishing a connection between peripherally recorded physiological arousal and cortical arousal as revealed in EEG. We also hypothesize that manipulation of musical features of a computer-generated musical stimulus track will produce changes in peripheral and cortical arousal. These musical stimuli and programmed interactions may be incorporated into music technology therapy, designed to reduce arousal or increase learning capability by increasing attention. We aim to provide a framework for the neural basis of emotion-cognition integration of learning that may shed light on education and possible applications to improve learning by emotion regulation.

Panoply is a crowdsourcing application for mental health and emotional wellbeing. The platform offers a novel approach to computer-based psychotherapy, targeting accessibility without stigma, engagement, and therapeutic efficacy. A three-week randomized-controlled trial with 166 participants showed Panoply conferred greater or equal benefits for nearly every therapeutic outcome measure compared to an active control task (online expressive writing). Panoply significantly outperformed the control task also on all measures of engagement, and is now being commercialized at itskoko.com.

Depression correlated with anxiety is one of the key factors leading to suicidal behavior, and is among the leading causes of death worldwide. Despite the scope and seriousness of suicidal thoughts and behaviors, we know surprisingly little about what suicidal thoughts look like in nature (e.g., How frequent, intense, and persistent are they among those who have them? What cognitive, affective/physiological, behavioral, and social factors trigger their occurrence?). The reason for this lack of information is that historically researchers have used retrospective self-report to measure suicidal thoughts, and have lacked the tools to measure them as they naturally occur. In this work we explore use of wearable devices and smartphones to identify behavioral, affective, and physiological predictors of suicidal thoughts and behaviors.

The SNAPSHOT study seeks to measure Sleep, Networks, Affect, Performance, Stress, and Health using Objective Techniques. It is an NIH-funded collaborative research project between the Affective Computing and Macro Connections groups, and Harvard Medical School's Brigham & Women's hospital. Since fall 2013, we've run this study to collect one month of data every semester from 50 MIT undergraduate students who are socially connected. We have collected data from about 170 participants, totaling over 5,000 days of data. We measure physiological, behavioral, environmental, and social data using mobile phones, wearable sensors, surveys, and lab studies. We investigate how daily behaviors and social connectivity influence sleep behaviors and health, and outcomes such as mood, stress, and academic performance. Using this multimodal data, we are developing models to predict onsets of sadness and stress. This study will provide insights into behavioral choices for wellbeing and performance.

Stories, language, and art are at the heart StoryScape. While StoryScape began as a tool to meet the challenging language learning needs of children diagnosed with autism, it has become much more. StoryScape was created to be the first truly open and customizable platform for creating animated, interactive storybooks that can interact with the physical world. Download the android app: https://play.google.com/store/apps/details?id=edu.mit.media.storyscape and make your own amazing stories at https://storyscape.io/.

Individuals with autism are known to have difficulties connecting with other people, reciprocating social interactions, and being emotionally regulated by others. Yet, until recently, very little attention has been given to the way people interact together, in a system, rather than by themselves. We propose a new way to collect data on how caregivers and their children, with and without autism, affect and are affected by each other (i.e., how they "sync up" with one another), both in their behavior and in their physiology. We also introduce a customizable digital-physical smart toy platform that will allow us to test hypotheses and collect data about patterns of caregiver-child synchrony in a naturalistic and engaging environment. MIT and Northeastern are forging a new collaboration between smart toy technology and autism research that will help uncover how the social brain develops.

The proliferation of smartphones and wearable sensors is creating very large data sets that may contain useful information. However, the magnitude of generated data creates new challenges as well. Processing and analyzing these large data sets in an efficient manner requires computational tools. Many of the traditional analytics tools are not optimized for dealing with large datasets. Tributary is a parallel engine for searching and analyzing sensor data. The system utilizes large clusters of commodity machines to enable in-memory processing of sensor time-series signals, making it possible to search through billions of samples in seconds. Users can access a rich library of statistics and digital signal processing functions or write their own in a variety of languages.

We are developing statistical tools for understanding, modeling, and predicting self-harm by using advanced probabilistic graphical models and fail-soft machine learning in collaboration with Harvard University and Microsoft Research.