Winslow Burleson, Rosalind W. Picard, K. Perlin

Abstract

Accurately interpreting and expressing affect is fundamental to empathetic relationships. A platform for sensing and interpreting several aspects of users’ nonverbal affective information and responding through an expressive agent has been developed. The platform includes integration of multi-modal affective sensors with a real time inference engine, a behavior engine, and a 3d scriptable expressive humanoid agent within a graphical virtual environment. Currently the sensors include a pressure-sensitive mouse, a BlueTooth wireless skin conductivity sensor, a TekScan pressure sensor on a chair, and a stereo head tracking system as well as an IBM Blue Eyes infrared-sensitive camera. These sensors feed into custom algorithms for analysis of individual channels of information, such as postural and facial expressions, which in turn are combined with additional channels of information to make an inference about the user’s affective state. The system further synchronizes this sensor data with the agent behaviors and with video of the user and his or her on-screen activity. This platform is seen as a generalpurpose tool applicable to research in several areas, including how to design an affective learning companion, and how to further basic understanding of empathy and emotion contagion in human-agent interaction.