Location

MIT Media Lab, E14-633

Description

This thesis introduces and examines methods for the capture and reproduction of music on the piano that maintains a tight coupling between the body and the physical instrument. Music playing originates in the body, which plays a central role in both performance and pedagogy. However, technology for the capture and reproduction of music has largely ignored the intricate relationship between the sound, the body and the instrument in the making and appreciation of music.
Drawing from research in telepresence, tangible interfaces, and augmented reality, Xiao proposes to bring the movement of the human body back into the picture by projection-mapping on a player piano, synchronizing image and sound with the piano’s moving keys. Xiao here focuses on two main projects: MirrorFugue and Andante. MirrorFugue is inspired by reflections on the surface of a lacquered grand piano and simulates the presence of a virtual pianist whose reflection is actually playing the physically moving keys. Andante depicts musical phrases as silhouettes of miniature animated figures that walk and dance across the piano keyboard. Drawing on human empathy and the importance of moving between sensory modalities and shifting standpoints, both installations are designed as immersive “sandboxes” for the playful exploration of musical ideas, embodied cognition, and the role of motion and emotion in making sense of our experience. Xiao discusses experiments and experiences of both projects for music learning and enjoyment.
Beyond these projects, this thesis will examine music as a source of insights on the mind, in order to derive strategies for better ways to learn. Inspired by Marvin Minsky’s concept of the Society of Mind, Xiao will use the process of learning music as a case study for how to build complex systems in the mind, demonstrating a way to think about individual learning by applying a “systems approach”. Xiao will also use examples from music to clarify different categories of embodied cognition, which Xiao will call “body-syntonic processes”, elaborating on Seymour Papert’s original concept of body-syntonicity. Better understanding of the interplay between different processes of the mind-body not only helps designers create better tools for music learning but also informs the design of learning technologies at large, and the design of richer interactive experiences with the computer.

Host/Chair: Hiroshi Ishii

Participant(s)/Committee

Ken PerlinEdith AckermannDonal Fox