I. Essa, A. Pentland

Abstract

Previous efforts at facial expression recognition have been based on the Facial Action Coding System (FACS), a representation developed in order to allow human psychologists to code expression from static facial ``mugshots.'' In this paper we develop new, more accurate representations for facial expression by building a video database of facial expressions and then probabilistically characterizing the facial muscle activation associated with each expression using a detailed physical model of the skin and muscles. This produces a muscle-based representation of facial motion, which is then used to recognize facial expressions in two different ways. The first method uses the physics-based model directly, by recognizing expressions through comparison of estimated muscle activations. The second method uses the physics-based model to generate spatio-temporal motion-energy templates of the whole face for each different expression. These simple, biologically-plausible motion energy ``templates'' are then used for recognition. Both methods show substantially greater accuracy at expression recognition than has been previously achieved.