MIThril Real-Time Context Engine

Richard W. DeVaul

MIThril, a borglab production. Richard W. DeVaul, Jonathan Gips, Michael Sung, Sandy Pentland

Anduin window manager

MIThril
hardware design

MIThril
project overview

The MIThril vision

MIThril diagrams

MIThril Enchantment Software

MIThril
Real-Time Contxt Engine

Wearables Home Page

The MIThril Real-Time Context Engine (formerly named the MIThril Inference Engine) is an light-weight modular architecture for the development and implementation of real-time context classifiers for wearable applications. We find that a systematic approach to activity classification based on modern machine learning techniques can greatly simplify the process of developing and implementing real-time activity classification models. The MIThril Real-Time Context Engine was developed to provide this systematic framework in a flexible, modular, open-source form. The MIThril Real-Time Context engine is a combination of methodology and supporting software tools. To train our models we use Kevin Murphy's Bayes Net Toolkit and Matlab. (We have a Gaussian Mixture Modeling Tutorial available online -- download the tutorial here in tar.gz format or zip format.). In addition to the tutorial, there is Real-Time Context Engine whitepaper explaining the real-time context engine and describing in some detail the development of a three-class accelerometer-based activity model. Real-time classification is implemented as Linux C/C++ code on top of the Enchantment Whiteboard and Signal systems -- see the inference section of the MIThril CVS repository One of the first projects implemented using the MIThril Real-Time Context engine was a five-state single-accelerometer based activity classification system. This page provides a simplified, non-technical overview of the real-time modeling and classification side of the MIThril Real-Time Context engine. For a detailed discussion of modern statistical classification techniques, see R.O. Duda, P.E. Hart, and D.G. Stork, Pattern Classification. Wiley-Interscience, 2nd edition, 2000. (Available from Amazon, among other sources.)
Architecture

The MIThril Real-Time Context Engine architecture is divided into four distinct processes or stages: Sensing Feature Extraction Modeling Inference The arrows in the diagram represent information flow, starting with the "World" and ending with the application. At each stage a distinct operation is performed that enables the next.
Sensing The user's world is complex. The only things we can know are what we measure thorough sensors or are explicitly told.	Measuring something in the "analog" world produces a sequence of discrete digital observations, or digital signal. The amount of information in the digital signal is a function of the complexity of the original analog process, the sensitivity and precision of the sensor, and the resolution and rate of the sampling process. For any particular modeling task, this digital signal contains some useful information and some extraneous information. For example, a three-axis accelerometer produces measurements that combine "true" acceleration, some systematic bias, and some random noise.
Feature Extraction In the feature extraction stage, a raw sensor signal is transformed into a feature signal more suitable for a particular modeling task. For our activity classifier we compute short-time-windowed power spectrum features	We can often simplify a modeling task by transforming raw observations into a form that is more appropriate for a particular modeling task. For example, features that use a pitch/energy representation might be more appropriate for a speaker-identification modeling task than raw audio samples.
Modeling In the modeling stage, a statistical model (such as a Gaussian mixture model, Hidden Markov Model, etc) is used to classify a feature signal in real-time.	For each context we want to model, a statistical model is created. These models allow us to answer the question "which state is the user in," or "how likely is state X?"
Inference In the inference stage, the results of the modeling stage, possibly combined with other information, are fed into a Bayesian inference system for complex interpretation and decision-making. The inference stage is only implemented under Matlab, at present.	At present, our applications use only the output of the modeling stage. However, we foresee the need for more complex applications in which the output of multiple modeling stages provides the input into a Bayesian inference system.
Actions

Action, or what to do with the output MIThril Real-Time Context Engine, is left up to the application. An important class of action is one that provokes a user interaction, which in turn effects the user's state and hence the "world."