|
The Memory Glasses is a wearable, proactive, context-aware memory aid
based on the MIThril platform and wearable sensors. The primary goal
of this project is to produce an effective memory aid and reminder
system that requires a minimum of the wearer's attention. Our
Memory Glasses paper that was published in ISWC 2003 describes
recent research on the use of subliminal visual cues for memory
support.
The function of our system is to deliver reminders to the wearer in a
timely, situation-appropriate way, without requiring intervention on
the part of the wearer beyond the initial request to be reminded. In
other words, the system behaves like a reliable human assistant that
remembers reminder requests and delivers them under appropriate
circumstances. Such a system is qualitatively different from a passive
reminder system (such as a paper organizer) or a context blind
reminder system (a modern PDA) which records and structures reminder
requests but which can not know the user's context.
|
The Problem
to be Addressed, or the Limitations of Context Blind Systems
|
The Problem
People's lives are increasingly complicated, and the need for
effective logistical and memory aids is on the rise. For years,
digital technology and computers have promised effective solutions for
the problems of organization and memory, but present technology offers
little more than an incremental improvement over the paper scheduler
and alarm clock of the 19th century. Even the most sophisticated PDA
is deaf and blind, and knows no more about the wearer's context than the
time of day.
|
The Memory Glasses Solution, a Proactive Context-Aware Memory Aid
|
The Solution
The solution we propose is a proactive, context-aware memory aid
called the Memory Glasses By creating a reminder system that
is context aware, the role of the system is transformed from a blind,
passive organizer to a proactive, perceptive entity akin to a human
assistant. The delivery of information is situation conditioned;
reminders are associated with context and are delivered when and where
appropriate.
|
Open Questions in the Design and Implementation of the Memory Glasses
|
Open Questions
There are a number of open questions to be addressed in the
construction of a proactive, context aware reminder system. These
questions may be broken down as follows:
- Defining Useful Context What is useful context
for proactive reminders? Assuming that we cannot know everything
about the wearer's state and actions, what are the most important
things to know? Of these, which are technologically feasible to sense
and classify?
Time and location are obviously useful. In addition, it may be
useful to know socially or logistically important features of the
wearer's activity state, such as "in a conversation with person X" or
"driving to work."
- Context Sensing and Classification How do we
sense and classify useful context? What types of sensors, signal
processing, and inference techniques are necessary? What are the
bandwidth and computational requirements for this classification?
Time and location are relatively easy to sense and classify. Aspects
of the wearer's activity state which are independent of time and
location are harder. High-bandwidth, computationally expensive
computer perception techniques may be used to provide
infrastructure-free sensing and classification, and low-bandwidth
low-computing power tag readers and tags may be used in cases where
tagging infrastructure, people, and objects is feasible.
Increasingly, we are seeing the value of medium-bandwidth sensing and
real-time classification of signals such as accelerometer data, as
described in the MIThril Real-Time
Context Engine page
A white paper discussing a preliminary implementation of a
high-bandwidth context sensing Memory Glasses implementation that
predates MIThril is available in HTML
and gzip'd
PostScript. A brief description of our current low-bandwidth
MIThril implementation is described below.
- Wearer Interaction Humans have limited
bandwidth, and a proactive information delivery system must always
assume that it is a secondary task. One of the hardest questions is
how to present information to the wearer in a way that supports their
interaction with and involvement in the world around them. Arguably
this is the most important question, since no amount of sophistication
and effectiveness in context definition, classification and sensing
can be effective if the wearer refuses to use it.
Bad interactions on the desktop are annoying, bad interactions on a
wearable can be life-threatening. If distracted at the wrong time
(crossing the street, driving a car, etc.) the wearer's life may be
placed in real physical danger by an ill-timed, distracting
interaction. In addition, different wearers will have different
preferences, primary tasks, perception limitations and levels of
cognitive function. (See Medical Applications,
below.)
Our recent research suggests that it is possible to support memory
recall without the user's conscious attention. The research behind
this exciting possibility is described in the
Memory Glasses paper that was published in ISWC 2003.
|
Medical Application Classes
|
Classes of Medical Application
One strong area of interest in our research is medical applications
for the Memory Glasses proactive context-aware reminder system. We
believe that almost anyone could benefit from an appropriate, well
implemented Memory Glasses system, but those who stand to gain the
most are people with substantial memory (amnesia) or
recognition (agnosia) problems. In the following section we
outline five classes of possible wearers, ranging from the
substantively unimpaired to the significantly cognitively or
perceptually impaired wearer. Each class presents different
challenges and opportunities. |
Memory Glasses Implementation for Substantively Unimpaired Wearers
|
Substantively Unimpaired Wearers
The substantively unimpaired wearer (a healthy young person, for
instance) who leads a busy life may significantly benefit from
artificial memory and logistical aids, as the author can confirm from
personal experience. The challenge this group of wearers presents is
to build a system which provides a significant benefit beyond what is
provided by innate memory alone and innate memory in combination with
context-blind memory aids.
We believe that the key features of a successful Memory Glasses
implementation is sufficient range in context classification and
well-designed interaction. To be useful to unimpaired users, the
Memory Glasses must function over a broad range of circumstances which
complicates the definition and sensing of relevant context. At the
very least, the Memory Glasses must do no worse than a conventional
context-blind reminder system in situations where the context is
ambiguous or unknown (which is to say, it should not get in the way)
and work reliably in at least some socially or occupationally
important circumstances.
Offsetting the demanding scope of this application class, the
unimpaired cognitive and perceptual capabilities of the wearer will
provide the maximum adaptation and accommodation to the additional
"channel" of input, imposing the fewest constraints on the interaction
design.
|
Memory Glasses Implementation in the Context of Normal Aging
|
Normal Older People
In normal aging, the learning of new material may become more
difficult and the time to recall learned material may increase, but
memory loss per se (amnesia) is not normal. (This is the
result of other diseases, most often Alzheimer's; see the discussion
of amnesia below.) A normal older person might
rely on memory aids somewhat more than an unimpaired younger person,
but the high-level cognitive and perceptual capabilities of normal
older and younger people are comparable.
Decreased acuity in vision and hearing resulting from the normal aging
process may impose somewhat tighter technical constraints on the
interaction design, and increased learning time may require a longer
training and adaptation period. Although a normal older person
(suffering from increased learning and recall time) might have more to
gain from a Memory Glasses system than a normal younger one,
persuading an older person to wear such a system may be difficult,
especially if they have developed effective coping strategies using
conventional memory aids.
|
Memory
Glasses Implementation in the Context of Amnesia, including early
onset Alzheimer's
|
Amnesia Patients
Amnesia is loss of memory resulting from brain damage
(physical trauma or disease) or severe emotional trauma. Amnesia may
occur suddenly or slowly, may effect short-term or long term memory,
and may be permanent or temporary depending on the cause. Effective
treatment of amnesia depends almost entirely on the cause; amnesia
resulting from some types of physical trauma or disease, such as
amnesia resulting from the pressure of tumors or swelling brain tissue,
may be effectively treated surgically or with drugs. Amnesia
resulting from emotional trauma, usually related to a specific event
(anterograde amnesia), may be effectively treated by addressing the
underlying emotional cause of the trauma. Other types of amnesia, such
as that resulting from Alzheimer's disease (also known as primary
degenerative dementia) are progressive and incurable.
We believe that patients with chronic short-term memory problems, such
as those resulting from certain types of head injuries and the early
stages of Alzheimer's, may significantly benefit from a proactive
reminder system. However, the high-level cognitive and perceptual
problems that often accompany amnesia symptoms may seriously restrict
the class of patient likely to benefit.
|
An
Active-Tag Based Implementation of the Memory Glasses for Amnesia
Patients
|
An active-tag based implementation of the Memory Glasses is feasible
in the case where context is circumscribed and the more-or-less
comprehensive tagging of important locations, objects and people is
feasible. Such a setting is often present in the institutionalized
care of amnesia patients, and an active-tag based implementation
greatly simplifies the context sensing and classification problem. If
an effective Memory Glasses implementation could be constructed for
this patient group and setting, it might provide significant
advantages to both patients and care providers by increasing patient
independence and reducing required supervision.
|
Memory
Glasses Implementation in the Context of Agnosia and Prosopagnosia
|
Agnosia (Prosopagnosia) Patients
Agnosia is the inability to recognize objects through the
senses, "a normal perception stripped of its meaning." Agnosia may be
subdivided into two types: associative agnosia is a failure
of recognition (association with memory) given a properly integrated
stimulus, and apperceptive agnosia is a disturbance of the
integration of otherwise normally perceived components of a stimulus
(disturbances in perceived orientation, inability to make comparisons
and matches, etc.). Agnosia typically effects only a single
sensory modality, vision being the most common. A particularly famous
(and clinically important) form of visual Agnosia is
prosopagnosia which is the inability to recognize faces. To
be properly called agnosia, the recognition problems must be
associated with normal or near-normal functioning in other aspects of
high-level cognition and perception.
Agnosia patients in general and prosopagnosia patients in particular
present an interesting application class because the impairment is so
specific. A reliable face recognizer could almost literally replace
the damaged brain functionality of the prosopagnosia patient, perhaps
allowing the patient to lead a normal or near-normal life.
The modality and type of agnosia has interesting implications for the
Memory Glasses interaction design, since the role of the application
becomes the recognition of and translation from an unrecognizable
perception into a recognizable one. For instance, an implementation
for associative prosopagnosia might present the wearer with both a
picture of the recognized face and the name, so that the wearer might
compare the picture with the person and confirm recognition through
visual comparison. However, an implementation for apperceptive
prosopagnosia might present the wearer with name and a recorded sample
of the recognized person's voice, so that the wearer could confirm the
identification aurally.
|
Memory
Glasses Implementation and Anomia, etc.
|
Patients with Other Disorders
It is possible that other disorders of memory and recognition could be
ameliorated through a context-aware reminder system. For instance, a
tag-reader based application which supplied the names of tagged objects
to a sufferer of anomia, an impairment in naming and word
choice that is associated with all forms of aphasia and present to
some degree in the production of normal speech (the "tip of the tongue"
feeling). However, anomia effects all word choice and such an aid
would only be useful for objects immediately at hand.
|
An Active-Tag based Prototype Amnesia/Agnosia Memory Glasses
Implementation
|
Prototype Memory Glasses Implementation
We have constructed an active-tag based amnesia/agnosia Memory Glasses
implementation using a vest-based MIThril system and the "squirt"
active IR tags and tag reader depicted on the MIThril photos page. This is a very simple
application which allows images and text to be associated with tagged
locations and people (or objects). For instance, a picture of Sandy
Pentland and his name might be mapped to tag ID# 18.85.16.46 "seen" by
the forward-looking IR detector mounted on the MicroOptical clip-on
display. A location, the dev-tty1 workstation in the borglab,
might be mapped to tag ID# 18.85.16.118 seen by the upward looking
detector.
Overt Cuing
Overt cues may be associated with people, locations, times, or
combinations of these contexts. If overt cues (in the form of
pictures) were associated with both Sandy Pentland and the
dev-tty1 workstation, a user at dev-tty1 talking to
Sandy might see the following on the head mounted display:
|
Image
and text output (overt cues) for the amnesia/agnosia application,
actual resolution
|
|
|
|
dev-tty1, borglab
|
Sandy Pentland
|
|
|
|
superimposed on one small part of the visual field of one eye.
Specific reminder text (beyond place and person names), other images,
or auditory content could be associated as well. Even if the cues are
nothing more than pictures and names, this application could be a
significant aid to a visual agnosia patient, providing them the means
to recognize the locations and people around them.
Subliminal Cuing Memory Glasses
Subliminal cues can also be associated with contexts such as locations
and people. These cues, in the form of images or text, are triggered
just like the overt cues but presented in such a way as to fall below
the threshold of conscious perception. None the less, our research
suggests that these cues can substantially and significantly improve
performance on a memory recall task (an improvment of about 1.5
compared to the uncued control, p=0.02 one-tailed student's
T). Perhaps more important than the positive effect of correct cuing,
our research suggests that incorrect or misleading subliminal
cues do not interfere with memory recall. This is in stark
contrast to the effect of overt miscues, which appear to have a large
misleading effect. These results are discussed further in our ISWC
2003 paper.
|
Conclusions and Future Work
Our work on the use subliminal visual cues and real-time context
classification continues. We believe that if our findings are
supported by future research, the use of subliminal cues for proactive
memory support could find wide application in a range of task support
applications.
Richard W. DeVaul, Tue Oct 28 16:05:43 EST 2003
|
