An older version of the wearable system incorporated LEDs in order to
perform active segmentation of objects that the wearer held up for
visual inspection. The following text describes an application of the
wearable system, in which the wearable behaves like a creature with its
own agenda to learn from a cooperative wearer.
Human intelligence relies on a wealth of commonsense acquired from a
lifetime of experience. In order to achieve the long term goals of
artificial human intelligence, researchers must find ways to endow
machines with this type of commonsense.
Humanoid robots could serve as a direct approach to the acquisition of
this type of competence, since a sufficiently sophisticated humanoid
robot would be able to experience much of the world in the same way as
humans. Currently, however, humanoid robots have very limited
experience with the world due to obstacles ranging from mechanical
design to social constraints on the use of autonomous robots.
Wearable computing systems have the potential to measure a great deal
of the sensory input and physical output of a person as he or she
experiences everyday activities. Much can be learned through passive
observation of these measurements. However, if we can also find ways
for the wearable system to strongly influence the behavior of the
person wearing the system, many learning tasks can be made easier.
We designed Duo, a wearable creature that works with a cooperative
human in order to learn about everyday objects in the world. The
wearable learns about the world by watching and sometimes making
requests of the wearer as he goes through activities in the day. By
using the same sensory input as the person and co-opting his output
behaviors, the wearable creature serves as a top layer of control in a
subsumption architecture with the human serving as a powerful
mechanical and computational infrastructure. A diagram of the
subsumption architecture for this human/wearable application is shown above.
As an initial exploration into this class of wearable applications, we
created a wearable system that attempts to learn about the everyday
objects with which the wearer interacts during everyday
activities. The creature uses a camera to see what the wearer is
seeing, and orientation sensors to estimate the kinematic
configuration of the wearer's head and dominant arm. The creature also
serves as a high level controller that attempts to co-opt the wearer's
behaviors by requesting actions through headphones. For example, the
creature was able to request that the wearer look at an object that
the wearer was manipulating, in order to see it better and segment it
using the LED array. Likewise Duo could ask that the wearer keep his
head still, in order to make perception easier. We hypothesize that a
broad array of actions useful for learning could be successfully
prompted by speech from Duo. In the future, an application such as
this could ask the wearer to repeat an action by uttering, ``do that
again!'', which should help the creature segment the activity into
meaningful parts. More generally, by requesting actions the wearable
creature could test hypotheses it has made about actions and their
effect in the world.
This figure shows two segmentations of common manipulable objects
by Duo. When Duo detects that the wearer has reached for an object,
Duo requests that the person look at the object via speech through the
headphones. When the person holds up the object to look at it, Duo
flashes the LEDs in order to produce the segmentations shown in this
figure. The first column shows Duo's view before the LED flash and the
second column shows the view during the LED flash. The third column
shows the difference between the flashed and non-flashed images. The
fourth column shows the object and hand mask produced by thresholding
this difference. The final column shows the masks applied to the
images to segment the hands holding the objects in the images.
These behaviors work together with a cooperative human to acquire
high-quality segmentations of everyday manipulable objects used by the
person wearing the system. When Duo detects that the wearer has
reached for an object it asks the wearer to look at it with speech
through the headphones. While looking at the object Duo flashes the
LED array in order to segment the hand and object, which are in the
foreground, from the rest of the world in the background. While
looking at the object, Duo also monitors the wearer's head movements.
If the wearer's head moves significantly, Duo requests that the wearer
keep his head still. Future work for a system of this nature could use
methods we have developed to segment images and detect, track, and
recognize these segments, while segmenting and recognizing actions
of the wearer other than reaching.
This research makes progress towards a viable system for the
acquisition of commonsense related to everyday human
activities. Future creatures could learn to control a set of common
behaviors performed by a cooperative human and learn to relate common
action patterns to the visual appearance of objects and to the
observed changes in the world.