Microsoft Research is showcasing a wrist-worn sensor for freehand 3-D interactions on the move at the
UIST 2012 - the 25th Association for Computing Machinery Symposium on User Interface Software and Technology held Oct. 7 to 10 in Cambridge, Mass.
The paper "Digits: Freehand 3D Interactions Anywhere Using a Wrist-Worn Gloveless Sensor" describes technology that recovers the full 3-D pose of the user's hand. Co-authored by David Kim, a Microsoft Research Ph.D. Fellow from Newcastle University's Culture Lab; Otmar Hilliges, Shahram Izadi, Alex Butler, and Jiawen Chen of Microsoft Research's U.K.-based Cambridge lab; Iason Oikonomidis of Greece's Foundation for Research & Technology; and Patrick Olivier of Newcastle University's Culture Lab' describes Digits, the paper describes a wrist-worn sensor for freehand 3-D interactions on the move. By instrumenting only the wrist, the user's entire hand is left to interact freely without "data gloves," input devices worn as gloves that are most often used in virtual-reality applications to facilitate tactile sensing and fine-motion control. The Digits prototype, whose electronics are self-contained on the user's wrist, optically image the entirety of the user's hand, enabling freehand interactions in a mobile setting.
"The Digits sensor doesn't rely on external infrastructure," Kim explains, "which means users are not bound to a fixed space. They can interact while moving from room to room or running down the street. This finally takes 3-D interaction outside the living room."
To enable ubiquitous 3-D spatial interaction anywhere, Digits had to be lightweight, consume little power, and have the potential to be as small and comfortable as a watch. At the same time, Digits had to deliver superior gesture sensing and "understand" the human hand, from wrist orientation to the angle of each finger joint, so that interaction would not be limited to 3-D points in space. Digits had to understand what the hand is trying to express - even while inside a pocket.
The researchers began with an infrared (IR) camera and an IR laser line generator, but they soon found themselves adding new data sources to enable even richer interaction possibilities. At the moment, the Digits prototype is built entirely from off-the-shelf hardware and is rather bulky: An infrared (IR) camera, an IR laser line generator, an IR diffuse illuminator, and an inertial-measurement unit (IMU) all track hand movements. Digits' main hardware components attached to a wrist brace. By instrumenting only the wrist, the user's entire hand is left to interact freely without wearing a data glove.
"These components enable us to quickly test and verify different ideas, configurations, and form factors without having to worry about the engineering side," Kim says. "We need this kind of flexibility to fail and improve early and to quickly iterate the design. Ultimately, we would like to reduce Digits to the size of a watch that can be worn all the time. We want users to be able to interact spontaneously with their electronic devices using simple gestures and not even have to reach for their devices."
One of the project's main contributions is a real-time signal-processing pipeline that robustly samples key parts of the hand, such as the tips and lower regions of each finger. Other important research achievements are two kinematic models that enable full reconstruction of hand poses from just five key points. The project posed many challenges, but the team agrees that the hardest was extrapolating natural-looking hand motions from a sparse sampling of the key points sensed by the camera.
"We had to understand our own body parts first before we could formulate their workings mathematically," Kim says. "We spent hours just staring at our fingers. We read dozens of scientific papers about the biomechanical properties of the human hand. We tried to correlate these five points with the highly complex motion of the hand. In fact, we completely rewrote each kinematic model about three or four times until we got it just right."
Digits is meant to be a general-purpose interaction platform. The researchers also experimented with eyes-free interfaces, which enable users to leave mobile devices in a pocket or purse and interact with them using hand gestures. Another exciting application area for Digits is in gaming. Currently, Kinect for Windows and commercial game consoles do not support finger tracking. Digits could be complementary to these existing sensing modalities; one option could be to combine Kinect's full-body tracker with Digits' high-fidelity freehand interaction.