FingerTrak: Continuous 3D tracking of hand contour images captured by wristband thermal sensors.

Capturing the complexity of human hands is a daunting task,media outlet New Atlas reported. Now, engineers at Cornell University and the University of Wisconsin-Madison have developed a new wearable system that uses thermal sensors to accurately predict hand positions, potentially for VR, robotics and sign language translation.

FingerTrak: Continuous 3D tracking of hand contour images captured by wristband thermal sensors.

The device, called FingerTrak, is basically a bracelet decorated with four small thermal cameras, each about the size of a pea. The camera takes an image of the contour of the wearer’s wrist from their respective positions. This is enough for a specially designed algorithm to accurately reconstruct the entire hand, including the position of each finger.

“It’s a big discovery for our team — by looking at the contours of your wrist, the technology can reconstruct your fingers in 3D.” Zhang Cheng, the study’s author, said. “This is the first system to rebuild your full hand position based on the contour of your wrist.”

The FingerTrak system uses machine learning to predict the position of 20 finger joints based on the outline of the wrist. The gestures of these hands can then be reproduced in the hands of virtual models and even robots. In the test, the device can accurately reproduce the flipping of books, writing with a pen, drinking water, using mobile phones and other actions.

At present, there are some devices using a variety of technology to try to track hand movements, some use depth sensor camera or infrared sensor to observe the finger, some use motion-sensing gloves, and some use electromagnetic sensors on the fingertips. But almost all of them are a little too bulky to be used in practice. FingerTrak’s design is lighter, and while it’s still not the most comfortable-looking technology product, it does look on the right track.

The team says the FingerTrak system can be used in a range of ways with a few more refinements. It allows virtual reality players to track their hand movements in the game, allows remote-controlled robots to mimic the movements of human operators, helps translate sign language into words or speech, or it can help monitor health problems affecting motor skills.

“How we move our hands and fingers usually tells us about our health, ” said Study author Li Yin. “Devices like this could be used to better understand how older people use their hands in their daily lives to help detect early signs of diseases such as Parkinson’s and Alzheimer’s disease.”

The study was published in the Computational Machinery Association’s Collection of Papers on Interaction, Mobility, Wearable, and Pan-Technology.