MIT researchers channel AI to turn hand gestures into robot training data

Humanoid robots struggling with tasks like grasping a cup have a new teacher - a person wearing an ultrasound wristband that captures the movement of muscles, tendons and ligaments beneath the skin.

Researchers at the Massachusetts Institute of Technology developed the tool to collect data of human hand motion that could eventually help robots achieve the dexterity that has been difficult for machines to master.

"Imagine people doing housework," said Xuanhe Zhao, an MIT professor of mechanical engineering. "We can use the data obtained by our system to train a robot to do exactly (that) housework with this dexterous hand motion."


As much of the tech world is still captivated with artificial intelligence assistants that are taking on computer-based tasks, Zhao is among the scientists trying to imbue AI with more sensory data from the physical world.

Beyond housework, the technology could help with other tasks that require flexing fingers and hands, such as surgery.

The wristband uses high-frequency sound waves to "see" through its wearer's skin. It relays images of the muscle and tendon movements to a computer that uses AI to enable a nearby robotic hand to mimic the gestures.

An AI algorithm is trained to decode images generated by the device into what engineers call degrees of freedom - specific ways a joint can bend or rotate. The human hand has 22 of them.

In earlier systems, tracking even a fraction of those movements was a significant challenge.

In laboratory demonstrations with eight volunteers, developers showed the wristband could precisely mirror hand gestures - including all 26 letters in American Sign Language - within 120 milliseconds.

The wristband can operate wirelessly, meaning the controlling person and the receiving robot need not be in the same room.

Beyond remote control, the team sees a path toward using the wristband to build huge datasets of human motion that could eventually enable humanoids to learn dexterous tasks without human guidance.