Application
Hardware
Human-computer interaction
End User
Research
Sector
Health care
Technology
Neural Network
Robotics
Andrew Rubin sits with a Surface tablet, watching a white skeletal hand open and close on its screen. Rubin’s right hand was amputated a year ago, but he follows these motions with a special device fitted to his upper arm.
Electrodes on his arm connect to a box that records the patterns of nerve signals firing, allowing Rubin to train a prosthetic limb to act like a real hand. “When I think of closing a hand, it’s going to contract certain muscles in my forearm,” he says. “The software recognizes the patterns created when I flex or extend a hand that I do not have.”
The 49-year-old college professor from Washington, DC, drives several times a month to Infinite Biomedical Technologies, a Baltimore startup company that is using deep learning algorithms to recognize the signals in his upper arm that correspond with various hand movements.
Each year, more than 150,000 people have a limb amputated after an accident or for various medical reasons. Most people are then fitted with a prosthetic device that can recognize a limited number of signals to control a hand or foot, for example.
But Infinite and another firm are taking advantage of better signal processing, pattern recognition software and other engineering advances to build new prosthetic controllers that might give Rubin and others an easier life. The key is boosting the amount of data the prosthetic arm can receive, and helping it interpret that information. “The goal for most patients is to get more than two functions, say open or close, or a wrist turn. Pattern recognition allows us to do that,” says Rahul Kaliki, CEO of Infinite. “We are now capturing more activity across the limb.”
Kaliki’s team of 14 employees are building the electronics that go inside other companies’ prosthetic arms. Infinite’s electronic control system, called Sense, records data from up to eight electrodes on his upper arm. Through many hours of training on the company’s tablet app, the device can detect the intent encoded in Rubin’s nerve signals when he moves his upper arm in a certain way. Sense then instructs his prosthetic hand to assume the appropriate grip.
Last Friday, Infinite’s Kaliki received notice from FDA officials that Sense had been approved for sale in the United States. Kaliki says he expects to begin installing them in prosthetic limbs by the end of November. In 2017, FDA officials approved a similar system by Chicago-based Coapt. Today more than 400 people are using the system at home, according to CEO Blair Lock.
Lock started as an engineer 13 years ago at the Rehabilitation Institute of Chicago, an affiliate of Northwestern University. He worked with surgeons who were repairing nerve damage in amputee patients. Over time, he realized that building better prosthetics would be easier if he could figure out a way to pick up better signals from the body, he says. “What’s new is providing a much more natural, more intuitive method of control using [bio-electronic] signals,” Lock says.
In earlier versions of prosthetic devices, electrodes recorded signal strengths “but it was like listening to an orchestra and only knowing how loud the instruments are playing,” Lock says. “It was a significant effort to learn the content and fidelity of the signals.” The Coapt system works inside an amputee’s prosthetic hand and costs about $10,000 to $15,000, depending on the amount of customization needed. Artificial limbs can costs anywhere from $10,000 to $150,000, according to Lock.
Nicole Kelly got a new prosthetic device with the Coapt control system about a year ago. Now the 28-year-old Chicagoan can grind fresh pepper into her food and hold playing cards with friends. She can also open a beer.
“For many things, it wasn't that I couldn't do them before, but suddenly I can do them much easier,” says Kelly, who was born without her lower left arm.nHer prosthetic “is not my body, and it's not 100 percent natural," she said. "There’s a learning curve of my body communicating with this technology. Even the process of the best way to hold the salt and pepper shakers, I am essentially doing it for the first time.”
The Coapt system includes a reset button that allows Kelly to reboot its pattern recognition system if the grips don’t seem to work the way she wants them to. “If at any time I feel like it is doing something funky, I can hit the reset button,” says Kelly, who was a former Miss America contestant and now a disability rights advocate. She says that retraining the hand currently takes about two minutes.
That's not the only innovation. Engineers at Infinite Biomedical are handing out RFID tags to amputee patients so they can stick them on door knobs, kitchen utensils and other household objects—any useful item that requires a specific grip. The idea is that a controller in their prosthetic limbs will detect the RFID signal and automatically change the grip from, say, the one needed to turn a doorknob to the one that lets you pick up a newspaper. The project is underway with funding from the NIH, according to CEO Kaliki.
These technologies are still new and not available for everyone yet. Learning to use one takes a lot of training, and of course, not all insurance companies pay for the most sophisticated prosthetics or these new control systems. Yet patients like Andrew Rubin are hoping that many of these advances come soon. Right now, if he wants to pick up a cup and then open a door, he has to use a smartphone app every time he wants to change the grip on his prosthetic device.
“It’s a slow process, I think eventually we will figure out something that will enable me to not have to rely on my phone to change the grips,” he says. Rubin says he enjoys the weekly training sessions at Infinite in Baltimore, as well as at a Johns Hopkins University bioengineering lab that is developing a glove that can feel pain just like a real hand. But Rubin—who suffered a systemic sepsis infection and also had his leg amputated several years ago— would like get to the point where he could use his right hand to push the shutter on his SLR camera, balance a bowl or perhaps even write with a pen. As the first person to try out Infinite's new pattern recognition system at home, he's not too far away.