Neural networks translated a paralyzed man’s brainwaves into conversational phrases.
What’s new: Researchers at UC San Francisco and UC Berkeley trained a system to interpret electrical impulses from the brain of a man who had lost the ability to speak 15 years ago, and displayed them as words on a video screen.
How it works: The researchers implanted an array of 128 electrodes into the region of the brain responsible for movement of the mouth, lips, jaw, tongue, and larynx. They connected the implant to a computer. Then they asked the patient to try to speak 50 common words and 50 common phrases and recorded the resulting brain activity. They trained the system on 22 hours of these signals, team member Sean Metzger at UC San Francisco told The Batch.
- A stack of three LSTMs detected portions of brain activity related to speech.
- An ensemble of 10 convolutional gated recurrent unit models classified speech signals as one of the 50 words.
- An n-gram language model predicted the probability that a given word would come next.
- A custom Viterbi decoder, an algorithm often used in communications that are subject to transmission errors, determined the most likely of the 50 phrases based on the models’ output.
Results: During tests, the system decoded a median of 15.2 words per minute and translated sentences with a median error rate of 25.6 percent.
Behind the news: The system was built on more than a decade of research by lead author and neurosurgeon Edward F. Chang into links between neurological activity and the sounds of spoken language. A similar project called BrainGate translated brain signals associated with the act of handwriting into text.
Why it matters: Accidents, diseases, and other tragedies rob countless people of their ability to communicate. This technology opens a pathway for them to reconnect.
We’re thinking: It’s wonderful to see natural language models restoring the most natural form of language.