Dr. Amir Amedi wears a camera and headphones, two key components in his EyeMusic SSD system, which helps blind people see shapes and colours using music. (Photo: Amir Amedi Lab)
Can music help blind people see? That's the theory being tested in a new app called EyeMusic SSD.
The app uses a camera to identify objects, then, after filtering the information through a computer algorithm, creates a series of musical cues to let the user know what they're looking at. The sounds can be read like language — with training, blind users can identify patterns of notes and what those notes represent. That can be everything from facial expressions to landscapes to individual letters and numbers — though right now, research and testing, as reported in the journal Restorative Neurology and Neuroscience, is focused just on shapes and colours.
The technology was developed by Dr. Amir Amedi, a neuroscientist at the Hebrew University of Jerusalem, along with his team at the Edmond and Lily Safra Center for Brain Sciences and the Institute for Medical Research Israel-Canada. Amedi's work builds on previous attempts at Sensory Substitution Devices (SSDs) that have been percolating for decades, but never took off. His research is showing that it's possible to use music to represent objects — and that, because the system only requires a smartphone with a camera and a pair of headphones — it could also be practical.
"This study is a demonstration of abilities showing that it is possible to encode the basic building blocks of shape using the EyeMusic," Amedi told Science Daily. "Furthermore, the success in associating colour to musical timbre holds promise for facilitating the representation of more complex shapes."
During testing, it usually took both blind people and blindfolded sighted people (who acted as a control group) two or three hours to learn rudimentary auditory symbols, while the full program available on the app takes about 70 hours to learn. If you have a bit of spare time, you can test Amedi's process on his website.
Here's Amedi's Ted Talk from 2012, in which he describes his process and his research in more detail:
While the app is evidence of progress in this field of research, it could only be the beginning of what's possible.
“The human brain is more flexible than we thought,” Amedi concluded in his report. “These results give a lot of hope for the successful regaining of visual functions using cheap, non-invasive SSDs or other, invasive sight restoration approaches. In the blind, brain areas have the potential to be ‘awakened’ to processing visual properties and tasks even after years or maybe even lifelong blindness.”
EyeMusic SSD is currently available for free in iTunes.
This has been a big couple of weeks for radical sensory innovations. Last week, a video of a woman having her cochlear implants — electronic devices that can help profoundly deaf people hear — turned on went viral. The woman in question was Jo Milne, a British charity worker who had never been able to hear before. In case you missed it, here's the emotional video:
Today, Milne published an article in The Guardian in which she talks about the experience of hearing for the first time, of becoming an overnight Internet sensation — and of living with Usher syndrome, which is causing her to lose her sight as well.