Activity Story: Cross-Modal Perception Workshop

This event and two others—Vision as Contemplation and The Garden of the Lights—emerged from a popular MIT class, Vision in Art and Neuroscience. The course is co-taught by three MIT instructors: Seth Riskin, the manager of the MIT Museum Studio and a light artist; Pawan Sinha, a professor of vision and computational neuroscience; and Sarah Schwettmann, a research Scientist at MIT’s Computer Science and Artificial Intelligence Laboratory and MIT’s IBM Watson AI Lab. During the course, students are exposed to the complete absence of light, and to manipulated experiences of light; these moments are the bedrock for lessons about how the brain perceives and constructs the world—and also serve as inspiration when the students are asked to make works of art that rely on light for effect. Now in its seventh year, Vision in Art and Neuroscience makes neurosciences accessible to all MIT students.

‍

People who are sighted experience the world predominantly through their vision—but people who are blind experience it predominantly through touch. That hadn’t occurred to more than a few people with working vision before they gathered in one of MIT Museum’s large activity rooms, on an afternoon in late February. They were there along with blind and visually impaired visitors from the nearby Perkins School for the Blind to make art and think about perception. 

By way of introducing the participants to the unusual event, MIT Museum Studio Manager Seth Riskin said, “Through art, we can delve into perception in a highly nuanced way, and think about how vision works, how touch works, how vision and touch might be connected, and how we can get to those connections.” To encourage the group to think about those connections, Riskin had everyone touch a specially-made plate—a flat surface with a single raised dot on it. Next, he drew their attention to a different plate, with two raised dots on it, and asked them to touch that. Riskin wanted to know: Did having that additional piece of tactile data introduce a sense of distance or direction, a new dimension, not present in the first plate? And similarly, sequentially, Riskin wanted to know if the next plate, with three dots, furthered any sense the participants might have of dimensionality—if it invited an awareness of depth, for instance. As time went on, the dots multiplied; other plates contained fields of them, radiating beams of them. 

One thing that many of the neuroscientists who’d joined in the event wanted to know was: When—if ever—did the blind participants begin to intuit, or project, depth in the mental images they formed of the raised dots they were touching? In fact, did they form mental images of them?

At least one congenitally blind attendee suggested that she experienced no sense of depth whatsoever. A neuroscientist seated next to her asked her what she “visualized” when she felt the dots. “The question confused her,” says Anchal Sharma, a Fulbright Visiting Ph.D. student who works with MIT Professor of Vision and Computational Neuroscience Pawan Sinha. 

“It seems that visualization wasn’t intuitive in her process of shape identification, as it is for the sighted.” For Sharma, who studies how to better convey three-dimensionality through two-dimensional tactile stimuli, the other woman’s bewilderment suggested something significant: “This is a critical area, where we need to dig in deeper and investigate further.” 

By bringing together people at different stages of blindness, the Cross-Modal Perception Workshop that Riskin spearheaded provided “a very rich platform to understand these concepts further,” as Sharma put it. Currently, no uniform or widely-accepted tactile models exist to help neuroscientists better understand shape and depth perception, she says, so they continue to draw from visual models. But the plates used at the event suggested a way forward.  

Although the question of depth perception in the blind has yet to be sufficiently answered, it has been alive since at least 1866,when the Irish scientist William Molyneux (1656–1698) proposed a thought experiment. In a letter to British philosopher John Locke, Molyneux asked his distinguished friend to imagine that a person blind since birth has their sight miraculously restored. In a case like that, would the person with new vision be able to distinguish a globe from a cube simply by sight? “People were wondering about how we acquire knowledge,” Sinha said, when he spoke to the room about what’s known as Molyneux’s Problem. People born with sight seemed to understand, interpret, or take in the world through all of their senses working together—by way of so-called cross-modal perception. But the puzzle for neuroscientists was, and remains, What drives that cross-modal—or cross-sensory—perception? Is it an innate ability, one that humans are born with? Or is it learned? 

This set of questions is especially alive for Sinha and the people who work in his lab. Since 2005, Sinha has led a joint humanitarian and scientific mission: His team provides a relatively inexpensive cataract surgery for blind children who live in poverty in India, thereby restoring their vision—and subsequently, he and his researchers studies how vision and cognition develops in those children once they can see. “From a basic science perspective, these studies provide unique windows into the brain’s fundamental mechanisms of learning and plasticity,” as Sinha explains on the site for his effort, Project Prakash. “Embedded in the humanitarian aspect of Project Prakash is an unprecedented opportunity to study one of the deepest scientific questions: How does the brain learn to extract meaning from sensory information?” He goes on to say, “The Prakash researchers have begun following the development of visual skills in these unique children to gain insights into fundamental questions regarding object learning and brain plasticity. This is a unique and unprecedented window into some of the most fundamental mysteries of how the brain learns to organize its sensorium.”

As the workshop continued, Riskin introduced a new layer to the activity: He handed out thick pieces of paper and invited all of the participants to use the set of plates on each table to make “rubbings.” Some participants made rubbings using only their fingers and hands, with no drawing implement, creating patterns of raised dots on their papers. Others chose to use black graphite chalk to draw their works. Sharma was fascinated by this aspect of the afternoon too: “I noticed that, at my table, the two visually challenged people and the one blind person applied more pressure to the chalk than the people with vision, because they weren’t able to see the pattern—so they relied on pressing to understand it.” 

Another research scientist who attended the workshop—Michal Fux, who studies the evolution of cognition at MIT’s Sinha Lab for Developmental Research—said, “When someone who has never been able to see feels a plate full of dots, what they are ‘seeing’ is just a collection of dots. But we [the sighted] can’t not see a shape.” But, as she pointed out, the questions that she and her colleagues are trying to answer about vision aren’t relevant merely for people who are blind or visually impaired but for everyone; they have much broader implications, too, when it comes to cognitive flexibility and capacities. 

In addition to the neuroscientists, administrators from both Perkins School and the MIT Museum were enlivened by their unusual sensory experience—and, more, by sharing it with people across the vision spectrum. “It felt so accessible, like everyone could take part and get something from it,” says Jessie Cronan, Senior Director of the CVI Center at The Perkins School for the Blind. So accessible, in fact, that she and her team have subsequently begun to plot with Riskin to recreate the workshop for some of the Perkins children. “We think it will be so interesting for the kids and for us,” she says. 

One of the most fascinating people in the workshop was an older man, Brian Charlson, director of technology at the Carroll Center for the Blind in neighboring Newton, MA. Blinded by a chemical burn accident when he was eleven, he pointed out that people like him sense the world not only through the skin on their hands, but through their feet, their canes, their movement through space. “We can feel the flow of air around us, or the flow of heat,” he said. He picks up cues through his ears, too, which helps to give him a sense of the size of a room, for instance, or what kind of materials it’s built with. After everyone within earshot gathered around him towards the end of the event, he told an anecdote about his wife, who was also blinded in mid-childhood: “Kim used to be able to walk down a sidewalk as a child and know where the parking meters were because of the way they blocked out the sound of traffic.” He concluded by saying, “Our sensory experiences are about more than just these two marvelous things”—and with that, he held up his hands. 

‍