We know that our senses do not operate in isolation or in parallel, but in concert: what we sense through touch is informed by what we see ourselves touch; what we see is shaped by what we hear. This occurs in an extreme way in people with the condition synaesthesia, who may, for example, see colour when they hear sounds or read words.
The process of replacing one sensory modality (e.g. sight) with another (e.g., sound), known as sensory substitution, is fundamental to the research in our Musical Spaces programme. By translating light into sound, we can begin to determine how the ‘hearing’ brain makes sense of the inherently meaningless light patterns that would otherwise fall onto the eyes, and directly explore the relationship between interaction and sensation – thereby explaining the principles of perception generally. Through sensory substitution, people can literally experience themselves making sense of the world, as they begin to hear patterns, and give meaning to those patterns.
Sensory substitution also offers the visually impaired the chance to navigate unfamiliar environments without constraint. And building on our research into sensory substitution, we can also explore the possibility of augmenting the body’s senses. Using sensory augmentation, we can explore further the potential of the human mind, which in turn can create new potential for art and design.
A collaboration with Brazilian composer Eduardo R. Miranda, performed using lottolab’s Soundwall.