When Constantine Caravassilis listens to stringed instruments, strange things happen. If he hears a chord played in the low range, his eyes might suddenly flood with colour: “a G,” he tells me, “is usually orange.” At other times, this type of sound can cause him to experience sweet or bitter tastes.

Caravassilis, an accomplished composer and doctoral student at the University of Toronto’s Faculty of Music, has an unusually strong case of synesthesia – a condition in which the stimulation of one sensory pathway leads automatically to the arousal of another.

Synesthesia isn’t unique to musicians, although they may be disproportionately affected by it. It wasn’t until his second year at university that Caravassilis learned that several other composers (such as Claude Debussy and Alexander Scriabin) shared what he thinks of as “an ability, not a malfunction. But you wouldn’t describe it as a negative or positive experience,” he says. “It just is.”

Up until recently, it would have been easy to dismiss Caravassilis as delusional: after all, creative people are known for having active imaginations. Now, however, what synesthetes say they experience is backed up by science. Using functional magnetic resonance imaging (fMRI), neuroscientists have discovered that there is much more crosstalk among the senses than we ever imagined before. It just so happens that Caravassilis’s is much louder than most.

But if neuroscience is telling us that the most profound synesthetes truly “see” a colour invisible to most of us, then what exactly do we mean when we talk about vision? Or for that matter, about taste, hearing, smell and touch?

Professor Mohan Matthen is trying his best to answer this question. He is currently the principal investigator at the Network for Sensory Research, an international team of philosophers headquartered at U of T who believe it is high time we developed a new conceptual framework for the senses.

It seems natural that philosophers should be leading this investigation; after all, it was Aristotle who originally conceived of the five-sense model to which we rigorously cling. And until the scientific method was developed in the 17th century, investigation of the senses belonged to the philosophers alone. Today, they share the stage with neuroscientists, psychologists, medical doctors and biologists. And findings within these fields are reframing philosophical thinking in fascinating ways.

Matthen himself came to philosophy via the sciences: his first degree was in physics, and he has also taught the philosophy of biology. His first exposure to the domain that would shape his life came when a teacher in his native India recommended that he read Appearance and Reality by the British metaphysician F.H. Bradley. On his chatty blog, Matthen jokes that the book (and his teacher) actually caused him “much misery”; nonetheless, it spurred him to study human perception.

Other philosophers around the world have been probing the mystery of the senses for some time. Barry Smith, codirector of the University of London’s Centre for the Study of the Senses, is best known as a specialist in flavour and smell. Fiona Macpherson, who is the director of the University of Glasgow’s Centre for the Study of Perceptual Experience, is an expert in the nature of visual experience, including optical illusions. Matthen has brought these researchers together – in person, when possible – with like-minded thinkers from Harvard, M.I.T. and elsewhere. “We want people to communicate, share each others’ work and get access to faculty members in other disciplines,” he says. “We’re particularly interested in multi-sensory integration and how the senses contribute to knowledge.”

A key question the network wants to address is whether Aristotle’s model is still relevant. “The traditional five senses are external, but we’re also interested in the internal senses – those that have to do with a sense of what your own body is doing,” Matthen says. These include proprioception (knowing where your body is in space); nociception (the feeling of pain); and thermoception (temperature sense), among others.

Matthen’s colleague Fiona Macpherson points out that animals have certain senses that we lack. “There are fish who are sensitive to electric fields. And there’s quite good evidence that some animals are sensitive to magnetic north, which we aren’t.” We humans might possess a vomeronasal organ – which animals famously use to sniff each others’ pheromones – but the jury is still out on whether a human sense functions this way. So if we no longer have five senses, then how many do we have?

Like a practiced synesthete, I can see Matthen’s head shaking over the phone. “There’s not much point in counting them,” he says. “What we’re more interested in is how they come together.” Barry Smith expands on this. “You could have more than one sense of smell, because you’ve got the smelling from the outside in when you take a breath. But you’re also smelling aromas that enter the sinus cavity from inside the mouth.”

An explanation: when I attend one of Smith’s talks, he offers everyone in attendance a jelly bean, and tells us to hold our nose while chewing. My jelly bean is coconut flavoured; with my nose held, I can only perceive that it’s “sweet” (in that respect, no different in any way from raspberry or chocolate). The coconut flavour only becomes apparent when I unplug my nose. Smith’s point is clear: what we call “flavour” is a blend of tongue-taste and smell. “None of the parts operate separately anyway,” he says. “So how can we think of them as parts?”

None of the parts operate separately. It’s an idea that completely upends what we all learned as schoolchildren: there are five individuated senses, some more cherished than others. And yet we know from experience how integrated they all must be. When we have a cold, for example, taste and smell are equally diminished. And instinctively, we know that beautifully presented food somehow tastes better.

Sensory fusion is also illustrated by the McGurk effect, where you watch a mouth forming the sound “ga” while the sound “ba” is being played. What you will then hear is wrong: it’s the sound “da,” the midpoint between the two. (There are several video demonstrations of this effect on YouTube.) “So the question is, do we partly hear with our eyes?” asks Smith. “And the thought seems to be, yes. You’re fusing hearing and vision to make some new product. The way we’re talking about hearing and vision no longer depends on input from just one sense, and as a result we’ve had to tear up our old ideas.”

And yet, it’s not as if Aristotle was completely wrong: there are dividing lines, but where are they? On a sunny day in May, Matthen gathers members of the network at a winetasting in the Niagara region of Ontario. Smith is, among other things, an oenophile – wine-tasting being a discipline that naturally combines all the senses at once. “Smell this!” he demands, proffering a glass of Riesling. “It has notes of diesel and lime.” This doesn’t sound inviting, and Smith is

right: what I inhale seems nothing less than mildly citric gasoline.

But tasting is a different matter altogether. On drinking the wine, I perceive it as sweet and floral, its flavour only a distant cousin to its scent. Smith says this disconnection is common in the flavour business. He points out the example of Époisses cheese, which tastes delightful but smells like a “teenager’s training shoe.” It’s clear that there are separate perceptual systems operating here. But the war may not be between smell and taste – instead, it could be one of my smell-senses rejecting the information from another.

It appears that we may have multiple sight senses, too. Take the remarkable example of Daniel Kish, a Californian who had his eyes removed as a toddler due to cancer. To navigate the world, Kish echolocates: he uses vocal clicks to activate a kind of sonar system more commonly associated with animals such as bats. A recent study showed that Kish’s method can help him tell a car from a lamppost, or a flat object from one that is convex. He can also stand near your car and tell you how far it is from the curb.

Amazingly, brain scans show that Kish’s visual cortex lights up when he is “looking” at something, even though he is echolocating the object instead of seeing it in a traditional manner. So in one very key sense, Kish has not lost the ability to see things – just the usual way of doing so.

And yet, as Fiona Macpherson points out, the very words “visual cortex” might be erroneous; after all, it’s a given that a person with no eyes cannot see. “This area of the brain is clearly doing a lot of visual processing – but is it exclusively visual?” she asks. “It might be better to call it a spatial-processing cortex.”

In any case, “when somebody loses a sense,” says Matthen, “they often manage to get the same information in a different way. That’s of vital interest to us.”

Matthen points out that whether disabled or not, all human beings use their senses in concert all the time, though they may not be conscious of it. When one sense fails or feels untrustworthy, we automatically let another take over. “If you don’t trust the colour of something, you might turn it over, use motion to manipulate the object and learn more about it,” he says. “Vision can make mistakes, but generally by interacting with an object in a multi-sensory way we can check those mistakes.”

So are we all synesthetic? Fiona Macpherson believes that a case such as that of Caravassilis – true synesthesia – is relatively rare. But she thinks we all experience cross-modal phenomena. Take the “Bouba-Kiki” experiment of 2001, in which people were shown two pictorial figures – one rounded, the other angular. Ninety-five per cent of participants assigned the name “kiki” to the angular figure and “bouba” to the rounded one, proving a link between visual and auditory faculties in the brain. Macpherson says we frequently make other synesthesia-like associations, too. “Suppose I gave you a blank piece of paper and a pen, and I asked you to draw how the days of the week were related to each other,” she says. “How would you do it?”

I tell her that it’s nonsense to think the days of the week are spatially related. But in her mind, they are. “I would draw a circle that goes anticlockwise, with Saturday and Sunday at the top,” she says. I tell her that strikes me as frankly weird – but she returns the favour when I tell her the appearance of sloppily printed letters can sometimes make the skin on my thumbs feel itchy.

“One of the nice things about these studies,” she says, “is that we’re realizing the way human beings think about things is really idiosyncratic. What goes on in our heads is so unique, because of the rich, complex people that we are.”

***

What is a “sense,” anyway? As a verb, it means to grasp, or feel, or understand. As a noun, it has traditionally referred to a bodily faculty that enables us to do these things.

And yet, even those simple definitions are currently up for review. It might even be possible to sense something without being aware of it. “There is emotional communication through chemical signalling,” says Smith, noting that researchers at the Weizmann Institute in Israel last year found evidence that chemical signals from a woman’s tears lower men’s sexual interest, even though tears give off no discernible odour.

Advances in science are not only doing away with how we view the senses, but how we view philosophy itself. Since the invention of the scientific method, a chasm has opened between the two disciplines. And unless philosophy works to keep up, a good deal of what we’ve traditionally thought risks invalidation.

“When philosophers start telling you how it is, I start to get worried,” says Smith. “Especially if they’re talking about the mind, or language or emotions, and they don’t look at the relevant recent science on these topics.” He points out that many outmoded philosophical views are vital links in a chain that is still snaking through history towards the truth. But if these views are no longer tenable, we should no longer teach them as gospel.

“We’ve given up the idea that the world is composed of four elements. Why do we hold on to Artistotle’s view that there are just five senses?” Smith asks. “Somehow this is a bit of folk ideology that still remains.”

Neuroscience has revolutionized philosophy. Technology such as fMRI offers a picture of the self that seems to contradict the fragile and unreliable accounts of it we like to give each other.

It may seem a broad statement, but Macpherson reminds me how fundamental these questions are to philosophers. “One of the big philosophical questions that everybody knows is, how do I know that the world around me really exists, and is as I take it to be? That question arose because people thought: well, maybe I’m just hallucinating it all. Maybe I’m in the Matrix, and sentient machines are tampering with my brain.”

So if science is discrediting much of what philosophers used to think about the nature of perception, why should philosophers participate in this debate at all? Matthen says that the examination of subjective experience – how it feels to be human, regardless of what any lab test might tell us – is still very much the province of philosophers, and has always been a significant area of study. Our preference for viewing the world unscientifically may be annoying and frustrating. But it’s also key to understanding who we are.

Smith agrees. “We all know the sun isn’t really moving, but we still talk about it ‘setting.’ How do we connect the lived experience, the way things seem to us, with what’s really going on? It’s the job of the philosopher to do that.”

“If the doors of perception were cleansed, everything would appear to man as it is, infinite.” Since William Blake wrote that over 200 years ago, many (most famously, the writer Aldous Huxley) have tried to alter their perceptual experience with drugs. But instead of stumbling through a drugged haze, some modern-day Huxleys are now tweaking their senses with different kinds of substances.

One of these is miraculin, a derivative of a west African berry that strips lemons of their acidity and makes them taste as sweet as peaches. Miraculin has more serious uses too. People undergoing chemotherapy – who often find that food tastes unpleasantly metallic – can use it to positively alter the flavour of what they eat.

Flavourless jelly beans, sweet lemons and wine that smells like gasoline: it’s not hard to believe Smith when he says that sensory research is “a lot of fun.” Those attracted to it are quirky sorts, preoccupied with questions that wouldn’t trouble most of us. “One of the things that’s really nice about this work,” says Matthen, “is that everything you do, even if it’s terribly mundane, suddenly takes on more meaning. You might notice that when you’re driving, you don’t have to see the corners of your car to know where they are. The car responds to your own movements and when it does that, it becomes integrated into your own bodily sense.”

Being most concerned with questions of taste, scent and flavour, Smith admits to having acquired an overdeveloped sense of smell. “I can’t turn it off!” he laughs. “I walk into rooms and I smell people, or the rooms themselves; when people walk by I’m noticing their different sensory fingerprints.”

Isn’t that unpleasant? Not at all, he says. “You think: all of this was going on, and I’ve been missing it. To put yourself back in touch with your animal nature, your senses, your contact with the environment is wonderful. You feel healthier, more complete.”

Back in Toronto, Constantine Caravassilis is working on something very special to him: a huge, colour-coded musical project, which he plans to finish in two or three years. “Instead of preludes and fugues in D major or C minor, we’ll have preludes and fugues in green or orange,” he says.

He uses software that converts his piano’s sounds to string sounds, and shifts them into a lower range. When that happens, his synesthesia kicks in and his mind erupts in colour, tastes and emotions. “With the part I’m working on now, I’m trying to stick with beige,” he says. “But it’s very difficult. I’ll spend days on just three bars, and then all of a sudden my fugue subject wants to turn red! So that’s the challenge. I have to find a way to keep it going . . . in the same colour.”