Valerie Pezeron's illustration for 'How do I look?'Both John Locke and Isaac Newton realised that objects are not coloured, and that against all appearances light is not coloured either. This is still not generally recognised even now, four hundred years later, because it seems so implausible. Yet it tells us something very important – that perceptions are not identical with what we perceive and may be very different.

Locke had the idea that experiences of colour – “qualia” in current philosophy-speak – are tokens for what exists in the object world, and may be as different as the word “cat” is from the living pet. For Locke the sky-blue one sees is not in the sky, but is created in the brain – so colours do not exist without eyes and brains, and are doubtless very different with different eyes or different brains. That there are objects in the world is not denied; but one might say we know them as somewhat doubtful hypotheses. These hypotheses are constructed by us, through a combination of sensory data and, crucially, past experience that conditions our expectations. In large part we see what we expect to see and we may be blind to that which is there but totally unexpected.

The most accurate historical account of perception is that of the 19th-century German scientist Hermann von Helmholtz. He thought that perceptions are unconscious inferences we make based on a combination of clues provided by the eyes and other senses and knowledge of the world. This idea of unconscious inference for perception preceded, by several years, Freud’s notion of the unconscious, and was treated with equal derision because it undermined the notion of humans as pre-eminently rational beings who could be held responsible for their actions. The notion of unconscious processes determining action and affecting perception was profoundly disturbing; how could there be fair blame or praise without conscious decisions?

Yet the present view of the brain sciences is that decisions are not made consciously. For consciousness is focused attention, while brain-decisions are best made by evaluating many sensory inputs simultaneously, and what they might mean.

Crucially perception of the present depends on rich, though of course not always correct or appropriate, knowledge from the past. We interpret sense data (what we hear, touch, taste, see and smell) from the present according to what we already know. It might be said that we see the present through knowledge from the past enabling us to predict the immediate future, which is what most concerns us for this is where danger and reward lie, and we must adjust our behaviour to avoid the one or achieve the other. This raises the question: if we see the present through memory, why aren’t past and present confused? They can be. The pioneering Russian neurologist Alexander Luria described the remarkable memory man Mr S as having just such confusions, for example mistaking seeing his clock for remembering it, and so failing to get up in the morning. This suggests that perhaps an important function of qualia is to flag the present – the Here and the Now.

Qualia of perception have a vividness that is rare for memories, which might be how we are able to separate them. Try this: look at something for a few seconds and then shut your eyes, and visualise it in memory. You will almost certainly find that the memory is pale by comparison with the perception. Perhaps this is why past and present are not normally confused. Luria’s Mr S had exceptionally vivid memories, and rich synaesthesia (experiencing qualia from another sense, such as musical notes experienced as colours), which may be why he confused seeing with having seen.

The complexity of processes involved in how we look, and the role that other senses and memory play in what we think we see (or don’t see), first impressed itself on me 45 years ago. With my colleague Jean Wallace, I studied the rare case of Sydney Bradford, a man who had been born blind but, through a corneal graft at the age of 52, suddenly found himself able to see. Almost immediately after the operation he was able to “see” but he could only see those things that he already knew about, having experienced them through touch. It was his touch memories that enabled him to perceive them with his eyes. After his operation Bradford was taken to the zoo. He proved utterly unable to see the elephant standing in front of him, as he had no knowledge to make sense of his perceptions.

The more recent case of Mike May in California is similar. Since his operation his sight has gradually improved as he learns to see, for example by understanding how shadows represent depth and tell us about the shape of things. Some of the consequences of May’s new-found vision were less happy. He had been a champion blind skier but following the operation he would have to shut his eyes while skiing to block out what he now found was a terrifying sight.

But acceptance of this intimate connection between memory and perception, even though it was first noticed in the 17th century, has been slow in brain science. Despite the fact that state-of-the-art brain imaging (fMRI) shows that perception animates parts of the brain associated with both present information and memory, most research on memory and perception is still undertaken as if these were separate processes. Seeing used to be thought of as taking place only in the eyes and in quite specialised brain regions; but now it seems that half the brain is occupied with seeing, requiring a lot of energy. Perhaps this is why we shut our eyes for a rest.

As the mask rotates we know the back is hollow, but our brains won't accept itIt is not just extreme cases like Mike May, but also much more common errors of seeing – illusions – that can reveal the crucial role of memory in governing what we (think we) see. The past can be profoundly misleading for the present, providing out-of-date maps. Perception depends on specific knowledge and probabilities. Our brains calculate the likelihood of what is out there, and when too far-fetched, perceptions are rejected. Highly improbable objects or situations are impossible to see; our brain refuses to see them as they are, and instead we see them as we expect them to be.

A dramatic and discomforting example is looking at a hollow face-mask, like the back of a joke shop mask. It will appear as a convex, nose-sticking-out face, unless seen very close up with both eyes or at an angle when characteristic features are hidden. If a mask is rotated our brains, convinced through experience that noses only ever poke out, will read both sides of the mask as convex though we “know” that one is concave. It is almost, if not quite, impossible to paint or photograph a hollow face to look as it is – hollow. Science often learns from what does not happen: not seeing a hollow face as hollow is the most revealing experiment on perception. The unsettling truth from brain science is that we see what, at some level, we expect to see, and often miss things as they really are.

Illustration by Valerie Pezeron