The Diverse Mechanisms of Sensory Experience

By Georgeann Sack

Can we close the explanatory gap between the brain and experience?

In a truly excellent review published in Frontiers in Psychology, authors Todd E. Feinberg and Jon Mallatt explain why conscious experience is so hard to understand through scientific inquiry. Not only is consciousness dependent on the complex processes of life itself, it also depends on the integration of information from diverse sensory inputs and processing mechanisms. Though I recommend reading their review in its entirety, here I will elaborate upon sensory experience, because I think it is fundamental for consciousness studies.

First, Feinberg and Mallatt make the critical point that embodiment is a requirement for consciousness.

Especially important for understanding the basis of feeling is that both life and feeling are embodied. Thus, as each living organism has a body with a boundary from the outer world, so consciousness needs a body for a subject to have a personal (first-person) point of view.

Ok, so we have this one body and this one life. How do we protect our body? By sensing and reacting to changes in the world around us that might affect our survival. We achieve the sensing part through sensory consciousness.

In Table 2, Feinberg and Mallatt have compiled a list of special neurobiological features of consciousness. Among them are the following features specific to sensory consciousness:

  • Specialized sensory organs.
  • Neurons arranged into maps of the outside world and body structures.
  • Extensive reciprocal communications in and between pathways for the different senses.

Considering human beings alone, we have distinct neural systems corresponding to each sensation. Each sensory organ detects a different feature of the physical world. Our eyes detect the intensity, wavelength, and motion of light. Our tongue detects soluble chemicals produced by the combination of food and saliva. Our nose detects airborne chemicals. Our ears detect the vibrations of air molecules. Our skin detects mechanical pressure, temperature, and noxious molecules.

Of course you know this already, but have you thought about how those entirely different sensations are then encoded into the brain? Each of our senses is encoded through a distinct neural system (and more finely tuned into distinct neuronal ensembles and microcircuits).

Scientists have been working to identify locations in the brain that become more active when a human research subject is exposed to a specific sensory input. The subject may be exposed to a visual scene, an odorant, or an audio recording, for example. This data has been used to create brain maps. Brain maps include a detailed picture of the structural anatomy of the brain, annotated with information about the function or functions that correlate with neural activity in defined anatomical regions. A major limitation of non-invasive human brain imaging and electrical recording is that we can only get quality functional data for the outermost layer of the human brain, the cerebral cortex. The reason why most human brain maps are of the cortex is because that is what we have the most data for. That said, the cortex is the most recent part of the brain to evolve, and is believed to be necessary for our most human cognitive functions — goal-directed behavior, decision making, and yes, consciousness.

The brain maps of sensory inputs are summarized in Feinberg and Mallatt’s Figure 2, below. The to-date consensus is that sensory signals are encoded into organized maps, and that these maps are remarkably consistent between individual human beings (though increasingly sophisticated analysis is revealing individual differences, see this paper in eLife for an example).

The cerebral cortex processes mapped signals from different senses. Detailed figure legend can be found here. From Feinberg and Mallatt, “Subjectivity “Demystified”: Neurobiology, Evolution, and the Explanatory Gap,” published in Frontiers in Psychology (2019).

That is a nice summary, but what do these maps really look like? Some of the earliest maps were made of the visual system. Imagine you are the research subject, looking into binocular eyepieces. It is dark, except for a periodic flash of light presented at different spots within your visual field of view. By recording activity in your cerebral cortex as you are viewing the flashes of light, researchers would find that for each region of your visual field, there is correlated activity in a specific region of the cerebral cortex. Every time the visual field is illuminated in that region, the cerebral cortex becomes active in a corresponding defined region. If your entire visual field is tested, researchers can create what is called a retinotopic map, like this:

Stimulation in specific regions of the visual field (right) produces activity in specific regions of the cerebral cortex (left), creating an orderly retinotopic map. From Holmes, 1918. Posted by Nicholas V. Swindale.

As you might imagine, brain maps have become increasingly detailed as new brain imaging technology has been invented.

A modern day example showing how increasingly sophisticated brain imaging technology and analysis can reveal individual differences in retinotopic maps, by Noah C Benson and Jonathan Winawer, eLife.

The types of stimulus being used have also become increasingly complex, with some researchers using natural visual scenes for their stimuli. For example, see the interactive brain map from Jack Gallant’s Lab here, made with data from their 2013 publication, “Attention during natural vision warps semantic representation across the human brain” (Cukur et al., Nature Neuroscience, 2013).

Putting the different sensations (and other functions encoded by the cerebral cortex) together, we can now create standardized maps like this:

Of course nobody is satisfied yet, so you can expect these maps to continue to be refined. Efforts are now underway to map deeper regions of the brain (see this paper by Jie Lisa Ji et al. in NeuroImage, published January 2019).

One more note — surprise! It looks like the cerebellum, those weird balls that hang off the underside of the brain, also has retinotopic maps of visual space (see this paper by Daniel M. van Es et al. in Current Biology, published May 2019). Lesion studies suggest that the cerebellum is unimportant for consciousness, but our understanding of cerebellar function is undergoing a revolution, so the jury is still out.

Putting it all together

To summarize the above, every sensation is encoded through organized, distinct neural systems. How can a living being make sense of this information? Feinberg and Mallatt write that “consciousness efficiently codes and organizes large amounts of sensory input into a unified set of phenomenal properties for choosing among may active responses,” but how, exactly, does that happen?

First, the brain needs to integrate sensory information into a coherent model of the world, which is where the extensive cross-talk within and between sensory areas of the brain, one of Feinberg and Mallatt’s special neurobiological features listed above, becomes necessary. This is very much an active area of research. For a nice overview of a recognized sensory systems integrator, the thalamus, see “The Human Thalamus Is an Integrative Hub for Functional Brain Networks” by Kai Hwang et al.

Second, to choose a response or action to take, the brain needs to give sensory inputs a value. Do I need to pay attention to this sensory stimulus? Is it somehow good or bad for my survival as an embodied individual? The human brain encodes value with emotion, also called affective states.

I will be writing more about both of those areas of research in future posts.

Feinberg and Mallatt argue that there are in fact three types of sensory experience.

  1. Exteroceptive, mental models of the world generated from the integration of sensory maps.
  2. Affective states caused by interpretation of sensory inputs as good or bad.
  3. Interoceptive, a mental model of one’s body, including awareness of whether incoming sensations are good or bad for the functioning of one’s body.

They point out that these types of sensory experience also have distinct neural architectures — defined structural arrangements and associated electrical connectivity between neurons. In human beings, exteroceptive processing occurs through sensory-specific hubs of the cerebral cortex as described above, while processing of affective states occurs in a wide distribution of deeper structures in the brain.

Exteroceptive circuits are organized to encode a mapped representation of space, whereas affective circuits encode positive or negative valences instead. …and the reticular formation, which distributes through much of the vertebrate brain stem, projects widely to control the attention and arousal aspects of consciousness.

Brain mechanisms for selective attention and arousal is another of the special neurobiological features of consciousness on Feinberg and Mallatt’s list. You can read more about the reticular formation that controls attention and arousal here.

The key point of telling you about all of these systems is that their existence as specialized neurobiological features of the brain is fundamental to the evolution of consciousness. According to Feinberg and Mallatt, the arrival of animal-on-animal predators about 540 million years ago was the driving force for the evolution of new sensory systems. For animals to function and survive as embodied organisms, they needed to evolve new neurobiological features that could integrate sensory information into a useful model of the world — they evolved perception.

Put simply, we are conscious because we have multiple sensory systems.

I am focusing on human beings, but Feinberg and Mallatt show that the neurobiological features of consciousness have arisen in many different arrangements throughout evolutionary history. This makes it more difficult to understand consciousness across species. That said, just because it is difficult to understand the diverse mechanisms of consciousness, Feinberg and Mallatt are optimistic that we can figure them out using a scientific approach.

But while this complexity makes it difficult to learn how brains create experience, this does not mean the process is mysterious or unexplainable. Many complex biological functions — including life itself — are multi-determined aggregate functions that cannot be reduced to a few factors, yet are nonetheless scientifically explainable.

Feinberg and Mallatt conclude that the “complex neurobiological uniqueness walled off within personal embodiment — makes consciousness appear ‘mysterious’ and inexplicable by known physical law. But we do not need to invoke a ‘mysterious’ explanation to account for the personal and unique aspects of subjectivity.”

A response to Chalmers and Levine

In closing, Feinberg and Mallatt respond to philosopher David Chalmers’ quandry of the character of conscious experience (“Why do individual experiences have their particular nature?” From Chalmers, 1996). Feinberg and Mallatt argue that the mechanism of consciousness can be explained by the diverse, specialized neurobiological features that create it.

The answer lies in the diverse neurobiology behind these varied subjective experiences. For instance, it is clear that the neural pathways of color processing, pain processing, and affect and so on show enormous neurobiological differences. Electromagnetic waves of light have many different physical properties than the mechanical forces of touch, and both differ from chemical odorants, so translating all three stimuli into similar feelings would miss the special properties that make each sense so especially informative. Therefore, these diverse sensations should not —and indeed could not — all have the same subjective “feel.” It should come as no surprise that the phenomenal experiences created by these varied neural architectures differ in how they are subjectively experienced. In other words, the qualitative features of phenomenal properties lie in the neural states themselves; they are not an “additional feature” to the neural states that create them.

Though there are a number of proposed mechanisms of consciousness (Feinberg and Mallatt list them in this paper as the “major-mechanism theories”), whether or not any of them is the true cause of subjective experience is unknown. Thanks to engaged readers and their comments, I have started to think about the mechanisms of consciousness in a new way. I think Feinberg and Mallatt are right — our subjective experiences can be understood as a neural systems mechanism. BUT I have two caveats for your consideration.

It is possible that our focus on electrical signaling through these neural systems is not the whole picture. Perhaps there are other types of signaling underlying consciousness that we simply do not have the technology to measure yet.

I have also come to agree that the mechanisms of experience and the individual feeling of the experience are not the same thing (see conversation with Paul Austin Murphy here).

I don’t feel my neural systems working in concert to give value to sensory inputs, just as I don’t feel my kidneys filtering my blood.

Blasphemy? Mysticism in the midst of my physicalist beliefs? No. Just two different scales of consideration. On one scale, we have our neural system. On the other, we have the individual person embedded in a larger culture.

The neural states of subjective experience can and will be described by systems neuroscience. Understanding the mechanisms of subjective experience will not, however, provide a useful guide for an individual trying to live a good life. It might help us medicate those with mental disorders, but it will not provide the answers that people crave to know. That is why we need to consider consciousness from different perspectives, including analysis of our own experiences, self-experimentation, philosophy, psychology, sociology, and art.

This point is subtle and very hard to explain without sounding like a dualist. I think it underlies much of the argument between neuroscientists and philosophers on the subject of consciousness.

Feinberg and Mallatt say that we can explain what produces subjective experiences through scientific inquiry and existing physical law. I agree, though I think there is always room to understand something more deeply, and perhaps new theoretical frameworks arising from physics or other disciplines will someday deepen our understanding of consciousness.

In the discussion they argue that taking both the embodied organism and the diverse, specialized neurobiological features into account closes the explanatory gap between the physical brain and subjective experiences. This is open for debate, and it depends quite a bit on how you interpret philosopher Joseph Levine (see Levine, 1983).

For example, in a recent paper by Jussi Jylkkä and Henry Railo, they argue that there is always going to be a gap between our mechanistic model of experience and the experience itself. That does not mean that the experience is somehow non-physical. It just means that the model and the thing itself are always going to be different, as is true of any model of the physical world.


Todd Feinberg is a Clinical Professor of Psychiatry and Neurology at Icahn School of Medicine at Mount Sinai, New York. Jon Mallatt is Associate Professor at Washington State University, researching molecular and evolutionary biology. Over the last 7 years they have had a working relationship dedicated to understanding consciousness. During this time, they have produced 3 research papers (paper 1paper 2paper 3) and 2 books on the subject (book 1book 2, additional books by Feinberg). Their most recent paper, discussed in part here, is impressive in its breadth and depth and I highly recommend that you read it.

Leave a reply