Hack 13. Understand Visual Processing

The visual system is a complex network of modules and pathways, all specializing in different tasks to contribute to our eventual impression of the world.

When we talk about "visual processing," the natural mode of thinking is of a fairly self-contained process. In this model, the eye would be like a video camera, capturing a sequence of photographs of whatever the head happens to be looking at at the time and sending these to the brain to be processed. After "processing" (whatever that might be), the brain would add the photographs to the rest of the intelligence it has gathered about the world around it and decide where to turn the head next. And so the routine would begin again. If the brain were a computer, this neat encapsulation would be how the visual subsystem would probably work.

With that (admittedly, straw man) example in mind, we'll take a tour of vision that shows just how nonsequential it all really is.

And one need go no further than the very idea of the eyes as passive receptors of photograph-like images to find the first fault in the straw man. Vision starts with the entire body: we walk around, and move our eyes and head, to capture depth information [Hack #22] like parallax and more. Some of these decisions about how to move are made early in visual processing, often before any object recognition or conscious understanding has come into play.

This pattern of vision as an interactive process, including many feedback loops before processing has reached conscious perception, is a common one. It's true there's a progression from raw to processed visual signal, but it's a mixed-up, messy kind of progression. Processing takes time, and there's a definite incentive for the brain to make use of information as soon as it's been extracted; there's no time to wait for processing to "complete" before using the extracted information. All it takes is a rapidly growing dark patch in our visual field to make us flinch involuntarily [Hack #32], as if something were looming over us. That's an example of an effect that occurs early in visual processing.

But let's look not at the mechanisms of the early visual system, but how it's used. What are the endpoints of all this processing? By the time perception reaches consciousness, another world has been layered on top of it. Instead of seeing colors, shapes, and changes over time (all that's really available to the eyes), we see whole objects. We see depth, and we have a sense of when things are moving. Some objects seem to stand out as we pay attention to them, and others recede into the background. Consciously, we see both the world and assembled result of the processing the brain has performed, in order to work around constraints (such as the eyes' blind spot [Hack #16] ), and to give us a head start in reacting with best-guess assumptions. The hacks in this chapter run the whole production line of visual processing, using visual illusions and anomalies to point out some detail of how vision works.

But before diving straight into all that, it's useful to have an overview of what's actually meant by the visual system. We'll start at the eye, see how signals from there go almost directly to the primary visual cortex on the back of the brain, and from there are distributed in two major streams. After that, visual information distributes and merges with the general functions of the cortex itself.

2.2.1. Start at the Retina

In a sense, light landing on the retinathe sensory surface at the back of the eyeis already inside the brain. The whole central nervous system (the brain and spinal column [Hack #7]) is contained within a number of membranes, the outermost of which is called the dura mater. The white of your eye, the surface that protects the eye itself, is a continuation of this membrane, meaning the eye is inside the same sac. It's as if two parts of your brain had decided to bulge out of your head and become your eyes, but without becoming separate organs.

The retina is a surface of cells at the back of your eye, containing a layer of photoreceptors, cells that detect light and convert it to electrical signals. For most of the eye, signals are aggregateda hundred photoreceptors will pass their signal onto a single cell further along in the chain. In the center of the eye, a place called the fovea, there is no such signal compression. (The population density of photoreceptors changes considerably across the retina [Hack #14] .) The resolution at the fovea is as high as it can be, with cells packed in, and the uncompressed signal dispatched, along with all the other information from other cells, down the optic nerve. The optic nerve is a bundle of projections from the neurons that sit behind the photoreceptors in the retina, carrying electrical information toward the brain, the path of information out of the eye. The size of the optic nerve is such that it creates a hole in our field of vision, as photoreceptors can't sit over the spot where it quits the eyeball (that's what's referred to as the blind spot [Hack #16] ).

2.2.2. Behind the Eyes

Just behind the eyes, in the middle, the optic nerves from each eye meet, split, and recombine in a new fashion, at the optic chiasm. Both the right halves of the two retinas are dispatched to the left of the brain and vice versa (from here on, the two hemispheres of the brain are mirror images of each other). It seems a little odd to divide processing directly down the center of the visual field, rather than by eye, but this allows a single side of the brain to compare the same scene as observed by both eyes, which it needs to get access to depth information.

The route plan now is a dash from the optic chiasm right to the back of the brain, to reach the visual cortex, which is where the real work starts happening. Along the way, there's a single pit stop at a small region buried deep within the brain called the lateral geniculate nucleus, or LGN (there's one of these in each hemisphere, of course).

Already, this is where it gets a little messy. Not every signal that passes through the optic chiasm goes to the visual cortex. Some go to the superior colliculus, which is like an emergency visual system. Sitting in the midbrain, it helps with decisions on head and eye orienting. The midbrain is an evolutionary, ancient part of the brain, involved with more basic responses than the cortex and forebrain, which are both better developed in humans. (See [Hack #7] for a quick tour.) So it looks as if this region is all low-level functioning. But also, confusingly, the superior colliculus influences high-level functions, as when it suddenly pushes urgent visual signals into conscious awareness [Hack #37] .

Actually, the LGN isn't a simple relay station. It deals almost entirely with optical information, all 1.5 million cells of it. But it also takes input from areas of the brain that deal with what you're paying attention to, as well as from the cortex in general, and mixes that in too. Before visual features have as been extracted from the raw visual information, sophisticated input from elsewhere is being addedwe're not really sure of what's happening here.

There's another division of the visual signal here, too. The LGN has processing pathways for two separate signals: coarse, low-resolution data (lacking in color) goes into the magnocellular pathway. High-resolution information goes along the parvocellular pathway. Although there are many subsequent crossovers, this division remains throughout the visual system.

2.2.3. Enter the Visual Cortex

From the LGN, the signals are sent directly to the visual cortex. At the lower back of the cerebrum (so about a third of the way up your brain, on the back of your head, and toward the middle) is an area of the cortex called either the striate or primary visual cortex. It's called "striate" simply because it contains a dark stripe when closely examined.

Why the stripes? The primary visual cortex is literally six layers of cells, with a thicker and subdivided layer four where the two different pathways from the LGN land. These projections from LGN create the dark band that gives the striate cortex its name. As visual information moves through this region, cells in all six layers play a role in extracting different features. It's way more complex than the LGNthe striate contains about 200 million cells.

The first batch of processing takes place in a module called V1. V1 holds a map of the retina as source material, which looks more or less like the area of the eye it's dealing with, only distorted. The part of the map that represents the foveathe high-resolution center of the eyeis all out of proportion because of the number of cells dedicated to it. It's as large as the rest of the map put together.

Physically standing on top of this map are what are called hypercolumns. A hypercolumn is a stack of cells performing processing that sits on top of an individual location and extracts basic information. So some neurons will become active when they see a particular color, others when they see a line segment at a particular angle, and other more complex ones when they see lines at certain angles moving in particular directions. This first map and its associated hypercolumns constitute the area V1 (V for "vision"); it performs really simple feature extraction.

The subsequent visual processing areas named V2 and V3 (again, V for "vision," the number just denotes order), also in the visual cortex, are similar. Information gets bumped from V1 to V2 by dumping it into V2's own map, which acts as the center for its batch of processing. V3 follows the same pattern: at the end of each stage, the map is recombined and passed on.

2.2.4. "What" and "Where" Processing Streams

So far visual processing has been mostly linear. There are feedback (the LGN gets information from elsewhere on the cortex, for example) and crossovers, but mostly the coarse and fine visual pathways have been processed separately and there's been a reasonably steady progression from the eye to the primary visual cortex.

From V3, visual information is sent to dozens of areas all over the cortex. These modules send information to one another and draw from and feed other areas. It stops being a production line and turns into a big construction site, with many areas extracting and associating different features, all simultaneously.

There's still a broad distinction between the two pathways though. The coarse visual information, the magnocellular pathway, flows up to the top of the head. It's called the dorsal stream, or, more memorably, the "where" stream. From here on, there are modules to spot motion and to look for broad features.

The fine detail of vision from the parvocellular pathway comes out of the primary visual cortex and flows down the ventral streamthe "what" stream. The destination for this stream is the inferior temporal lobe, the underside of the cerebrum, above and behind the eyes.

As the name suggests, the "what" stream is all about object recognition. On the way to the temporal lobe, there's a stop-off for a little further processing at a unit called the lateral occipital complex (LOC). What happens here is key to what'll happen at the final destination points of the "what" stream. The LOC looks for similarity in color and orientation and groups parts of the visual map together into objects, separating them from the background.

Later on, these objects will be recognized as faces or whatever else. It represents a common method: the visual information is processed to look for features. When found, information about those features is added to the pool of data, and the whole lot is sent on.

2.2.5. Processing with Built-in Assumptions

The wiring diagram for all the subsequent motion detection and object recognition modules is enormously complex. After basic feature extraction, there's still number judgment, following moving objects, and spotting biological motion [Hack #77] to be done. At a certain point, the defining characteristic of the cortex as a whole must come into play, and visual information is processed enough to be associated with memory, language, and reading emotions. This is where it blends in to the higher-order functions of the whole brain.

In the hacks that follow, we'll explore the effects of early and late visual processing. A common thread through these effects will be the assumptions the visual system has made about the visual world to expedite its computationand by looking at the quirks of vision, we can draw some of these out. Assumptions like the visual world remaining relatively stable from second to second (so we don't notice if it doesn't [Hack #40] ) and supposing that dark areas are shadows, which is the quirk that makeup takes advantage of [Hack #20] .

In a sense, the fact that we can observe these assumptions suggests that the visual system assumes as much about the external environment as about its own modules. The visual system's expectation that the motion module will report motion correctly (and therefore our confusion when the module doesn't identify motion correctly [Hack #25] ) is much the same as the visual system's expectation that a shadow is reporting 3D shape correctly. While we could think of the visual system as entirely in the brain, really we should include the eyes, the head, the body, and the environment as components in this big, messy, densely connected human visual processing system, all of which report their conclusions into the mix.

And somehow, in all of this, the visual perception we know and love somehow springs into existence. There doesn't seem to be a single place where all this visual processing is reassembled, no internal television screen that we watch (and even if there were, who would watch it?). It's distributed over the whole visual system, and over the environment too. Not just a picture at the retina, after all.

Hack 12. Build Your Own Sensory Homunculus

All abilities are skills; practice something and your brain will devote more resources to it.

The sensory homunculus looks like a person, but swollen and out of all proportion. It has hands as big as its head; huge eyes, lips, ears, and nose; and skinny arms and legs. What kind of person is it? It's you, the person in your head. Have a look at the sensory homunculus first, then make your own.

1.13.1. In Action

You can play around with Jaakko Hakulinen's homunculus applet (http://www.cs.uta.fi/~jh/homunculus.html; Java) to see where different bits of the body are represented in the sensory and motor cortex.

This is the person inside your head. Each part of the body has been scaled according to how much of your sensory cortex is devoted to it. The area of cortex responsible for processing touch sensations is the somatosensory cortex. It lives in the parietal lobe, further toward the back of the head than the motor cortex, running alongside it from the top of the head down each side of the brain. Areas for processing neighboring body parts are generally next to each other in the cortex, although this isn't always possible because of the constraints of mapping the 3D surface of your skin to a 2D map. The area representing your feet is next to the area representing your genitals, for example (the genital representation is at the very top of the somatosensory cortex, inside the groove between the two hemispheres).

The applet lets you compare the motor and sensory maps. The motor map is how body parts are represented for movement, rather than sensation. Although there are some differences, they're pretty similar. Using the applet, when you click on a part of the little man, the corresponding part of the brain above lights up. The half of the man on the left is scaled according to the representation of the body in the primary motor cortex, and the half on the right is scaled to represent the somatosensory cortex. If you click on a brain section or body part, you can toggle shading and the display of the percentage of sensory or motor representation commanded by that body part. The picture of the man is scaled, too, according to how much cortex each part corresponds to. That's why the hands are so much larger than the torso.

Having seen this figure, you can see the relative amount of your own somatosensory cortex devoted to each body part by measuring your touch resolution. To do this, you'll need a willing friend to help you perform the two-point discrimination test.

Ask your friend to get two pointy objectstwo pencils will doand touch one of your palms with both of the points, a couple of inches apart. Look away so you can't see him doing it. You'll be able to tell there are two points there. Now get your friend to touch with only one pencilyou'll be able to tell you're being touched with just one. The trick now is for him to continue touching your palm with the pencils, sometimes with both and sometimes with just one, moving the tips ever closer together each time. At a certain point, you won't be able to tell how many pencils he's using. In the center of your palm, you should be able to discriminate between two points a millimeter or so apart. At the base of your thumb, you've a few millimeters of resolution.

Now try the same on your backyour two-point discrimination will be about 4 or 5 centimeters.

To draw a homunculus from these measurements, divide the actual width of your body part by the two-point discrimination to get the size of each part of the figure.

My back's about 35 centimeters across, so my homunculus should have a back that's 9 units wide (35 divided by 4 centimeters, approximately). Then the palms should be 45 units across (my palm is 9 centimeters across; divide that by 2 millimeters to get 45 units). Calculating in units like this will give you the correct scalesthe hand in my drawing will be five times as wide as the back.

That's only two parts of your body. To make a homunculus like the one in Hakulinen's applet (or, better, the London Natural History Museum's sensory homunculus model: http://owen.nhm.ac.uk/piclib/www/image.php?img=87494&cat=6), you'll also need measurements all over your face, your limbs, your feet, fingers, belly, and the rest. You'll need to find a fairly close friend for this experiment, I'd imagine.

1.13.2. How It Works

The way the brain deals with different tactile sensations is the way it deals with many different kinds of input. Within the region of the brain that deals with that kind of input is a surface over which different values of that input are processeddifferent values correspond to different actual locations in physical space. In the case of sensations, the body parts are represented in different parts of the somatosensory cortex: the brain has a somatotopic (body-oriented) map. In hearing, different tones activate different parts of the auditory cortex: it has a tonotopic map. The same thing happens in the visual system, with much of the visual cortex being organized in terms of feature maps comprised of neurons responsible for representing those features, ordered by where the features are in visual space.

Maps mean that qualities of stimuli can be represented continuously. This becomes important when you consider that the evidence for each qualityin other words, the rate at which the neurons in that part of the map are firingis noisy, and it isn't the absolute value of neural firing that is used to calculate which is the correct value but the relative value. (See [Hack #25] on the motion aftereffect for an example of this in action.)

The more cells the brain dedicates to building the map representing a sense or motor skill, the more sensitive we are in discriminating differences in that type of input or in controlling output. With practice, changes in our representational maps can become permanent.

Brain scanning of musicians has shown that they have larger cortical representations of the body parts they use to play their instruments in their sensory areasmore neurons devoted to finger movements among guitarists, more neurons devoted to lips among trombonists. Musicians' auditory maps of "tone-space" are larger, with neurons more finely tuned to detecting differences in sounds,¹ and orchestra conductors are better at detecting where a sound among a stream of other sounds is coming from.

It's not surprising that musicians are good at these things, but the neuroimaging evidence shows that practice alters the very maps our brains use to understand the world. This explains why small differences are invisible to beginners, but stark to experts. It also offers a hopeful message to the rest of us: all abilities are skills, if you practice them, your brain will get the message and devote more resources to them.

1.13.3. End Note

1. Münte, T. F., Altenmüller, E., & Jäncke, L. (2002). The musician's brain as a model for neuroplasticity. Nature Neuroscience Reviews, 3, 473-478. (This is a review paper rather than an original research report.)

1.13.4. See Also

• Pantev, C., Oostenveld, R., Engelien, A., Ross, B., Roberts, L. E., & Hoke, M. (1998). Increased auditory cortical representation in musicians. Nature, 392, 811-814.

• Pleger B., Dinse, H. R., Ragert, P., Schwenkreis, P., Malin, J. P., & Tegenthoff, M. (2001). Shifts in cortical representations predict human discrimination improvement. Proceedings of the National Academy of Sciences of the USA, 98, 12255-12260.

Hack 11. Why People Don't Work Like Elevator Buttons

More intense signals cause faster reaction times, but there are diminishing returns: as a stimulus grows in intensity, eventually the reaction speed can't get any better. The formula that relates intensity and reaction speed is Pieron's Law.

It's a common illusion that if you are in a hurry for the elevator you can make it come quicker by pressing the button harder. Or more often. Or all the buttons at once. It somehow feels as if it ought to work, although of course we know it doesn't. Either the elevator has heard you, or it hasn't. How loud you call doesn't make any difference to how long it'll take to arrive.

But then elevators aren't like people. People do respond quicker to more stimulation, even on the most fundamental level. We press the brake quicker for brighter stoplights, jump higher at louder bangs. And it's because we all do this that we all fall so easily into thinking that things, including elevators, should behave the same way.

1.12.1. In Action

Give someone this simple task: she must sit in front of a screen and press a button as quickly as she can as soon as she sees a light flash on. If people were like elevators, the time it takes to press the button wouldn't be affected by the brightness of the light or the number of lights.

But people aren't like elevators and we respond quicker to brighter lights; in fact, the relationship between the physical intensity of the light and the average speed of response follows a precise mathematical form. This form is captured by an equation called Pieron's Law. Pieron's Law says that the time to respond to a stimulus is related to the stimulus intensity by the formula:

Reaction Time
R0 + kI-b

Reaction Time is the time between the stimulus appearing and you responding. I is the physical intensity of the signal. R₀ is the minimum time for any response, the asymptotic value representing all the components of the reaction time that don't vary, such as the time for light to reach your eye. k and b are constants that vary depending on the exact setup and the particular person involved. But whatever the setup and whoever the person, graphically the equation .

1.12.2. How It Works

In fact, Pieron's Law holds for the brightness of light, the loudness of sound, and even the strength of taste.¹ It says something fundamental about how we process signals and make decisionsthe physical nature of a stimulus carries through the whole system to affect the nature of the response. We are not binary systems! The actual number of photons of light or the amplitude of the sound waves that triggers us to respond influences how we respond. In fact, as well as affecting response time, the physical intensity of the stimulus also affects response force as well (e.g., how hard we press the button).

A consequence of the form of Pieron's Law is that increases in speed are easy for low-intensity stimuli and get harder as the stimulus gains more intensity. It follows a log scale, like a lot of things in psychophysics. The converse is also true: for quick reaction times, it's easier to slow people down than to speed them up.

Pieron's Law probably results because of the fundamental way the decisions have to be made with uncertain information. Although it might be clear to you that the light is either there or not, that's only because your brain has done the work of removing the uncertainty for you. And on a neural level, everything is uncertain because neural signals always have noise in them.

So as you wait for light to appear, your neuronal decision-making hardware is inspecting noisy inputs and trying to decide if there is enough evidence to say "Yes, it's there!" Looking at it like this, your response time is the time to collect enough neural evidence that something has really appeared. This is why Pieron's Law applies; more intense stimuli provide more evidence, and the way in which they provide more evidence results in the equation shown earlier.

To see why, think of it like this: Pieron's Law is a way of saying that the response time improves but at a decreasing rate, as the intensity (i.e., the rate at which evidence accumulates) increases. Try this analogy: stimulus intensity is your daily wage and making a response is buying a $900 holiday. If you get paid $10 a day, it'll take 90 days to get the money for the holiday. If you get a raise of $5, you could afford the holiday in 60 days30 days sooner. If you got two $5 raises, you'd be able to afford the holiday in 45 daysonly 15 days sooner than how long it would take with just one $5 raise. The time until you can afford a holiday gets shorter as your wage goes up, but it gets shorter more slowly, and if you do the math it turns out to be an example of Pieron's Law.

1.12.3. End Note

1. Pins, D., & Bonnet, C. (1996). On the relation between stimulus intensity and processing time: Pieron's law and choice reaction time. Perception & Psychophysics, 58(3), 390-400.

1.12.4. See Also

• Stafford, T., & Gurney, K. G. (in press). The role of response mechanisms in determining reaction time performance: Pieron's law revisited. Psychonomic Bulletin & Review (in press).

• Luce, R. D. (1986). Response Times: Their Role in Inferring Elementary Mental Organisation. New York: Clarendon Press. An essential one stop for all you need to know about modeling reaction times.

• Pieron, H. (1952). The Sensations: Their Functions, Processes and Mechanisms. London: Frederick Muller Ltd. The book in which Pieron first proposed his law.

Hack 10. Detect the Effect of Cognitive Function on Cerebral Blood Flow

When you think really hard, your heart rate noticeably increases.

The brain requires approximately 20% of the oxygen in the body, even during times of rest. Like the other organs in our body, our brain needs more glucose, oxygen, and other essential nutrients as it takes on more work. Many of the scanning technologies that aim to measure aspects of brain function take advantage of this. Functional magnetic resonance imaging (fMRI) [Hack #4] benefits from the fact that oxygenated blood produces slightly different electromagnetic signals when exposed to strong magnetic fields than deoxygenated blood and that oxygenated blood is more concentrated in active brain areas. Positron emission tomography (PET) [Hack #3] involves being injected with weakly radioactive glucose and reading the subsequent signals from the most active, glucose-hungry areas of the brain.

A technology called transcranial Doppler sonography takes a different approach and measures blood flow through veins and arteries. It takes advantage of the fact that the pitch of reflected ultrasound will be altered in proportion to the rate of flow and has been used to measure moment-to-moment changes in blood supply to the brain. It has been found to be particularly useful in making comparisons between different mental tasks. However, even without transcranial Doppler sonography, you can measure the effect of increased brain activity on blood flow by measuring the pulse.

1.11.1. In Action

For this exercise you will need to get someone to measure your carotid pulse, taken from either side of the front of the neck, just below the angle of the jaw. It is important that only very light pressure be useda couple of fingertips pressed lightly to the neck, next to the windpipe, should enable your friend to feel your pulse with little trouble.

First you need to take a measure of a resting pulse. Sit down and relax for a few minutes. When you are calm, ask your friend to count your pulse for 60 seconds. During this time, close your eyes and try to empty your mind.

With a baseline established, ask your friend to measure your pulse for a second time, using exactly the same method. This time, however, try and think of as many species of animals as you can. Keeping still and with your eyes closed, think hard, and if you get stuck, try thinking up a new strategy to give you some more ideas.

During the second session, your pulse rate is likely to increase as your brain requires more glucose and oxygen to complete its task. Just how much increase you'll see varies from person to person.

1.11.2. How It Works

Thinking of as many animals as possible is a type of verbal fluency task, testing how easily you can come up with words. To complete the task successfully, you needed to be able to coordinate various cognitive skills, for example, searching your memory for category examples, generating and using strategies to think up more names (perhaps you thought about walking through the jungle or animals from your local area) and checking you were not repeating yourself.

Neuropsychologists often use this task to test the executive system, the notional system that allows us to coordinate mental tasks to solve problems and work toward a goal, skills that you were using to think up examples of animals. After brain injury (particularly to the frontal cortex), this system can break down, and the verbal fluency task can be one of the tests used to assess the function of this system.

Research using PET scanning has shown similar verbal fluency tasks use a significant amount of brain resources and large areas of the cortex, particularly the frontal, temporal, and parietal areas.¹

Interestingly, in this study people who did best used less blood glucose than people who did not perform as well. You can examine this relationship yourself by trying the earlier exercise on a number of people. Do the people who do best show a slightly lower pulse than others? In these cases, high performers seem to be using their brain more efficiently, rather than simply using more brain resources.

Although measuring the carotid pulse is a fairly crude measure of brain activity compared to PET scanning, it is still a good indirect measure of brain activity for this type of high-demand mental task, as the carotid arteries supply both the middle and anterior cerebral arteries. They supply blood to most major parts of the cortex, including the frontal, temporal, parietal, and occipital areas, and so would be important in supplying the needed glucose and oxygen as your brain kicks into gear.

One problem with PET scanning is that, although it can localize activity to certain brain areas, it has poor temporal resolution, meaning it is not very good at detecting quick changes in the rate of blood flow. In contrast, transcranial Doppler sonography can detect differences in blood flow over very short periods of time (milliseconds). Frauenfelder and colleagues used this technique to measure blood flow through the middle and anterior cerebral arteries while participants were completing tasks that are known to need similar cognitive skills as the verbal fluency exercise.² They found that the rate of blood flow changed second by second, depending on exactly which part of the task the participant was tackling. While brain scanning can provide important information about which areas of the brain are involved in completing a mental activity, sometimes measuring something as simple as blood flow can fill in the missing pieces.

1.11.3. End Notes

1. Parks, R. W., Loewenstein, D. A., Dodrill, K. L., Barker, W. W., Yoshii, F., Chang, J. Y., Emran, A., Apicella, A., Sheramata, W. A., & Duara, R. (1988). Cerebral metabolic effects of a verbal fluency test: A PET scan study. Journal of Clinical and Experimental Neuropsychology, 10(5), 565-575.

2. Schuepbach, D., Merlo, M. C., Goenner, F., Staikov, I., Mattle, H. P., Dierks, T., & Brenner, H. D. (2002). Cerebral hemodynamic response induced by the Tower of Hanoi puzzle and the Wisconsin card sorting test. Neuropsychologia, 40(1), 39-53.

Hack 9. The Neuron

There's a veritable electrical storm going on inside your head: 100 billion brain cells firing electrical signals at one another are responsible for your every thought and action.

A neuron, a.k.a. nerve cell or brain cell, is a specialized cell that sends an electrical impulse out along fibers connecting it, in turn, to other neurons. These guys are the wires of your very own personal circuitry.

What follows is a simplistic description of the general features of nerve cells, whether they are found sending signals from your senses to your brain, from your brain to your muscles, or to and from other nerve cells. It's this last class, the kind that people most likely mean when they say "neurons," that we are most interested in here. (All nerve cells, however, share a common basic design.)

Don't for a second think that the general structure we're describing here is the end of the story. The elegance and complexity of neuron design is staggering, a complex interplay of structure and noise; of electricity, chemistry, and biology; of spatial and dynamic interactions that result in the kind of information processing that cannot be defined using simple rules.1 For just a glimpse at the complexity of neuron structure, you may want to start with this free chapter on nerve cells from the textbook Molecular Cell Biology by Harvey Lodish, Arnold Berk, Lawrence S. Zipursky, Paul Matsudaira, David Baltimore, and James Darnell and published by W. H. Freeman (http://www.ncbi.nlm.nih.gov/books/bv.fcgi?call=bv.View..ShowSection&rid=mcb.chapter.6074), but any advanced cell biology or neuroscience textbook will do to give you an idea of what you're missing here.

The neuron is made up of a cell body with long offshootsthese can be very long (the whole length of the neck, for some neurons in the giraffe, for example) or very short (i.e., reaching only to the neighboring cell, scant millimeters away). Signals pass only one way along a neuron. The offshoots receiving incoming transmissions are called dendrites. The outgoing end, which is typically longer, is called the axon. In most cases there's only one, long, axon, which branches at the tip as it connects to other neuronsup to 10,000 of them. The junction where the axon of one cell meets the dendrites of another is called the synapse. Chemicals, called neurotransmitters, are used to get the signal across the synaptic gap. Each neuron will release only one kind of neurotransmitter, although it may have receptors for many different kinds. The arrival of the electric signal at the end of the axon triggers the release of stores of the neurotransmitter that move across the gap (it's very small, after all) and bind to receptor sites on the other side, places on the neuron that are tuned to join with this specific type of chemical.

Whereas the signal between neurons uses neurotransmitters, internally it's electrical. The electrical signal is sent along the neuron in the form of an action potential.² This is what we mean when we say impulses, signals, spikes, or refer, in brain imaging speak, to the firing or lighting up of brain areas (because this is what activity looks like on the pictures that are made). Action potentials are the fundamental unit of information in the brain, the universal currency of the neural market.

The two most important computational features are as follows:

• They are binary. A neuron either fires or doesn't, and each time it fires, the signal is the same size (there's more on this later). Binary signals stop the message from becoming diluted as neurons communicate with one another over distances that are massive compared to the molecular scale on which they operate.

• Neurons encode information in the rate at which they send signals, not in the size of the signals they send. The signals are always the same size, information encoded in the frequency at which signals are sent. A stronger signal is indicated by a higher frequency of spikes, not larger single spikes. This is called rate coding.

Together these two features mean that the real language of the brain is not just a matter of spikes (signals sent by neurons), but spikes in time.

Whether or not a new spike, or impulse, is generated by the postsynaptic neuron (the one on the receiving side of the synapse) is affected by the following interwoven factors:

• The amount of neurotransmitter released

• The interaction with other neurotransmitters released by other neurons

• How near they are and how close together in space and time

• In what order they release their neurotransmitters

All of this short-term information is affected by any previous history of interaction between these two neuronstimes one has caused the other to fire and when they have both fired at the same time for independent reasonsand slightly adjusts the probability of interaction happening again.³

Spikes happen pretty often: up to once every 2 milliseconds at the maximum rate of the fastest-firing cells (in the auditory system; see Chapter 4 for more on that). Although the average rate of firing is responsive to the information being represented and transmitted in the brain, the actual timing of individual spikes is unpredictable. The brain seems to have evolved an internal communication system that has noise added to only one aspect of the information it transmitsthe timing, but not the size of the signals transmitted. Noise is a property of any biological system, so it's not surprising that it persists even in our most complex organ. It could very well also be the case that the noise [Hack #33] is playing some useful role in the information processing the brain does.

After the neurotransmitter has carried (or not carried, as the case may be) the signal across the synaptic gap, it's then broken down by specialized enzymes and reabsorbed to be released again when the next signal comes along. Many drugs work by affecting the rate and quantity of particular neurotransmitters released and the speed at which they are broken down and reabsorbed.

Hacks such as [Hack #11] and [Hack #26] show some of the other consequences for psychology of using neurons to do the work. Two good introductions to how neurons combine on a large scale can be found at http://www.foresight.gov.uk/cognitive.html. This is a British government Department of Trade and Industry project that aimed to get neuroscientists and computer scientists to collaborate in producing reviews of recent advances in their fields and summarize the implications for the development of artificial cognitive systems.

1.10.1. End Notes

1. Gurney, K. N. (2001). Information processing in dendrites II. Information theoretic complexity. Neural Networks, 14, 1005-1022.

2. You can start finding out details of the delicate electrochemical dance that allows the transmission of these binary electrical signals on the pages about action potentials that are part of a series of lecture notes on human physiology (http://members.aol.com/Bio50/LecNotes/lecnot11.html), the Neuroscience for Kids site (http://faculty.washington.edu/chudler/ap.html), and The Brain from Top to Bottom project (http://www.thebrain.mcgill.ca/flash/a/a_01/a_01_m/a_01_m_fon/a_01_m_fon.html).

3. But this is another storya story called learning.

1.10.2. See Also

• How neurons are born, develop, and die is another interesting story and one that we're not covering here. These notes from the National Institutes of Health are a good introduction: http://www.ninds.nih.gov/health_and_medical/pubs/NINDS_Neuron.htm.

• Neurons actually make up less than a tenth of the cells in the brain. The other 90-98%, by number, are glial cells, which are involved in development and maintenancethe sysadmins of the brain. Recent research also suggests that they play more of a role in information processing than was previously thought. You can read about this in the cover story from the April 2004 edition of Scientific American (volume 290 #4), "The Other Half of the Brain."

Hack 8. Tour the Cortex and the Four Lobes

The forebrain, the classic image of the brain we know from pictures, is the part of the brain that defines human uniqueness. It consists of four lobes and a thin layer on the surface called the cortex.

When you look at pictures of the human brain, the main thing you see is the rounded, wrinkled bulk of the brain. This is the cerebrum, and it caps off the rest of the brain and central nervous system [Hack #7].

To find your way around the cerebrum, you need to know only a few things. It's divided into two hemispheres, left and right. It's also divided into four lobes (large areas demarcated by particularly deep wrinkles). The wrinkles you can see on the outside are actually folds: the cerebrum is a very large folded-up surface, which is why it's so deep. Unfolded, this surfacethe cerebral cortexwould be about 1.5 m² (a square roughly 50 inches on the side), and between 2 and 4 mm deep. It's not thick, but there's a lot of it and this is where all the work takes place. The outermost part, the top of the surface, is gray matter, the actual neurons themselves. Under a few layers of these is the white matter, the fibers connecting the neurons together. The cortex is special because it's mainly where our high-level, human functions take place. It's here that information is integrated and combined from the other regions of the brain and used to modulate more basic functions elsewhere in the brain. The folds exist to allow many more neurons and connections than other animals have in a similar size area.

1.9.1. Cerebral Lobes

The four cerebral lobes generally perform certain classes of function.

You can cover the frontal lobe if you put your palms on your forehead with your fingers pointing up. It's heavily involved in planning, socializing, language, and general control and supervision of the rest of the brain.

The parietal lobe is at the top and back of your head, and if you lock your fingers together and hook your hands over the top back, that's it covered there. It deals a lot with your senses, combining information and representing your body and movements. The object recognition module for visual processing [Hack #13] is located here.

You can put your hands on only the ends of the temporal lobeit's right behind the ears. It sits behind the frontal lobe and underneath the parietal lobe and curls up the underside of the cerebrum. Unsurprisingly, auditory processing occurs here. It deals with language too (like verbal memory), and the left hemisphere is specialized for this (non-linguistic sound is on the right). The curled-up ends of the temporal lobe join into the limbic system at the hippocampus and are involved in long-term memory formation.

Finally, there's the occipital lobe, right at the back of the brain, about midway down your head. This is the smallest lobe of the cerebrum and is where the visual cortex is located.

The two hemispheres are joined together by another structure buried underneath the lobes, called the corpus callosum. It's the largest bundle of nerve fibers in the whole nervous system. While sensory information, such as vision, is divided across the two hemispheres of the brain, the corpus callosum brings the sides back together. It's heavily coated in a fatty substance called myelin, which speeds electrical conduction along nerve cells and is so efficient that the two sides of the visual cortex (for example) operate together almost as if they're adjacent. Not bad considering the corpus callosum is connecting together brain areas a few inches apart when the cells are usually separated by only a millimeter or two.

1.9.2. Cerebral Cortex

The cortex, the surface of these lobes, is divided into areas performing different functions. This isn't exact, of course, and they're highly interconnected and draw information from one another, but more or less there are small areas of the surface that perform edge detection for visual information or detect tools as opposed to animate objects in much higher-level areas of the brain.

How these areas are identified is covered in the various brain imaging and methods hacks earlier in this chapter.

The sensory areas of the cortex are characterized by maps, representations of the information that comes in from the senses. It's called a map because continous variations in the value of inputs are represented by continuous shifts in distance between where they are processed in the cortical space. In the visual cortex, visual space is preserved on the retina. This spatial map is retained for each stage of early visual processing. This means that if two things are next to each other out there in the world they will, at least initially, be processed by contiguous areas of the visual cortex. This is just like when a visual image is stored on photographic negative but unlike when a visual image is stored in a JPEG image file. You can't automatically point to two adjoining parts of the JPEG file and be certain that they will appear next to each other in the image. With a photographic film and with the visual cortex, you can. Similarly, the auditory cortex creates maps of what you're hearing, but as well as organizing things according to where they appear in space, it also has maps that use frequency of the sound as the coordinate frame (i.e., they are tonotopic). And there's an actual map in physical space, on the cortex, of the whole body surface too, called the sensory homunculus [Hack #12] . You can tell how much importance the brain gives to areas of the map, comparatively, by looking at how large they are. The middle of the map of the primary visual cortex corresponds with the fovea in the retina, which is extremely high resolution. It's as large as the rest of the visual map put together.

When the cortex is discussed, that means the function in question is highly integrated with the rest of the brain. When we consider what really makes us human and where consciousness is, it isn't solely the cortex: the rest of the brain has changed function in humans, we have human bodies and nervous systems, and we exist within environments that our brains reflect in their adaptations. But it's definitely mostly the cortex. You are here.