Saccades, Perception, and Shmups

During a recent episode of Mark_MSX’s podcast — the Electric Underground — I spoke about how saccades may interfere with our ability to play shmups at a high level. You can listen to that here on Mark’s podcast.

I’d like to add more to the discussion of how vision works, with a focus on saccades and perception. Our eyes process a significant amount of information at any given time while playing a shmup. If we learn more about how that takes place, perhaps we can compensate and play our games faster and with more accuracy.

Buckle up for a biology lesson.

A saccade is a rapid-eye movement that reorients the center of your vision to a new target. Our vision is not a stream of visual data. Rather, our vision is a string of slices of individual images, akin to a webcam or a film reel. We see a collage of still images mixed with snippets of blurred motion and depth information. This unsorted data is crammed into our brain-meat and translated into an approximation of the outside world.

Saccades are an essential part of how humans visualize the world. Our eyes are not passive cameras or windows that the brain looks through. Quite the opposite: the brain is constantly readjusting our eyes and snapping to new targets in order to best understand the physical world surrounding us. This is especially true when we are attempting to perceive an object in motion. Our brain is an active player in how we see things. We move our eyes involuntarily due to these underlying biological mechanics at play.

This section offers a brief overview on the four types of eye movements, but it is the below piece on saccades that relates directly to our topic:

After the onset of a target for a saccade (in this example, the stimulus was the movement of an already fixated target), it takes about 200 ms for eye movement to begin. During this delay, the position of the target with respect to the fovea is computed (that is, how far the eye has to move), and the difference between the initial and intended position, or “motor error” (see Chapter 19), is converted into a motor command that activates the extraocular muscles to move the eyes the correct distance in the appropriate direction.

Saccadic eye movements are said to be ballistic because the saccade-generating system cannot respond to subsequent changes in the position of the target during the course of the eye movement. If the target moves again during this time (which is on the order of 15–100 ms), the saccade will miss the target, and a second saccade must be made to correct the error.

An (approximated) 200ms delay between noticing movement and the start of a saccade seems rather important. If an error occurs (as in, your brain miscalculated the final position of the visual target) or if the target changes trajectory, this causes an additional 15ms – 100ms delay to make secondary adjustments. These smaller adjustments are ‘corrective saccades’.

We need more information before reaching a conclusion. I’ll take several excerpts from the succinct article here:

Saccades direct the fovea onto an object or region of interest which enables subsequent high-acuity detailed visual analysis at that location. In normal viewing, several saccades are made each second and their destinations are selected by cognitive brain process without any awareness being involved.

“High-acuity detailed visual analysis.” Huh, that seems to match other research indicating that foveal vision is responsible for symbol recognition. Quite important for navigating dense bullet patterns. The next part is even more important:

Vision is dependent upon the information taken in during fixation pauses between saccades: no useful visual information is taken in while the eyes are making a saccadic movement.

In other words, your eyes do not “see” during a saccade. Considering what we know about the delay in saccadic eye movement and the 15-100ms penalty for misjudging a target, this seems important. More saccades = more time spent with no useful visual information taken in.

Move your eyes more and you see less.

This is fundamental.

After all, your screen is outputting a fixed amount of frames. So if your eyes are blanking during all those saccades as you dart your eye around the TV screen, you are cutting out valuable visual information, replacing it instead with assumptions (discussed later).

Each frame in a 60-frames-per-second shmup takes 16.67ms. Do the math (or bust out the Texas Instruments if that’s too challenging) on how many frames have gone by during your 200ms response delay prior to a saccade.

The Remote Distractor Effect also appears to hinder the shmup player:

The remote distractor effect is a related automatic effect on saccadic latencies found when a visual onset occurs elsewhere in the visual field simultaneously with the appearance of a saccade target (Walker, Deubel, Schneider, & Findlay, 1997 Figure 4). Such an occurrence results in a prolongation of saccadic latency whether or not the location of the target is completely predictable. The timing of the distractor onset has been shown to modulate the magnitude of the RDE with distractors presented within ±50ms of the target producing the greatest effect.

So if there are other items that appear in your field of view while attempting to track a target during a saccade, there is an increase in delay. Interesting.

What about memorization? Surely if you’ve memorized a bullet pattern it will be easier to perceive its details.

The memory-guided saccade is similar but in this case the target is only flashed briefly so that saccades are directed to a remembered location. Such saccades tend to show a decrease in accuracy in normal individuals[…]

So when relying on memory of where to orient your eyes, your accuracy goes down. I wonder if that is to blame for all those times I could’ve sworn I was moving to a safe area, only to be snagged by a previously-unnoticed bullet.

Our current body of scientific research indicates that our vision is highly fallible when it comes to perceiving discrete details, especially during eye-movement. But if my eyes are constantly “shutting off” during saccades, why don’t I see flashes of darkness?

Good question. That brings us to the other half of the matter: perception.

That’s what your brain does with the raw visual information delivered via the optic nerve from your eyes. It perceives. The data is translated, in a manner of speaking. We all learned in school about how the image in our retinas is flipped, right? The brain must translate the visual data before we actually “perceive” it.

Sounds to me like some input lag is hiding inside our own skull.

Unfortunately, there are even more problems introduced by human perception. Our perception, it turns out, is not at all like a videocamera recording visual information. Instead, our brain cheats.

Remember, our brain-meat is interested in identifying patterns as quickly as possible. So, it will often cheat to get to an answer faster at the cost of accuracy. That’s why things can end up looking different in poor visual conditions (fog, darkness), because your brain is leaping to conclusions.

More specifically, it is filling in blank spots with false data. The visual information you “see” during a saccade isn’t actually there. It’s a predictive image made by your brain. This avoids any sort of stutter in your stream of vision. It also isn’t exactly accurate. It’s a good guess (based on the last fixed image) but it is only a guess.

Have you ever been surprised by a bullet that you just didn’t see? Of course. There’s a chance you were tricked by your own brain as it filled in information based on assumptions.

There’s also the matter of how perception can be involuntarily altered if you are fixed on a certain goal.

See how you do on the challenge below:

Levin and Simons (and later Simons, Franconeri, and Reimer) conducted a series of tests to illustrate how our perception can be easily tricked. If you are interested in learning more about lapses in perception, follow the links below:

The Door Study

Movie Perception Test

Spot the Change

To sum: the biological and psychological characteristics of human vision impact our ability to play shmups at high levels of skill. An understanding of the characteristics of human vision and perception allows one to adjust accordingly, resulting in improved response times, awareness, and acuity.

Can we do anything about this? Possibly.

Heartbeat has been tied to the rate of saccades. Heartbeat goes up and saccades go up, too. An increase in saccade latency has been linked to elevated stress. It seems the practical advice here would be to keep calm and keep staring forward.

In some of our earlier documentation, there is reference to another facet of our perception, the smooth pursuit system.

Smooth pursuit movements are much slower tracking movements of the eyes designed to keep a moving stimulus on the fovea. Such movements are under voluntary control in the sense that the observer can choose whether or not to track a moving stimulus (Figure 20.5). (Saccades can also be voluntary, but are also made unconsciously.) Surprisingly, however, only highly trained observers can make a smooth pursuit movement in the absence of a moving target. Most people who try to move their eyes in a smooth fashion without a moving target simply make a saccade.

The next section offers more insight. Replace “stripes in a rotating cylinder” with “pink bullet” and you can see why this matters:

The smooth pursuit system can be tested by placing a subject inside a rotating cylinder with vertical stripes. (In practice, the subject is more often seated in front of a screen on which a series of horizontally moving vertical bars is presented to conduct this “optokinetic test.”) The eyes automatically follow a stripe until they reach the end of their excursion. There is then a quick saccade in the direction opposite to the movement, followed once again by smooth pursuit of a stripe. This alternating slow and fast movement of the eyes in response to such stimuli is called optokinetic nystagmus. Optokinetic nystagmus is a normal reflexive response of the eyes in response to large-scale movements of the visual scene.

Perhaps, based on this information, it is possible to train the eye to follow a moving object without saccades. More experimentation to follow.