BarajasCaldwell
Introduction
Our eyes are constantly moving, even when observing a stationary object. In fact, there are two distinct types of movements: larger, sporadic scanning, called saccades; and much smaller, high frequency movements called ocular drift. Past experiments have suggested that saccades are responsible for preventing image fading on our retina— that these movements "refresh" our visual system so that we have continual neural responses to static scenes. This result has largely satisfied inquiries of the purpose of eye movements for the visual system, and many in the scientific community assume both types of eye movements serve the single purpose of preventing fading. However, Michele Rucci and Jonothan D. Victor argue that this is an oversimplification and that in particular, ocular drift serves the more profound role of amplifying higher spatial frequencies on the retina to improve visual resolution. Our work throughout this project centers on assessing this hypothesis.
Background
Literature focuses on ocular drift, which generally occurs in the period between the larger saccade movements, and is thus also referred to as "fixational" eye movement. Rucci and Victor estimate ocular drift has a mean speed of 50 minutes of arc per second, with a distribution of gaze position that disperses as time passes. An important idea of theirs is that these small motions cause fluctuations in luminance on the retina that would not be possible with a motionless eye. Since temporal fluctuations result in higher spectral power at the frequencies of the fluctuations, and since higher spectral power is associated with amplification of an image by the visual system, it follows that fixational eye movements can lead to visual amplification.
The above figure show Rucci and Victor's qualitative argument. The left plot shows higher modulations in luminance for faster eye movements, and the right plot shows their conclusion that retinal amplification increases with spatial frequency. In our project we seek to support these plots with a more quantitative foundation. An interesting result of theirs is that amplification stops increasing after about 30 cycles per degree. They note that this is around the spatial frequency that coincides with the maximum resolution of the retina itself. They also note that "natural scenes" generally have lower contrast for higher spatial frequencies and higher contrast for lower spatial frequencies. Accounting for this, they argue that amplification of higher spatial frequencies by ocular drift equalizes the power across all spatial frequencies on the retina.
Methods
The primary simulation tool was ISETBIO, which is a custom-built MATLAB library oriented towards simulating the human visual system and several processes that occur within the retina. In order to simulate our fixational eye movements, we utilized the cone mosaic interface. This tool provided various relevant statistics, and the graphical interface allowed us to visualize both absorptions and and photocurrent at the retina – either as a time series, an updating movie, or an image of average values. The figure on the left is one such figure displaying average absorption for the cone located at position (47,47) from the 47x47 cone mosaic. The figure on the left displays the cone types of the same mosaic.
Objectives
The primary objective was to identify and characterize the improvements caused by the movements in the eye. This was executed by simulating scenarios with and without a simple horizontal eye movement, and plot the photocurrent and absorption measurements (as well as the ratio from scenarios with eye movements to scenarios to scenarios "without" movements) to identify where improvements were being made. The simple eye movement path used was a sweep sinusoidally from the origin to positive x-direction for a frequency of 3 cycles per degree and amplitude of 3 cone widths. It was also important to take the photocurrent time series and take the spectrum to perform frequency analysis, which we knew would reveal the more pertinent information. Additionally, categorizing cone types was part of the analysis to observe whether or not the cone type affected the response due to movements.
Next, we targeted the relationship between spatial frequency of the scene and absorption and photocurrent measurements. Once again, the frequency spectrum of the photocurrent served as the focus of this analysis given that spatial frequency would have the greatest impact in the frequency domain.
Finally, we aimed to characterize the relationship of between temporal frequency and amplitude of the eye movement, and the same measurements. This would reveal information about the mechanics of the fixational eye movements and how they are optimized for amplification. This is an area for extensive additional research, given the applications to both enhanced human vision and also computer vision.
Results
Absorption tests
Absorption levels were used as the first metric to observe differences between static and dynamic eye scenarios, and also between cone types. This was intended to reveal whether or not the eye was gathering more information with its small movements. To the contrary, the results revealed virtually no amplification or increased information with or without eye movements.
The figure below displays a colormap of the 47x47 cone mosaic and the ratio of total absorptions "with" movement to total absorptions "without" movement at each cone in the mosaic. As we see, the mosaic has fairly low variance, with a smattering of values no greater than 0.3 deviation from unity. This suggested that each cone has nearly identical absorption behavior, with or without movement.
When categorized by cone type, each of he L,M, and S type cones in the mosaic returned very similar results, with ratios within .002 of unity. The figure below illustrates this effect. These ratios represent the average ratio over all cones of each type.
Investigation of absorption levels and their lack of dependence on movements reinforced the prediction that frequency analysis would reveal the amplification / improvements being reaped by the eye's movement.
Photocurrent tests
In this step we examined the spectra of the photocurrent generated by the cone mosaic from the stimulus. With some simple simulations, we already saw some unexpected results. With eye movements restricted to the horizontal axis and varying sinusoidally over time, we modified the sine parameters (amplitude and frequency) so that during the simulation, the eye completed an integer number of cycles over the grating. Interestingly, no matter the parameters, the average magnitude spectrum of the photocurrent over the cone mosaic was peaked at the second fft bin (7.8125 Hz). The below plot shows values of this temporal frequency at ten different spatial frequencies, from 0 to 36 cycles per degree (scaled by the number of time samples). These temporal frequencies were found by averaging the spectra across the entire cone mosaic in the absence of noise, and with horizontal eye movements adjusted to move one cycle (bright then dark fringe) over the simulation at 4 cycles per degree. Although there is some attenuation over spatial frequency, this peak value is always very high.
This result may be related to the findings of Ehud Kaplan and Ethan Benardete that primate ganglion cells are most receptive to temporal frequencies around 8-10 Hz (1). Next, we looked at the amplification of the temporal photocurrent spectra with eye movements relative to spectra without eye movements. The plot below shows our results. We found the ratio of the spectra (between movements and no movements) at each cone on the mosaic for each of five spatial frequencies, and averaged these ratios. Our eye movements were constant over spatial frequency and tuned to move one cycle (bright then dark fringe) over the simulation at 30 degrees per cycle.
We then replicated the calculations above for each cone type individually, to determine whether different cones behaved differently. Using the same parameters, we generated the plots below.
Since the ratios are above unity for all spatial spectra, we see that eye movements definitely amplify photocurrent spectra relative to no eye movements. However, we do not see a clear relationship between increased spatial frequency and amplification, since the order of the height of the curves does not follow the order of spatial frequencies. Subsequently, we generated the same ratio plot, but isolating each type of cone individually. Below are the results for L, M, and S cones.
We see that the L and M cones follow a similar pattern to the cone mosaic as a whole, while the S cones produce steeper curves that are more closely varying. All individual cone types amplify the spectra when movements are present (since all curves are above unity), but it still certainly not clear for the cones that higher spatial frequencies are amplified more than lower spatial frequencies.
Conclusions
- Total absorption levels remain unaffected by the eye movements performed by the eye. This reinforces the idea that these movements are enhancing the amplification in frequency and not absolute value (for both absorption and photocurrent).
- It is clear that eye movements amplify temporal photocurrent spectra of a scene relative to observations of the scene without eye movements. Since the visual system is more responsive to fluctuating retinal signals than the average value of the signals, the presence of eye movements amplifies viewed scenes. A similar relationship would be expected for the spectra of absorptions as well, given the proportionality of absorptions and photocurrent.
- However, it is not clear whether there is a strong relationship between spatial frequency and retinal amplification. In fact, we saw that lower spatial frequencies were amplified more than higher spatial frequencies, which defied our predictions based on the ideas of Rucci and Victor.
- Individual cone types (L, M, and S) do not show a noticeable amplification of higher spatial frequencies than lower spatial frequencies, but do provide more spectral power with eye movements than without eye movements.
References
Rucci, Michele, and Jonathan D. Victor. "The Unsteady Eye: An Information-processing Stage, Not a Bug." Sciencedirect.com. N.p., Apr. 2016. Web. Nov.-Dec. 2016.
Kaplan, Ehud, and Ethan Benardete. "The Dynamics of Primate Retinal Ganglion Cells." Elsevier Science, n.d. Web. 16 Dec. 2016.
ISETBIO, a software developed by Professor Brain Wandell, et al., for simulating the human visual system
Appendix 1
NOTE: the following MATLAB scripts require ISETBIO to run.
Photocurrent Code (Variations on the following script were used to create the photocurrent plots on this Wiki page. )
Appendix 2
Gabriel focused on simulating photocurrent spectra and ratios. Trevor focused on simulating absorption data across the cone mosaic. Both of us collaborated extensively to determine which aspects of Rucci and Victor's arguments we would investigate, and on verifying each other's findings. We formulated both the presentation and this Wiki page together, reviewing each other's contributions. We greatly thank Professor Brain Wandell for mentoring us throughout this project and being indispensable in introducing this fascinating topic to us, as well as guiding us through ISETBIO.