ISETBIO Baseball Simulation Experiment

Introduction

Hitting a pitched baseball places extreme demands on the visual system. In the few hundred milliseconds between release and home plate, a batter must detect the ball, track its motion, and judge its future location well enough to decide whether and where to swing. Understanding what information is actually available at the level of the eye before any higher-level processing can provide useful insight into how difficult different pitches are to track and discriminate.

For this project, we explore how different baseball pitch trajectories drive cone responses over time in a simplified model of the batter’s eye. We start from rendered synthetic image sequences of pitched balls and use ISETCam to model the optical image at the retina under controlled field-of-view and display conditions. We then use ISETBio’s cone mosaic framework to compute noise-free cone absorptions over time as the ball travels from release to home plate, summarizing responses in foveal, mid-peripheral, and far-peripheral regions. Finally, we analyze these cone absorption sequences to see how well they do with pitch classification and understand their relationship to pitch characteristics.

Background

Prior work on baseball vision has shown that hitters operate under severe visual constraints. The ball is small relative to viewing distance, it travels quickly, and for most of its flight it remains in peripheral or near-peripheral vision. A key result from Shapiro et al. [1] demonstrated that when an object moves from peripheral vision into the fovea, the visual system can misinterpret the motion and produce an illusory “break.” This finding is often used to explain why hitters sometimes report a sudden directional change in pitches like curveballs even though the actual trajectory is smooth.

In addition to these perceptual factors, the underlying pitch physics are more complex than the standard Magnus-force explanation. Recent aerodynamic work has shown that the raised seams create asymmetric airflow patterns—seam-shifted wakes [2]. Importantly, simply changing seam orientation can alter the ball’s movement even when velocity, spin rate, and spin axis are held constant. Two pitches that would be identical in a simplified model can therefore diverge in flight purely due to seam geometry.

There is also evidence linking hitting performance to core visuomotor abilities such as tracking accuracy, predictive saccades, and eye–hand coordination [3]. This means that what the batter “sees” on any given pitch reflects a combination of the retinal input, the physical flight of the ball, and the batter’s gaze strategy.

Taken together, these findings motivate the central question of this project: given the perceptual distortions that can arise from the visual system itself, the aerodynamic differences introduced by seams, and the reliance on active gaze behavior, what does the signal actually look like at the level of the cone mosaic during a pitch? That is the specific gap this analysis aims to address.

Methods

Pitch Trajectory Simulation

To generate realistic ball-flight trajectories, I implemented a JavaScript port of the aerodynamic model developed by Alan Nathan (Nathan, University of Illinois), which predicts pitch trajectories given physical parameters including:

• backspin, sidespin, and gyro spin
• initial velocity vector
• pitcher release point
• drag coefficient and lift coefficient
• spin axis direction

Nathan’s model [4] numerically integrates the equations of motion under Magnus force, drag force, gravity and outputs 3D coordinates from release point to home plate. By embedding this model directly into my simulation backend, I was able to generate new pitch trajectories on-demand for arbitrary combinations of speed, spin rate, spin axis, while also altering seam orientation (for visual purposes only).

Seam-Orientation Impacts

A baseball’s seam orientation - which is the 3D rotation matrix that maps the ball’s spin axis to the stitched seam geometry — is a major contributor to pitch movement. For example, four-seam fastballs typically use a seam alignment where the ball’s spin axis is perpendicular to the horseshoe seams, maximizing Magnus lift. While seam orientation is known to have impacts on pitch movement, my focus on this project is not to estimate those effects, but rather attempt to measure the effects on retinal capabilities as it relates to a batter facing different types of pitches with different orientations, speeds, movement.

To study how seam orientation alters the retinal image, I represented each orientation as a 3×3 rotation matrix applied to the baseball texture. Examples include:

• canonical orientations for the following four pitch types:
  • 4-Seam Fastball
  • Curveball
  • Slider
  • Changeup
• Rotated seam orientations for each pitch type

During rendering, the baseball texture and normal map are rotated according to this matrix so that identical pitch trajectories with different seam matrices produce different appearance sequences, even when spin rate and velocity are held constant.

This setup allowed me to isolate how the orientation of the baseball’s seams while keeping all physical pitch attributes (velocity, spin rate, movement profile) constant — affects the cone-level retinal response. As an example, the figure below shows the three seam-orientation conditions used for the four-seam fastball: 0° (the standard 4-seam look), 45°, and 90°. These rotations don’t change the physics of the pitch in this simulation, but they do change what part of the seam pattern the batter sees.

Visual Rendering of the Spinning Ball Trajectory

I used Three.js to render synthetic image sequences of each pitch as it traveled along the simulated trajectory. The renderer produced a frame-by-frame animation of the ball’s position, orientation, and seam rotation over time. For each pitch type (4-seam fastball, slider, changeup, curveball), I ran 27 total simulations by perturbing seam orientation, spin rate, and velocity around their “normal” values.

For example, for the 4-seam fastball I used:

• Three seam orientations:
  • normal | rotated 45° | rotated 90°
• Three spin rates:
  • 3000 RPM (baseline) | +500 RPM | –500 RPM
• Three velocities:
  • 96 mph (baseline) | +3 mph | –3 mph

I repeated this structure for all four pitch types, using pitch-specific baseline values (e.g., a slider has a different default seam orientation and a baseline velocity closer to 87 mph rather than 96 mph).

Each simulation produced a little over 100 PNG frames showing the ball from roughly 55 feet out all the way to the strike zone and which was used in the retinal simulation stage.

Retinal Image Formation and Motion Blur (ISETCam)

The rendered sequences did not all contain the same number of frames (because different pitch types and velocities produced slightly different durations when exported). To allow direct, frame-aligned comparisons of cone absorptions across all conditions, each run was temporally normalized to 80 frames using the following resampling procedure.

For a pitch run with N_actual exported PNG frames, I generated a normalized index vector:

frameIdx = round(linspace(1, N_actual, 80))

which maps each of the 80 target frames to the closest corresponding source frame. This ensures that:

• every pitch is represented by exactly 80 time points
• early–mid–late portions of the trajectory align across runs
• seam-orientation conditions remain perfectly matched in time

Each normalized rendered baseball frame was augmented with a small amount of synthetic motion blur to approximate the temporal integration of the visual system during fast motion. Following the formulation used in Chrysos et al. (2019) [5] for generating realistic motion-blurred images from high-frame-rate video, a blurred image $B_t$ can be represented as the temporal average of $L$ consecutive “sharp” frames $I_{t+i}$:

$B_t=\frac{1}{L}\sum _{i=0}^{L-1}{I}_{t+i}$

Chrysos et al. demonstrate that this simple temporal average “simulates the natural displacement of the scene objects” during the exposure window of a real imaging system, producing realistic motion-blurred stimuli suitable for downstream perceptual modeling. In our implementation, we used $L=4$, providing a modest blur consistent with the high angular speed of a pitched baseball while preserving frame-to-frame temporal structure in the retinal signal.

This temporal normalization step ensures that any observed differences in retinal activity arise from pitch physics and seam appearance, not from differences in render length or frame sampling density.

After generating the motion-blurred frame $B_t$, the image was passed into ISETCam to simulate the optics of the human eye. Each frame was converted into a physical radiance scene, assigned a constant foveal field of view so as to focus more on relative changes in the retinal signal from frame to frame and processed through ISETCam’s shift-invariant optical model using the standard optical-image computation:

${\text{OI}}_t=\mathrm{o}\mathrm{i}\mathrm{C}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{u}\mathrm{t}\mathrm{e}(\text{Optics},B_t).$

This computation applies wavelength-dependent diffraction, aberrations, and retinal defocus to yield the spectral irradiance incident on the photoreceptor layer. This optical image ${\text{OI}}_t$ is the input to the next stage of the pipeline, where cone absorptions are computed using ISETBio’s cone mosaic model.

Cone Mosaic Modeling (ISETBio)

From there, I used ISETBio’s foveal cone mosaic to turn each optical image into predicted cone absorptions. The mosaic had realistic L/M/S ratios, normal photopigment integration times, and optional temporal blur. I ran these in noise-free mode so I could isolate the stimulus-driven differences without adding stochastic noise on top.

For every pitch sequence, this produced a full time series of cone activity across the mosaic. I summarized these by calculating:

• total cone activity per frame
• variability of the cone response over time
• simple temporal profiles of how the retinal signal evolves as the ball gets closer

These metrics let me directly compare different seam orientations while keeping all the physical parameters identical.

Results

The cone-mosaic simulations produced 8,640 retinal samples (27 parameter presets × 4 pitch types × 80 normalized frames). Each sample contains the full retinal response (foveal, mid-peripheral, far-peripheral absorptions, temporal metrics), paired with the physical state of the pitch at that moment (i.e. spin rpm, horizontal break, vertical break, pitch velocity, etc).

The resulting dataset allowed me to focus on two main questions:

• How does seam orientation affect foveal cone activity when the physical trajectory, velocity, spin, and movement are held constant?
• At what point in flight does the retinal signal contain enough information to distinguish pitch types?

Seam Orientation foveal effects

Across all pitch types, seam orientation in this simulation had no measurable effect on foveal cone activity until the ball was within roughly 10–12 ft of home plate.

The three orientations began to diverge once the ball entered the final 5-10 feet. This makes sense from the standpoint that seam features are much easier to see from closer, however, it is a bit harder conceptualize versus reality given this is a graphics animation. The size of this effect varied strongly by pitch type. Changeups showed the largest modulation: the 90° orientation increased foveal absorptions by more than 4,000 units at ~2.5 ft, and both the 45° and 90° orientations reached statistical significance. Sliders and curveballs showed moderate late-flight differences, while fastballs showed only small deviations that never reached significance.

One question that came up during analysis was whether the stronger seam-orientation effects I saw in changeups were actually driven by differences in velocity or spin rate, rather than seam orientation itself. To isolate the role of velocity, I examined the signed β(v0) coefficients across all pitch types, seam angles, and distance bins. In the graph below, blue suggests that slower speeds cause greater foveal activity, while red colors suggest faster speeds increase foveal activity. The pattern seems to follow that most β(v0) values are negative, indicating that higher velocities suppress foveal cone activity, while slower velocities enhance it. The effect is strongest inside ~15 ft of home plate, where slower pitches spend more time in the foveal window and produce higher photon catch, while faster pitches suffer additional motion-blur and reduced local contrast. Therefore the part of the “changeup advantage” in seam-related effects is simply likely that the batter’s fovea has more time to resolve the ball in that window of time compared to the faster pitch types.

Pitch Discrimination

To answer Question 2, I trained a Random Forest classifier to see if you could identify pitch type using nothing more than the retinal features and the distance bin. I kept the predictors simple: total foveal absorptions (fovea_total), a basic frame-to-frame change metric (corr_prev), and the distance bin (distance ball is to plate). I evaluated the model separately in each 5-ft window from 60 ft all the way to the plate.

From about 55 ft down to roughly 15 ft, the model sits in the 50-60% accuracy range which hints at some pitch-type signal in the retinal data early on, albeit from a batter’s point of view, a 50% estimate is rather weak given how quickly one has to react. The picture changes in the last 5–10 ft. Accuracy jumps quickly and peaks to over 70% in the final 0–5 ft bin. At that point the ball is big enough in the fovea, and the temporal pattern of the cone signal becomes informative enough, that the classifier can actually separate pitch families.

Fastballs in this simulation are the easiest for the retina to distinguish in this study, with my model suggesting with over 70% certainty of the pitch type once the pitch is within 35 feet of homeplate - however, fastballs are also the most unique amongst these pitch types in terms of velocity and trajectory. Curveballs, Changeups, and Sliders all have atleast some negative induced vertical break, while fastballs given the backspin they possess tend to have more positive vertical break. This is one plausible reason why fovea levels are so different for fastballs compared to the other pitch types in this simulation, allowing for the model to pick it up quicker and more accurately than others.

Overall, this suggests that the retina doesn’t deliver fully reliable pitch-type information until very late in the flight, which seems to somewhat lineup with intuition given the size of the ball and how fast it’s moving. It is interesting that the probabilistic results are not completely linear however, as for example, the model suggests the retina has greater confidence in predicting changeups from 45-50 feet away compared to 5 feet away. While there could be an explanation in terms of the trajectory, or the side spin changeups produce, it is more likely that this is noise due to a study run on a limited amount of data. Perhaps with more time, and adding in more robust simulations, we would see a gradual increase in probabilities as the ball approaches homeplate.

The figure below shows the full pattern of model confusions across distance, plotted as a series of 4×4 confusion matrices (one for each 5-ft bin). Within each panel, the rows correspond to the actual pitch type and the columns to the predicted pitch type, with darker cells indicating more trials.

Across almost all distances, fastballs show the strongest diagonal: the “actual fb → predicted fb” cell is consistently the darkest entry in each panel. This matches the earlier accuracy results and reinforces that fastballs are the easiest pitch for the model to recognize from the retinal features. In contrast, the offspeed and breaking pitches remain heavily entangled. Curveballs frequently get classified as sliders or changeups, changeups are often called sliders or occasionally fastballs, and sliders are regularly confused with curveballs. That structure appears as persistent off-diagonal patches in the cb/ch/sl rows and columns.

As the ball gets closer to the plate, the diagonal entries generally strengthen, but the confusion among curveballs, sliders, and changeups never fully disappears, even in the final 0–5 ft bin. Taken together, these matrices show that retinal information is very good at separating fastballs from everything else, but much less reliable for discriminating among the three non-fastball pitch types.

Another interesting way to look at these results is to examine how physical characteristics relate to pitch discrimination using retina characteristics.

The table shows, for each pitch type, the correlation between four physical pitch characteristics—spin rate, initial speed (v0), horizontal break (hb), and vertical break (vb)—and the Random Forest’s probability of assigning the correct pitch label based on the retinal features. Red cells indicate traits where higher values are associated with greater classifier confidence (higher true-class probability), and blue cells indicate traits where higher values are associated with lower confidence. The clearest pattern is the positive association between spin rate and classification confidence for changeups, fastballs, and sliders (r ≈ 0.26–0.49), with a similar positive effect of velocity for fastballs (and to a lesser extent changeups). Because spin and speed jointly shape the amount and direction of movement (hb and vb), a reasonable interpretation is that pitch instances with more distinctive movement profiles generate more separable retinal signals, which in turn makes them easier for the model—and by extension, potentially the visual system—to tell apart.

Conclusions

This project should be viewed as a first-pass exploration of what the cone-level retinal signal might look like for pitched baseballs under a tightly controlled, highly simplified environment. The goal wasn’t to recreate real hitting conditions, but to get an initial sense of how seam orientation, velocity, and movement translate into differences in the retinal input a batter might receive during a pitch.

Even in this idealized setup, it was interesting to see how certain visual cues—especially seam orientation and pitch speed—modulate foveal activity in the last few feet of flight. While these simulations are far from capturing the real complexity of human vision during hitting, they offer a starting point for thinking about how early in flight the retina even has access to information that could help distinguish pitch types.

A more complete next iteration would build directly on this simulation pipeline: incorporating more realistic rendering, expanding the dataset to hundreds or thousands of trajectories, and introducing richer peripheral and temporal metrics. Those additions would help paint a fuller picture of how the entire retina—not just the fovea—responds over the course of a pitch, and how those signals evolve in a way that might matter to a real hitter.

Appendix

Code: https://github.com/alexlipman13/pitch-trajectory-optics

References

[1] Shapiro, Arthur et al. “Transitions between central and peripheral vision create spatial/temporal distortions: a hypothesis concerning the perceived break of the curveball.” PloS one vol. 5,10 e13296. 13 Oct. 2010, doi:10.1371/journal.pone.0013296

[2] Driveline Baseball. “An Introduction to Seam-Shifted Wakes and Their Effect on Sinkers - Driveline Baseball.” Driveline Baseball, 2 Nov. 2020, www.drivelinebaseball.com/2020/11/more-than-what-it-seams-an-introduction-to-seam-shifted-wakes-and-their-effect-on-sinkers/?srsltid=AfmBOoqliVabrmDnlZ3HGsBPo971G9OlsJvPZF9ulQ8VAZEbrRoTDk5w. Accessed 7 Dec. 2025.

[3] Kato, Takaaki, and Tadahiko Fukuda. “Visual search strategies of baseball batters: eye movements during the preparatory phase of batting.” Perceptual and motor skills vol. 94,2 (2002): 380-6. doi:10.2466/pms.2002.94.2.380

[4] Nathan, Alan. “Trajectory Calculator.” Illinois.edu, 2016, baseball.physics.illinois.edu/trajectory-calculator-new.html.

‌[5] Grigorios Chrysos, et al. “Motion Deblurring of Faces.” International Journal of Computer Vision, vol. 127, no. 6-7, 1 June 2019, pp. 801–823, https://doi.org/10.1007/s11263-018-1138-7. Accessed 22 Apr. 2023.

[6] Kagan, David, and Alan M. Nathan. “Statcast and the Baseball Trajectory Calculator.” Physics Teacher, vol. 55, no. 3, 2017, eric.ed.gov/?id=EJ1142622, https://doi.org/10.1119/1.4976652.