ISETBIO Baseball Simulation Experiment

Introduction

Hitting a pitched baseball places extreme demands on the visual system. In the few hundred milliseconds between release and home plate, a batter must detect the ball, track its motion, and judge its future location well enough to decide whether and where to swing. Understanding what information is actually available at the level of the eye before any higher-level processing can provide useful insight into how difficult different pitches are to track and discriminate.

For this project, we explore how different baseball pitch trajectories drive cone responses over time in a simplified model of the batter’s eye. We start from rendered synthetic image sequences of pitched balls and use ISETCam to model the optical image at the retina under controlled field-of-view and display conditions. We then use ISETBio’s cone mosaic framework to compute noise-free cone absorptions over time as the ball travels from release to home plate, summarizing responses in foveal, mid-peripheral, and far-peripheral regions. Finally, we analyze these cone absorption sequences to see how well they do with pitch classification and understand their relationship to pitch characteristics.

Background

Prior work on baseball vision has shown that hitters operate under severe visual constraints. The ball is small relative to viewing distance, it travels quickly, and for most of its flight it remains in peripheral or near-peripheral vision. A key result from Shapiro et al. (2010, PLoS ONE) demonstrated that when an object moves from peripheral vision into the fovea, the visual system can misinterpret the motion and produce an illusory “break.” This finding is often used to explain why hitters sometimes report a sudden directional change in pitches like curveballs even though the actual trajectory is smooth.

In addition to these perceptual factors, the underlying pitch physics are more complex than the standard Magnus-force explanation. Recent aerodynamic work has shown that the raised seams create asymmetric airflow patterns—seam-shifted wakes (Barton & Nathan, 2021; Driveline Baseball, 2020). Importantly, simply changing seam orientation can alter the ball’s movement even when velocity, spin rate, and spin axis are held constant. Two pitches that would be identical in a simplified model can therefore diverge in flight purely due to seam geometry.

There is also evidence linking hitting performance to core visuomotor abilities such as tracking accuracy, predictive saccades, and eye–hand coordination (Kato & Fukuda, 2021). This means that what the batter “sees” on any given pitch reflects a combination of the retinal input, the physical flight of the ball, and the batter’s gaze strategy.

Taken together, these findings motivate the central question of this project: given the perceptual distortions that can arise from the visual system itself, the aerodynamic differences introduced by seams, and the reliance on active gaze behavior, what does the signal actually look like at the level of the cone mosaic during a pitch? That is the specific gap this analysis aims to address.

Methods

Pitch Trajectory Simulation

To generate realistic ball-flight trajectories, I implemented a JavaScript port of the aerodynamic model developed by Alan Nathan (Nathan, University of Illinois), which predicts pitch trajectories given physical parameters including:

• backspin, sidespin, and gyro spin
• initial velocity vector
• pitcher release point
• drag coefficient and lift coefficient
• spin axis direction

Nathan’s model numerically integrates the equations of motion under Magnus force, drag force, gravity and outputs 3D coordinates from release point to home plate. By embedding this model directly into my simulation backend, I was able to generate new pitch trajectories on-demand for arbitrary combinations of speed, spin rate, spin axis, while also altering seam orientation (for visual purposes only).

(Reference: https://baseball.physics.illinois.edu/trajectory-calculator.html)

Seam-Orientation Impacts

A baseball’s seam orientation - which is the 3D rotation matrix that maps the ball’s spin axis to the stitched seam geometry — is a major contributor to pitch movement. For example, four-seam fastballs typically use a seam alignment where the ball’s spin axis is perpendicular to the horseshoe seams, maximizing Magnus lift. While seam orientation is known to have impacts on pitch movement, my focus on this project is not to estimate those effects, but rather attempt to measure the effects on retinal capabilities as it relates to a batter facing different types of pitches with different orientations, speeds, movement.

To study how seam orientation alters the retinal image, I represented each orientation as a 3×3 rotation matrix applied to the baseball texture. Examples include:

• canonical orientations for the following four pitch types:
  • 4-Seam Fastball
  • Curveball
  • Slider
  • Changeup
• Rotated seam orientations for each pitch type

During rendering, the baseball texture and normal map are rotated according to this matrix so that identical pitch trajectories with different seam matrices produce different appearance sequences, even when spin rate and velocity are held constant.

This setup allowed me to isolate how the orientation of the baseball’s seams while keeping all physical pitch attributes (velocity, spin rate, movement profile) constant — affects the cone-level retinal response. As an example, the figure below shows the three seam-orientation conditions used for the four-seam fastball: 0° (the standard 4-seam look), 45°, and 90°. These rotations don’t change the physics of the pitch in this simulation, but they do change what part of the seam pattern the batter sees.

Visual Rendering of the Spinning Ball Trajectory

I used Three.js to render synthetic image sequences of each pitch as it traveled along the simulated trajectory. The renderer produced a frame-by-frame animation of the ball’s position, orientation, and seam rotation over time. For each pitch type (4-seam fastball, slider, changeup, curveball), I ran 27 total simulations by perturbing seam orientation, spin rate, and velocity around their “normal” values.

For example, for the 4-seam fastball I used:

• Three seam orientations:
  • normal
  • rotated 45°
  • rotated 90°
• Three spin rates:
  • 3000 RPM (baseline)
  • +500 RPM
  • –500 RPM
• Three velocities:
  • 96 mph (baseline)
  • +3 mph
  • –3 mph

I repeated this structure for all four pitch types, using pitch-specific baseline values (e.g., a slider has a different default seam orientation and a baseline velocity closer to 87 mph rather than 96 mph).

Each simulation produced a little over 100 PNG frames showing the ball from roughly 55 feet out all the way to the strike zone and which was used in the retinal simulation stage.

Retinal Image Formation and Motion Blur (ISETCam)

Before computing the retinal image, each rendered baseball frame was augmented with a small amount of synthetic motion blur to approximate the temporal integration of the visual system during fast motion. Following the formulation used in Chrysos et al. (2019) for generating realistic motion-blurred images from high-frame-rate video, a blurred image $B_t$ can be represented as the temporal average of $L$ consecutive “sharp” frames $I_{t+i}$:

$B_t=\frac{1}{L}\sum _{i=0}^{L-1}{I}_{t+i}$

Chrysos et al. demonstrate that this simple temporal average “simulates the natural displacement of the scene objects” during the exposure window of a real imaging system, producing realistic motion-blurred stimuli suitable for downstream perceptual modeling. In our implementation, we used $L=4$, providing a modest blur consistent with the high angular speed of a pitched baseball while preserving frame-to-frame temporal structure in the retinal signal.

After generating the motion-blurred frame $B_t$, the image was passed into ISETCam to simulate the optics of the human eye. Each frame was converted into a physical radiance scene, assigned a constant foveal field of view so as to focus more on relative changes in the retinal signal from frame to frame and processed through ISETCam’s shift-invariant optical model using the standard optical-image computation:

${\text{OI}}_t=\mathrm{o}\mathrm{i}\mathrm{C}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{u}\mathrm{t}\mathrm{e}(\text{Optics},B_t).$

This computation applies wavelength-dependent diffraction, aberrations, and retinal defocus to yield the spectral irradiance incident on the photoreceptor layer. This optical image ${\text{OI}}_t$ is the input to the next stage of the pipeline, where cone absorptions are computed using ISETBio’s cone mosaic model.

Cone Mosaic Modeling (ISETBio)

From there, I used ISETBio’s foveal cone mosaic to turn each optical image into predicted cone absorptions. The mosaic had realistic L/M/S ratios, normal photopigment integration times, and optional temporal blur. I ran these in noise-free mode so I could isolate the stimulus-driven differences without adding stochastic noise on top.

For every pitch sequence, this produced a full time series of cone activity across the mosaic. I summarized these by calculating:

• total cone activity per frame
• variability of the cone response over time
• simple temporal profiles of how the retinal signal evolves as the ball gets closer

These metrics let me directly compare different seam orientations while keeping all the physical parameters identical.

Results

The cone-mosaic simulations produced 8,640 retinal samples (27 parameter presets × 4 pitch types × 80 normalized frames). Each sample contains the full retinal response (foveal, mid-peripheral, far-peripheral absorptions, temporal metrics), paired with the physical state of the pitch at that moment (i.e. spin rpm, horizontal break, vertical break, pitch velocity, etc).

The resulting dataset allowed me to focus on two main questions:

• How does seam orientation affect foveal cone activity when the physical trajectory, velocity, spin, and movement are held constant?
• At what point in flight does the retinal signal contain enough information to distinguish pitch types?

Seam Orientation foveal effects

Across all pitch types, seam orientation in this simulation had no measurable effect on foveal cone activity until the ball was within roughly 10–12 ft of home plate.

The three orientations began to diverge once the ball entered the final 5-10 feet. This makes sense from the standpoint that seam features are much easier to see from closer, however, it is a bit harder conceptualize versus reality given this is a graphics animation. The size of this effect varied strongly by pitch type. Changeups showed the largest modulation: the 90° orientation increased foveal absorptions by more than 4,000 units at ~2.5 ft, and both the 45° and 90° orientations reached statistical significance. Sliders and curveballs showed moderate late-flight differences, while fastballs showed only small deviations that never reached significance.

One question that came up during analysis was whether the stronger seam-orientation effects in changeups were actually driven by differences in velocity or spin rate, rather than orientation itself. To check this, I fit models that included angle × spin, angle × velocity, and angle × spin × velocity interaction terms within each 5-ft distance bin. As the heatmap below shows, these interaction terms stayed near zero across all pitch types, with no consistent brightening even in the late-flight region where the main seam-orientation effects appear. This indicates that the retinal sensitivity to seam orientation is essentially independent of spin rate and velocity in this simulation, instead, the effect arises from geometric visibility (i.e., how the seams face the viewer) rather than from dynamic factors like RPM or initial speed.

Pitch Discrimination

To answer Question 2, I trained a Random Forest classifier to see if you could identify pitch type using nothing more than the retinal features and the distance bin. I kept the predictors simple: total foveal absorptions (fovea_total), a basic frame-to-frame change metric (corr_prev), and the distance bin (distance ball is to plate). I evaluated the model separately in each 5-ft window from 60 ft all the way to the plate.

From about 55 ft down to roughly 15 ft, the model sits in the 50-60% accuracy range which hints at some pitch-type signal in the retinal data early on, albeit from a batter’s point of view, a 50% estimate is rather weak given how quickly one has to react. The picture changes in the last 5–10 ft. Accuracy jumps quickly and peaks to over 70% in the final 0–5 ft bin. At that point the ball is big enough in the fovea, and the temporal pattern of the cone signal becomes informative enough, that the classifier can actually separate pitch families.

Fastballs in this simulation are the easiest for the retina to distinguish in this study, with my model suggesting with over 70% certainty of the pitch type once the pitch is within 35 feet of homeplate - however, fastballs are also the most unique amongst these pitch types in terms of velocity and trajectory. Curveballs, Changeups, and Sliders all have atleast some negative induced vertical break, while fastballs given the backspin they possess tend to have more positive vertical break. This is one plausible reason why fovea levels are so different for fastballs compared to the other pitch types in this simulation, allowing for the model to pick it up quicker and more accurately than others.

Overall, this suggests that the retina doesn’t deliver fully reliable pitch-type information until very late in the flight, which seems to somewhat lineup with intuition given the size of the ball and how fast it’s moving. It is interesting that the probabilistic results are not completely linear however, as for example, the model suggests the retina has greater confidence in predicting changeups from 45-50 feet away compared to 5 feet away. While there could be an explanation in terms of the trajectory, or the side spin changeups produce, it is more likely that this is noise due to a study run on a limited amount of data. Perhaps with more time, and adding in more robust simulations, we would see a gradual increase in probabilities as the ball approaches homeplate.

Another interesting way to look at these results is to examine how physical characteristics relate to pitch discrimination using retina characteristics.

The plot above shows how four physical pitch characteristics: spin rpm, ball speed (v0), horizontal break (hb), and vertical break (hb) correlate to the probability of the eye discriminating the pitch type. The thing that stands out the most to me, is the relation between spin rate and velocity on pitch type classification. Across changeups, fastball, and sliders, as spin rate increases, so does the probability that the pitch will be classified correctly. Because both spin rate and velocity impact movement (horizontal and vertical break), this tells me that this greater separation of movement ultimately allows the batter to discriminate pitches more effectively.