View source for Ribera

== Abstract ==
The wide field of view in head-mounted displays implies that most of the content shown by the display appears at high eccentricities.
However, most image quality metrics assume that the image is viewed by the fovea, where the human visual system presents its highest acuity.
In this project, we extend an image quality metric based on ISETBIO to wide field of views.
We apply publicly available measurements of the Optical Transfer Function of the human cornea and lens at different eccentricities.
In this manner, our Delta E error map becomes eccentricity-dependent.
After incorporating more accurate OTF models to our method, we expect to use this image quality metric as a tool to design newer displays technologies that take advantage of the loss of acuity in higher eccentricities.


== Introduction == 
Head-mounted displays are pushing the display specifications (e.g, resolution or refresh rate) much beyond those of traditional displays.
One of the reasons is the need to cover a huge field of view with a higher pixel density, since the resolvable pixel size at such close distances is much smaller.
This unavoidably leads to a tremendously higher need for bandwidth in the transmission lines that drive the display.
Unfortunately the demand outpaces the speed at which newer transmission lines increase their available bandwidth.
In this context, image and video compression plays an important role by decreasing the required bandwidth [1].
In order to ensure that the compression algorithms do not degrade the visual quality of the displayed image, compression engineers use image quality metrics.

However, most image quality metrics assume that the image is viewed by the fovea, where the Human Visual System (HVS) presents its highest acuity.
This motivates the need for image quality metrics that take into account a high field of view.
Such image quality metrics can become a valuable tool to design display technologies for future Virtual Reality headsets.

== Background ==
This project is based on an image quality metric called ISETBIOLAB proposed in [2], which in turns follows the ISETBio approach of modelling every step in the pipeline of the HVS.
An schematic of ISETBIOLAB is shown in Figure 1.
First, it computes the radiance of the scene emitted from the display by considering the subpixel arrangement and spectral power distribution of the display primaries.
Then the modulation transfer function of the human optics (the optical transfer function) is applied to each wavelength of the scene to obtain the irradiance on the retina, i.e, the optical image (also called retinal image).
This process is repeated for the second image (the image being compared with).
Finally, both retinal images are converted to XYZ and a color difference metric (such as CIEDE1976) is applied pixelwise.
This creates a Delta E error map between the original and reconstructed image.
Figure 2 shows an example of an image going through ISETBIOLAB.

[[File:Isetbiolab.png|400px|thumb|center|Figure 1. Schematic of ISETBIOLAB]]

[[File:Isetbiolab_pictorial.png|400px|thumb|center|Figure 2. Example of an image going through ISETBIOLAB]]

On the other hand, the Optical Transfer Function (OTF) of the HVS has been measured in a variety of ways, including double-pass interferomic measurements [4, 5] and wavefront sensors [6, 7].
These OTFs depend on the eccentricity of the incoming rays crossing the human pupil, thus applying an OTF obtained from an on-axis analysis to an image observed with a wide field of view is not correct.

== Methods ==
We did some literature review and found no lack of eccentricity-dependent OTFs derived from different measures of the human optics.
The following table summarizes our findings:

[[File:OTFmodelsTable.PNG|800px|thumb|center|Table 1. Critical characteristics of a few OTF models.]]

One can note in recent works (Jaeken [6] and Polans [7]) the higher granularity of the measurements and that they did not did to paralyze accommodation to take the measurements.
This is because of the use of more modern instrumentation that can operate much faster.
Also one could make use of the ISO12233 standard to measure the MTF of the HVS ourselves, and incorporate for example the effect of the subsampling and demosaicking of the cones in the retina.
However this goes beyond the scope of this project.
Instead, we decided to take the OTF derived from one of these models and adapt ISETBIOLAB, with the hope that other researchers take more accurate OTF measurements in the future and we can easily incoporate those thanks to ISETBIO's framework.

We decided to use Jaeken's OTF derived from measurements of the wavefront aberration due to (1) it is already included in ISETBio, (2) its highest granularity (1 degree steps), and (3) their claim to have the most accurate model at the time of their publication.
ISETbio includes the function ''data/dataroutines/wvf/wvfLoadWavefrontOpticsData.m'' that loads the wavefront aberration from Jaken at a particular eccentricity and converts it to a Point Spread Function.
Then applies a Fourier transform to obtain the OTF at a specific eccentricity.

[[File:Figure3_Ribera.PNG|500px|thumb|center|Figure 3. Eccentricity-dependent OTF applied to the scene in ISETBIOLAB]]

Here we describe the pipeline to make ISETBIOLAB eccentricity-dependent.
This only affects the steps between the scene irradiance and the optical image, and it is depicted in Figure 3.
After the scene irradiance has been computed, we independently apply the OTF at a range of eccentricities, for example 0°, 5°, 10°, ..., 40°.
Since the OTF is only valid at a small area of the image, we must only keep the filtered image in the appropriate image region.
Assuming the axis of the eye is aligned with the line perpendicular to the display, we can easily obtain which eccentricity each pixel has.
For example, if the display is assumed to be flat and our eye axis is aligned with the line perpendicular to the display and going through the center of the image, by simple trigonometry we can get

[[File:FoVformula.PNG|300px|thumb|center|Equation 1. Formula to obtain the vertical and horizontal eccentricity (alpha_w and alpha_v, respectively) from from the display size (HxW) in pixels, the Pixels Per Inch (delta) and the distance to the display (d) in meters.]]

[[File:Angles.png|300px|thumb|center|Figure 4. Eccentricity values throughout all the pixels in a 1600x1440 Head-Mounted Display.]]

Note in the example shown in Figure 4 that eccentricities can be very high for the corners of the images shown in a Head-Mounted Display.
If the the display is circular (as shown in Figure 5), one can just divide the display size in pixels by the Pixels Per Degree (PPD) in each dimension.

[[File:CircularFoV_viewing_person.PNG|200px|thumb|center|Figure 5. Eccentricities in a circular display.]]

After computing the irradiance image on the retina at a set of eccentricities, an thin annular region centered at each eccentricity is created and used to select the appropriate region of the retinal images.
This is shown in Figure 6.
All these retinal images are merged into the final retinal image by multiplying the selecting annular regions (1 in the annulus, 0 outside of it), and summing across images.
Note that this assumes that the eye is circularly symmetric.
If this assumption is removed, once should design the new selector images appropriately.

[[File:Merging eccentricities.PNG|600px|thumb|center|Figure 6. Merging of retinal images at different eccentricities.]]


This pipeline is repeated for both original and reconstructed image, as shown in Figure 7.
Both retinal images are converted into CIE XYZ coordinates and a color difference metric (such as CIEDE1976) is applied pixelwise.

[[File:Double pipeline.PNG|600px|thumb|center|Figure 7. Entire eccentricity-variable ISETBIOLAB pipeline for both original and reconstructed image.]]

== Results ==

The original and reconstructed images used in this study can be downloaded from [[File:Fly.zip]].
The reconstructed image has been compressed and decompressed using JPEG with a relatively low Quality Factor.
The original image has been obtained from Wikimedia [[https://commons.wikimedia.org/wiki/File:Female_Striped_Horse_Fly_(Tabanus_lineola).png]].
From the following link: http://scarlet.stanford.edu/teach/images/3/30/Flickering_fly_errormap.gif , you can see the error map obtained via the proposed eccentricity-dependent ISETBIOLAB, flickering at 10Hz with the traditional on-axis ISETBIOLAB.
Note that, except for the central circular region, the Delta E values are higher than in the traditional on-axis ISETBIOLAB.
This effect increases as the eccentricity increases, as expected, suggesting a progressive decrease in the ability to discern compression artifacts as the eccentricity increases.

All the Delta E error maps shown in this report are using the jet colormap, linearly scaled between 0 and 8.
Any Delta E value above 8 has been clipped to 8.

== Limtiations of the model and future work == 
The most important limitation of this model is the computationally inefficiency
Furthermore, the method approximates the radial increase of eccentricity with step functions.
This makes it innaccurate unless the number of steps is very high.
Increasing the number of steps makes the method increasingly wasteful of computational resources.
Another limitation is the fact that the Jaeken model was not measured using a light in the visible spectrums.
However this is a limitation of the OTF mode and not the eccentricity-variant pipeline.
We expect such OTF can be extended to other wavelengths.

== Conclusions == 
In this project we have proposed and implemented a procedure to extend an image quality metric for high field of view viewing conditions. 
We have used an image quality metric that follows the computational observer approach of ISETBio, and simulates the physics of every step in the pipeline of the Human Visual System.
Our porposed method is able to incoporate OTFs from multiple sources.
We expect this method to be a useful tool to design future Virtual Reality headsets.

== Files == 
[http://scarlet.stanford.edu/teach/images/8/86/Slides_20191210_psych221.zip Presentation slides]

== References ==

[1] F. G. Walls and A. S. MacInnis, "VESA Display Stream Compression for Television and Cinema Applications," IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 6, no. 4, pp. 460-470, December 2016. https://doi.org/10.1109/JETCAS.2016.2602009

[2] G. W. Cook, J. Ribera, D. Stolitzka and W. Xiong, "A Subpixel‐based Objective Image Quality Metric with Application to Visually Lossless Image Compression Evaluation," Society for Information Display Digest of Technical Papers, vol. 49, no. 1, May 2018. https://doi.org/10.1002/sdtp.12103

[3] https://github.com/isetbio/isetbio

[4] R. Navarro, P. Artal, and D. R. Williams, "Modulation transfer of the human eye as a function of retinal eccentricity," vol. 10, no. 2, pp. 201-212, 1993. https://doi.org/10.1364/JOSAA.10.000201

[5] D. R. Williams, P. Artal, R. Navarro, M. J. McMahon and D. H. Brainard, "Off-axis optical quality and retinal sampling in the human eye", vol. 36, no. 8, April 1996. https://doi.org/10.1016/0042-6989(95)00182-4

[6] B. Jaeken and P. Artal, "Optical Quality of Emmetropic and Myopic Eyes in the Periphery Measured with High-Angular Resolution," Journal of Visual Psychophysics and Physiological Optics, vol. 53, no. 7, June 2012. https://doi.org/10.1167/iovs.11-8993

[7]  J. Polans, B. Jaeken, R. P. McNabb, P. Artal, and J. A. Izatt  "Wide-field optical model of the human eye with asymmetrically tilted and decentered lens that reproduces measured ocular aberrations," Optica, vol. 2, no. 2, pp. 124-134, 2015. https://doi.org/10.1364/OPTICA.2.000124