Scielabfrontendoptics

Introduction

In this project, we are looking to see if the spatial extension of CIELAB (S-CIELAB) provides results that model front-end optics. We focus on comparing ∆E error maps between two images that have been obtained in two ways. The first error map is obtained by using S-CIELAB and finding ∆E between two original images. The second error map is taken by ∆E using standard CIELAB, but is found using cone responses that have been simulated by IsetBio.

Background

The standard form of CIELAB is useful for measuring the ∆E value between two uniform fields of color, but it has been found that the human eye responds to fields with fine differences in colors differently[1]. Therefore, the S-CIELAB space was created to account for this. An image is turned into S-CIELAB space by first separating it into different color components, filtering each component on the eye’s sensitivity to each component, reconstructing each into XYZ space, then turning it into LAB. This spatial filtering performed is supposed to account for the human eye’s front-end optics.

Further experimentation has been done to compare the S-CIELAB ∆E metrics to visual differences in images that human observers can discern[2]. While the S-CIELAB ∆E gives a better result than the standard CIELAB ∆E, it does not completely align with data provided by human targets. Thus, we would like to compare the S-CIELAB ∆E measurements with measurements given by human cone responses, and see to what extent S-CIELAB actually models front-end optics.

Methods

Experiment Design

We start by analyzing the noise that comes from the cone responses. This is necessary for us to interpret the results correctly, since we need to know which effects show up a as a result of noise and which ones show up as a result of S-CIELAB calculation or the front-end optics model (excluding noise). While the results below show the noise for an integration time of 50ms, we got rid of noise of the noise by using an integration time of 1000s for most experiments. In that case, we observed a maximum ∆E due to noise of about 0.7. We then compare two scenes of two uniformly colored squares of different colors to get information about the dependency of S-CIELAB, CIELAB, and the front-end optics model in complete absence of spatial frequencies. In the last experiment, we add the frequency component to the experiment by comparing spoke targets of different colors. Spoke targets contain spatial frequencies across a wide spectrum and are well suited to test to which extent S-CIELAB models front end optics. Figure 1 shows the general experiment setup for all three experiments.

Figure 1: Experiment Setup

Noise Analysis

In order to analyze the noise in the cone responses, we get look at a Macbeth Colorchecker. We record the and the MSE, the SNR for each of the three channels (L, M, S or R, G, B) separately on the demosaicked cone response image.We also compare two identical Macbeth Colorchecker scenes and calculate the CIELAB ∆E between the two cone responses coming from two identical scenes. We use the white from the bottom left corner of the color chart as a reference white for the ∆E calculations.

Uniform Squares

We experiment on 28 pairs of similarly colored squares, with a ∆E of 10 or lower between each color pair. We take 7 original colors (red, yellow, green, green-blue, blue, violet, and white), and modify each in 4 ways to get 4 different comparisons for each square. The 4 modifications we make are as follows:

1. Change just the red channel (turn 0 into 0.05, or 1 into 0.95)
2. Change just the green channel
3. Change just the blue channel
4. Change all three channels

We obtain two error maps for each pair of squares:

1. S-CIELAB ∆E between the two original images (white point = [1,1,1])
2. ∆E between demosaicked cone response images (white point = [0.97,0.88,0.18])

We also take the average XYZ value of each cone response and find the ∆E between these two to have a metric of an average ∆E value that is not affected by noise. This value is compared to the mean of the S-CIELAB ∆E error map (which is constant).

Spoke Targets

In the last experiment, we record the two error measurements (S-CIELAB between the scenes and CIELAB between cone responses) as we did in the uniform squares experiment. Two different modes of comparison are made: The first mode is a comparison of a spoke with two colors, and a spoke with the same two colors, but with a phase shift of 180° (see Figure 2.1). The second mode is a comparison between a bi-colored spoke and a spoke where one color is slightly changed (see Figure 2.2). We started out by comparing colors that have a very large ∆E, but we also compared colors that have an almost unnoticeable ∆E, shown in Figure 2.3.

Figure 2.1: Two bi-colored spoke targets with a 180° phase shift

Figure 2.2: Two bi-colored spoke targets where one color is different

Figure 2.3: Two bi-colored spoke targets (180° shift) of very similar colors

Results

Noise Analysis

We found that the SNR (introduced by mainly photon / absorption noise) differs significantly depending on the color. Overall, blue cones show the fewest absorptions and the lowest SNR. We conducted the experiments for an integration time of 50ms, acknowledging that changing the integration time would only scales the photon noise by a constant factor and not yield significant additional information. Figure 5 shows the SNR for the cone responses. We labeled the Macbeth Colorchecker according to the scheme shown in Figure 4. From Figure 5 we can see that the SNR is generally higher for the L and M cones than it is for the S cones. Furthermore, the more absorptions we expect due to the color, the higher the SNR. This effect is especially visible for the white square, where all cones have high absorption rates.

Figure 4: Macbeth Color Checker

Figure 5: SNR by cone type for each color

We also calculated the CIELAB ∆E between two instances of cone responses that result from two identical scenes, to get a feeling for how much of a color (∆E) difference the noise is responsible for (see Figure 6.1). In order to reduce the influence of photon noise on our experiments, we chose the integration time to be 1000s, which resulted in a ∆E of 0.7 or less, as shown in Figure 6.2.

Figure 6.1: ∆E for each color with 50ms integration time

Figure 6.2:∆E for each color with 1000s integration time

Uniform Squares

When comparing the ∆E of the difference of average cone responses to the average ∆E of S-CIELAB, we noticed that the cone response ∆E generally was much smaller. While the average S-CIELAB ∆E between each pair of colors was 4.58, the ∆E average for cone responses was 1.45. The average ratio between the two metrics for each pair was also 0.24 with a standard deviation of 0.36, and the median of the 28 ratios was 0.03. However, while we found a general trend of the cone responses providing a smaller ∆E than S-CIELAB, there are currently no general patterns in how changing a color affects the ∆E in the optical responses. For example, adjusting all the channels of the red square provided the largest S-CIELAB ∆E among the red experiments, but also yielded the smallest cone response ∆E among them too.

There were also a few interesting outliers in our measurements -- for instance, when modifying only the red channel of magenta ([1,0,1]), the ∆E ratio between cones and S-CIELAB was 1.47, with the cones showing a ∆E average of 6.72 compared to the S-CIELAB value of 4.59 (Figure 8.2). Another curious example was when the reference color was blue ([0,0,1]). Each of these 4 experiments yielded an average ∆E for cone responses of 0, showing that the cone responses were not differentiating at all between the two shades of blue (FIGURE 7.2).

Figure 7.1: The original blue squares (left) and the squares reconstructed from cone responses (right).

Figure 8.1:The original magenta squares (left) and the squares reconstructed from cone responses (right).

Figure 7.2: ∆E values for two blue squares. The cone responses (right) give an average ∆E of 0.

Figure 8.2: ∆E values for two magenta squares. The cone responses (right) give a higher average ∆E than S-CIELAB (left).

Figure 9 shows the measurements and calculated metrics for each of the 28 experiments, as well as the average ∆E for S-CIELAB, the cone responses, and the average ratio between ∆E of cones and ∆E of S-CIELAB.

Figure 9: ∆E values calculated from similarly colored uniform squares. For each color, the experiments from left to right are changing all 3 channels, changing the red channel, changing the green channel, and changing the blue channel.

Spoke Targets

To add the spatial frequency component to the experiments, we now compare spoke targets that contain low patial frequencies on the outside and high spatial frequencies on the inside. Figure 10.1 and 10.2 show a 180° phase shift comparison of a magenta / violet target, and Figure 11.1 and 11.2 show a comparison of a yellow blue target, where the ∆E difference between the two colors is even bigger than in the magenta / violet case.

Figure 10.1: Magenta / Violet 180° phase shift visual comparison between scene (left) and reconstruction (right)

Figure 10.2: S-CIELAB ∆E between input scenes (left) and CIELAB ∆E between cone responses (right)

In this case, it seems that the S-CIELAB ∆E is overestimating the ability to see high spatial frequencies in the magenta / violet case. That can be seen from the size of the "circle / hole" in the middle of the error map, which is larger in the cone responses than in the input scene. Let's see whether that's the case for the high contrast blue / yellow scene as well (Figure 11.1 and 11.2).

Figure 11.1: Blue / Yellow 180° phase shift visual comparison between scene (left) and reconstruction (right)

Figure 11.2: S-CIELAB ∆E between input scenes (left) and CIELAB ∆E between cone responses (right)

In the blue / yellow case, the mean S-CIELAB ∆E is very high. The mean ∆E of the reconstructed images is lower than the ∆E of the scenes. This is a recurring pattern we already noticed when comparing uniform squares. In addition, since the colors are fundamentally different, we can see that it’s much easier for the human eye to see high spatial frequencies, compared to the magenta / violet case. The error images show the high spatial frequencies very well, meaning that the human eye can still discriminate those frequencies, which is intuitive given the larger contrast between blue and yellow compared to magenta and violet. Is seems that S-CIELAB does not capture this relation as well as the cone responses do. The “circle / hole ” of the spoke target (that is, the threshold above which high spatial frequencies can’t be seen anymore) seems to be larger in the S-CIELAB ∆E, meaning the threshold lies at a lower frequency, than the cone responses suggest.

We observe a similar interesting behavior when comparing very similar colors with a small ∆E (Figure 12).

Figure 12 Error maps for two pairs of magenta spokes. The first pair has a 5% difference in the blue channel, and the second pair has a 5% difference in all channels. Left: S-CIELAB, Right: CIELAB

In the first comparison, we compared two magenta scenes that had a slight difference in the blue channel (5%). It seems that S-CIELAB is overestimating the ability to discriminate high spatial frequencies. In the second case, we use a magenta spoke target again, but this time all three channels have a 5% color change. As expected, the S-CIELAb ∆E becomes larger, and according to S-CIELAB its a bit easier now to discriminate higher spatial frequencies. This behavior, however, does not show up in the∆E of the cone responses. Instead, the “hole” in the middle of the ∆E of the cone responses appears bigger than it did in the first case. Again, we can also find cases, where S-CIELAB underestimated the ability to discriminate colors, as shown in Figure 13.

Figure 13 Error maps for two green-blue spokes, one with a 5% difference in each channel. Left: S-CIELAB, Right: CIELAB

Here, it appears as if the cone responses recorded a higher spatial resolution of the two slightly different colors (Green-Blue with vs. Green-Blue with a 5% difference in each channel) than was predicted by S-CIELAB.

Conclusions

The results from the noise experiment give us information about the accuracy of our measurements, and they tell us that noise is more significant for the S cones than it is for L and M cones. From the results of the uniform square experiments, it seems that cones are generally less capable of discerning slight differences in uniform colors. While it is still unclear exactly how changing certain parameters affects the ∆E values calculated in each way, the cones typically return an average ∆E of about ¼ the magnitude of the S-CIELAB ∆E.

The spoke target experiments show some interesting results. It seems that there is a difference between the S-CIELAB ∆E and the information that is captured in the cone responses. Depending on the color combination, the image on the retina contains sometimes more, sometimes less spatial information than a S-CIELAB analysis would suggest. This behavior is observed for both very different and very similar color combinations. This means that either the model of the front-end optics of the human eye is inaccurate or not representative for all humans, or that S-CIELAB accounts for more than just the front-end optics of the human eye. Further exploration on this phenomenon might give more insights into which phenomena S-CIELAB really captures.

The goal of this project was to start exploring the differences between S-CIELAB and cone responses. Our hope is that with these preliminary results, our project may provide insight for future experimentation and research on the matter.

Appendix

Here is a Github repository containing the relevant code we used for this project. The code found here can create scenes out of spokes and uniform squares, recreate images based on cone responses, and run our full experiments. Enjoy!

https://github.com/LCJebe/ColorCones

References

(1) http://scarlet.stanford.edu/~brian/scielab/scielab3/scielab3.pdf

(2) http://scarlet.stanford.edu/~brian/scielab/compare/compare.pdf