WangMoreno

Introduction

Over the past decade, the market for compact digital cameras has slowly eroded in light of improvements in mobile phone camera technology. However, consumers in the market for a new mobile phone face a difficult challenge when attempting to compare the quality of different cameras. While other components such as battery and processor have relatively clear-cut metrics, such as hours of battery life and frequency/number of cores, there are no such metrics accurately representing camera quality of phones across the market.

Traditionally, consumers looked to the megapixel count as a measure of image quality. In the past, with many digital cameras in the 1-2 megapixel range, this metric could mean the difference between a pixelated photo print and a clear one. However, megapixel count is a very poor measure of the perceived quality of the images produced by a camera; it does not take into consideration many important camera qualities such as color accuracy, signal to noise ratio (SNR), or sharpness. Additionally, with many manufacturers pushing higher and higher megapixel counts, most of today's megapixel counts run well in excess of the amount required for detail in printing or viewing. And yet, because many of these same manufacturers are not pairing these with higher quality or larger sensors, this metric has become poor even as a description of image detail and printable size alone.

Background

Image Quality Metrics

The International Standards Organization (ISO), recognizing the limitations of existing metrics, has started the I3A Camera Phone Image Quality (CPIQ) initiative. This development is detailed in the paper Development of the I3A CPIQ spatial metrics[1]. The goal of the initiative is to develop a relevant set of camera metrics which correspond to subjective perceived image quality. To do so, metrics have been developed which measure the spatial resolution, noise, and color accuracy of mobile phone cameras. These metrics attempt to capture the differences discernible to humans viewing the images on a computer display or paper printout, while ignoring the qualities that do not correspond well to perceived quality.

In our project, we are primarily focusing on color accuracy. The measure for color accuracy which we are using is the International Commission on Illumination (CIE) distance metric ΔE* (Delta E). In the image below, the CIE XYZ chromaticity diagram can be seen. The XYZ color space was originally developed in 1931 where Y is a luminance, and combinations of X and Z represent all possible chromaticities. The Lab (L*, a*, b*) color space was developed by the CIE in 1976, which was derived from the prior XYZ color space with the intention of being more perceptually uniform. In Lab space, colors are again represented through three dimensions: L for lightness, and a and b for opposing dimensions of color. Each of these values for a color can be easily computed from their corresponding XYZ values.

The metric ΔE* represents the Euclidean distance between two colors in a Lab color space, calculated from their L*, a*, and b* values via the following formula:

$\Delta E^{*}=\sqrt{(L_2^{*}-L_1^{*}{)}^2+(a_2^{*}-a_1^{*}{)}^2+(b_2^{*}-b_1^{*}{)}^2}$

In this project, we will explore how effective ΔE* is as a measure of color accuracy to compare between different cameras. We will also look at the subjective appearance of various images as viewed by humans on a display, and how well a camera's ΔE* value correlates with perceived image quality.

Color Accuracy in Modern Digital Cameras

The color of the light radiating from an object is partly determined by the illuminant's spectrum and partly by the properties of the object itself. For example, a white object can be made to appear to be any color by illuminating it with different light sources. However, the human visual system is capable, to a great extent, of adapting and neutralizing to a significant degree, the coloring effect of different illuminants. In order to accurately capture a scene, then, one of the tasks a camera system must perform during color conversion (the translation of data from the sensor's color space, determined by its color filters and sensor technology, to that of the output image, usually the sRGB standard) is to determine or estimate the illuminant incident on the scene, and "neutralize" its effect on the apparent color of objects (that is, show the colors as they would be under a standard illuminant such as D65), such that it can be reproduced later by a display system in an accurate manner. Neglecting attention to this results in a color cast on the image, often described as a lack of color balance.

Accurate color conversion of real-world images can be a tricky proposition because, unless it is told explicitly by the user, or learned through, e.g., a supplementary device, or a photograph of a known object in the target light conditions, the camera is fundamentally unable to discern to what extent the perceived light is influenced by the illuminant spectrum rather than the target's own surface properties. Despite this challenge, modern consumer cameras generally do not insist on knowing the illuminant spectrum (though most professional cameras can be used in this fashion), and must settle for guessing, based on the content of the scene, the nature of the illuminant. There is significant literature exploring the methods for this illuminant estimation.

Under very biased or dim illuminants (e.g., lacking sufficient energy in a given range of visible wavelengths) color balance can be further complicated by noise or visible quantization from insufficient signal data in certain parts of the spectrum (consider, for example, the color of objects when illuminated solely by orange sodium lamps in a nighttime urban scene). In these cases, even knowledge of the illuminant is no guarantee of proper color balance.

Auto White Balance

Without knowledge of the scene's illuminant, the color conversion process may result in a significant color cast. To help avoid this, most modern cameras offer an auto White Balance processing step. One of the most popular methods for auto White Balance operates by assuming, as a heuristic, that the average chromaticity across all pixels in a properly color-balanced image should approximate zero. A single correction is then calculated and applied to the full image, such that this overall neutrality assumption holds.

In a large portion of real-world photographs, this kind of automatic White Balance has a positive effect with no user involvement.

Methods

Experimental Setup

We used ISET for full camera system simulation. Aside from the data structures and built-in modeling, we leveraged some of the sample scripts, especially those regarding macbeth delta E, multispectral scene loading, and saving .tif files, for example, as initial examples, and developed our own similar scripts to automate our experimentation.

Scene-wise we started with the default Macbeth chart, used for obtaining conversion matrices as well as measuring deltaE for a given set of conditions.

Later we also downloaded and used several multispectral scenes. We also leveraged ISET's ability to change the scene's illuminant to simulate photographing the same scenes under different conditions. We used several of the built-in illuminant profiles, including D65, D50, Tungsten, and Fluorescent, as they represent the most popular lighting conditions in real-world use.

We simulated a relatively standard camera with simple optics, but varying many sensor and processing parameters to explore their effect on the color accuracy of resulting images.

We used the macbeth-based CCM calculation method to simulate a "best possible" single-step conversion from sensor space to sRGB. We then used these calculated CCMs when computing other images under each corresponding set of simulated conditions. When exploring the effect of mismatched / unknown illuminant conversion, we also evaluated the built-in auto white balance algorithms.

Finally, we saved the resulting images as .tif files for our subjective evaluation, again leveraging ISET's included examples of code for this purpose.

(Todo)

Describe techniques you used to measure and analyze. Describe the instruments, and experimental procedures in enough detail so that someone could repeat your analysis. What software did you use? What was the idea of the algorithms and data analysis?

Results

(just roughing in possible sections, feel free to change)

consistency / upper bounds: variations of the deltaE metric itself vs conditions / camera qualities

In the following tables, we show the optimal deltaE value as a function of sensor pixel size, scene luminance, and scene illuminant. This is the deltaE value measured on an image after being processed through an optimal color correction matrix.

Optimal DeltaE values, Small Pixel Size

	CIE D65	CIE D50	Fluorescent	Tungsten
0.1 Lux	17.20	15.54	12.31	13.32
75 Lux	3.95	4.23	2.92	4.88

Optimal DeltaE values, Standard Pixel Size

	CIE D65	CIE D50	Fluorescent	Tungsten
0.1 Lux	6.76	7.21	8.02	9.00
75 Lux	3.86	4.20	2.88	4.83

Optimal DeltaE values, Large Pixel Size

	CIE D65	CIE D50	Fluorescent	Tungsten
0.1 Lux	4.66	5.04	4.92	6.17
75 Lux	3.96	4.22	2.84	5.04

As expected, we see that in all cases, the low scene luminance of 0.1 lux results in less color accuracy than a luminance of 75 lux. The lower light levels result in much fewer photons and more noise in the final image, and thus is less color accurate.

We also see that reducing the size of the sensor pixel reduces color accuracy in the low-light case. This makes sense because the smaller the individual pixels, the fewer photons hit each individual pixel. With a larger pixel, more photons are collected, giving better accuracy. When the light level is only 0.1 lux, this difference is significant; for D65 light, we see the deltaE improve from 17.20 to 4.66 when moving from a small pixel to a large pixel. For normal light levels, however, we observed no real difference between the pixel sizes. Even with a small pixel size, enough photons were detected per pixel that the color accuracy was more or less optimal.

The illuminant with which we saw the best deltaE values was fluorescent lighting. The different spectrums of natural light came next, with D65 performing slightly better than D50. Tungsten light tended to be the least color accurate, having the highest deltaE values in almost all cases.

The following two charts show the spectral power distribution of our illuminants. The first shows four CIE standard illuminants, including D65 and D50. The latter shows the spectrum of tungsten (incandescent) light in red, and fluorescent light in blue.

The wavelengths corresponding to visible light for humans ranges from about 390nm to 700nm. We can clearly see that the spectral power distribution of fluorescent light falls almost entirely within this range. While the spectrum is clearly not uniform, we are able to correct for this via a color correction matrix. As a result, the maximum deltaE achievable under fluorescent light is better than for the other illuminants.

The spectral power distribution for the two CIE standard illuminants, D65 and D50, are clearly more spread out than for fluorescent light. However, they are both still mostly concentrated within the visible light spectrum. The differences between the two are fairly minor: D65 has slightly more power for wavelengths below 560nm, while D50 has slightly more above 560nm. D65 has a slightly higher proportion of its spectrum within the visible spectrum, resulting in a slightly better optimal deltaE.

In contrast to the other illuminants, we see that tungsten/incandescent light has a spectrum heavily biased towards higher wavelengths; most of the spectrum falls outside the visible spectrum of light. Thus, many of the photons reaching the sensor would not even be visible to a human viewing the same scene. This results in a higher maximum deltaE than the other illuminants in most cases, corresponding to less color accuracy even in well-lit scenes.

The one major exception to the above observations was the case with a small pixel size in the sensor and a low light level of 0.1 lux. In this case, tungsten light had much better deltaE color accuracy than the CIE illuminants, and D50 also performed slightly better than D65. This may be a result of the energy-wavelength relationship of photons, where longer wavelengths correspond to more energy. Given the combination of sensor pixel size and light level, the number of photons being absorbed by any given pixel in the sensor is clearly very low. As a result, the presence of high-energy photons should be beneficial towards color accuracy. We observed above that the spectrum of tungsten/incandescent light is very heavily skewed towards longer wavelength light, and D50 light also has more high wavelength spectrum than D65 light. This means that the average photon will have more energy, resulting in better color accuracy in this scenario.

 (variations vs. illuminant with "best match" processing: discuss / illustrate best/worst cases)
    for dim light, discussion might hint at different lower limits of sensitivity of sensor elements?
    also I wonder if we could show some illuminant spectrum plots to help explain some of the effects?

 (variations vs. sensor design choices: mention things that had no effect; discuss sensitivity to pixel size)
    
 (variations vs. camera processing: again mention things that had no effect, discuss effectiveness of white balance in correcting "mismatched" processing, focus on deltaE terms.)

deltaE vs. viewer opinion of image quality

 (pixel size: does it matter to "humans" in proportion with deltaE? show examples)

 (illuminant-mismatched processing: do humans care? (yup!)does auto white balance really help? again, does deltaE match opinion)
   good opportunity to show faces, discuss skin tones, indoor illuminants.

   outdoor sunset light vs. noon light

 (small pixels, low light: ugly ugly, but does our judgement match?)

---

Organize your results in a good logical order (not necessarily historical order). Include relevant graphs and/or images. Make sure graph axes are labeled. Make sure you draw the reader's attention to the key element of the figure. The key aspect should be the most visible element of the figure or graph. Help the reader by writing a clear figure caption.

Conclusions

Describe what you learned. What worked? What didn't? Why? What should someone next year try?

• Large differences in deltaE seem useful for avoiding awful photos

• Fine-grain differences (e.g., <10) don’t always match subjective judgement.

• Auto White Balance unreliable way to correct improper conversion, both subjectively and by deltaE measure.

Future Work / What We Didn't Try

• LEDs are becoming popular as indoor lighting; Their color accuracy performance should be evaluated.
• Night photography vs. human night vision: what does color accuracy mean in this situation?
• DeltaE after Auto White Balance: content-dependence of "grayworld" and "whiteworld" methods requires a more carefully designed experiment ($\Delta E$ calculation uses macbeth chart scene, but we want to check the white balance transform generated from a different scene).

References

Daxter, Donald; Frederic Cao; Henrik Eliasson; Jonathan Phillips. "Development of the I3A CPIQ spatial metrics." Web. 8 Mar 2014. <http://proceedings.spiedigitallibrary.org/data/Conferences/SPIEP/64097/829302_1.pdf>.

Appendix I

Multispectral Scenes Used

For the deltaE calculations, we used the basic Macbeth Color Chart scene from ISET. For the subjective evaluations, we obtained several other multispectral scenes, both indoor and outdoor, from the available archives, noted below.

• Included with ISET:
  • Macbeth scene(s)
  • StuffedAnimals_tungsten-hdrs.mat
• LoResFemale6_Cx.mat
• LoResMale4_Cx.mat
• StanfordDish_Cx.mat
• StanfordMemorial_Cx.mat

Upload source code, test images, etc, and give a description of each link. In some cases, your acquired data may be too large to store practically. In this case, use your judgement (or consult one of us) and only link the most relevant data. Be sure to describe the purpose of your code and to edit the code for clarity. The purpose of placing the code online is to allow others to verify your methods and to learn from your ideas.

Scripts

makeCamera: a script to instantiate a camera model (optics,sensor,and processor). The goal is to have a consistent camera setup, where the parameters that turn out to be relevant to the experiment can be easily varied by simple inputs.

makeImage:

TODO: include (attach?) scripts used and description of what they do.

Appendix II

(for groups only) - Work breakdown. Explain how the project work was divided among group members.