WangMoreno

Introduction

Over the past decade, the market for compact digital cameras has slowly eroded in light of improvements in mobile phone camera technology. However, consumers in the market for a new mobile phone face a difficult challenge when attempting to compare the quality of different cameras. While other components such as battery and processor have relatively clear-cut metrics, such as hours of battery life and frequency/number of cores, there are no such metrics accurately representing camera quality of phones across the market.

Traditionally, consumers looked to the megapixel count as a measure of image quality. In the past, with many digital cameras in the 1-2 megapixel range, this metric could mean the difference between a pixelated photo print and a clear one. However, megapixel count is a very poor measure of the perceived quality of the images produced by a camera; it does not take into consideration many important camera qualities such as color accuracy, signal to noise ratio (SNR), or sharpness. Additionally, with many manufacturers pushing higher and higher megapixel counts, most of today's megapixel counts run well in excess of the amount required for detail in printing or viewing. And yet, because many of these same manufacturers are not pairing these with higher quality or larger sensors, this metric has become poor even as a description of image detail and printable size alone.

Background

Image Quality Metrics

The International Standards Organization (ISO), recognizing the limitations of existing metrics, has started the I3A Camera Phone Image Quality (CPIQ) initiative. This development is detailed in the paper Development of the I3A CPIQ spatial metrics[1]. The goal of the initiative is to develop a relevant set of camera metrics which correspond to subjective perceived image quality. To do so, metrics have been developed which measure the spatial resolution, noise, and color accuracy of mobile phone cameras. These metrics attempt to capture the differences discernible to humans viewing the images on a computer display or paper printout, while ignoring the qualities that do not correspond well to perceived quality.

In our project, we are primarily focusing on color accuracy. The measure for color accuracy which we are using is the International Commission on Illumination (CIE) distance metric ΔE* (Delta E). In the image below, the CIE XYZ chromaticity diagram can be seen. The XYZ color space was originally developed in 1931 where Y is a luminance, and combinations of X and Z represent all possible chromaticities. The Lab (L*, a*, b*) color space was developed by the CIE in 1976, which was derived from the prior XYZ color space with the intention of being more perceptually uniform. In Lab space, colors are again represented through three dimensions: L for lightness, and a and b for opposing dimensions of color. Each of these values for a color can be easily computed from their corresponding XYZ values.

The metric ΔE* represents the Euclidean distance between two colors in a Lab color space, calculated from their L*, a*, and b* values via the following formula:

$\Delta E^{*}=\sqrt{(L_2^{*}-L_1^{*}{)}^2+(a_2^{*}-a_1^{*}{)}^2+(b_2^{*}-b_1^{*}{)}^2}$

In this project, we will explore how effective ΔE* is as a measure of color accuracy to compare between different cameras. We will also look at the subjective appearance of various images as viewed by humans on a display, and how well a camera's ΔE* value correlates with perceived image quality.

Color Accuracy in Modern Digital Cameras

The color of the light radiating from an object is partly determined by the illuminant's spectrum and partly by the properties of the object itself. For example, a white object can be made to appear to be any color by illuminating it with different light sources. However, the human visual system is capable, to a great extent, of adapting and neutralizing to a significant degree, the coloring effect of different illuminants. In order to accurately capture a scene, then, one of the tasks a camera system must perform during color conversion (the translation of data from the sensor's color space, determined by its color filters and sensor technology, to that of the output image, usually the sRGB standard) is to determine or estimate the illuminant incident on the scene, and "neutralize" its effect on the apparent color of objects (that is, show the colors as they would be under a standard illuminant such as D65), such that it can be reproduced later by a display system in an accurate manner. Neglecting attention to this results in a color cast on the image, often described as a lack of color balance.

Accurate color conversion of real-world images can be a tricky proposition because, unless it is told explicitly by the user, or learned through, e.g., a supplementary device, or a photograph of a known object in the target light conditions, the camera is fundamentally unable to discern to what extent the perceived light is influenced by the illuminant spectrum rather than the target's own surface properties. Despite this challenge, modern consumer cameras generally do not insist on knowing the illuminant spectrum (though most professional cameras can be used in this fashion), and must settle for guessing, based on the content of the scene, the nature of the illuminant. There is significant literature exploring the methods for this illuminant estimation.

Under very biased or dim illuminants (e.g., lacking sufficient energy in a given range of visible wavelengths) color balance can be further complicated by noise or visible quantization from insufficient signal data in certain parts of the spectrum (consider, for example, the color of objects when illuminated solely by orange sodium lamps in a nighttime urban scene). In these cases, even knowledge of the illuminant is no guarantee of proper color balance.

Auto White Balance

Without knowledge of the scene's illuminant, the color conversion process may result in a significant color cast. To help avoid this, most modern cameras offer an auto White Balance processing step. One of the most popular methods for auto White Balance operates by assuming, as a heuristic, that the average chromaticity across all pixels in a properly color-balanced image should approximate zero. A single correction is then calculated and applied to the full image, such that this overall neutrality assumption holds.

In a large portion of real-world photographs, this kind of automatic White Balance has a positive effect with no user involvement.

Methods

experimental setup

We used ISET for full camera system simulation. Aside from the data structures and built-in modeling, we leveraged some of the sample scripts, especially those regarding macbeth delta E, multispectral scene loading, and saving .tif files, for example, as initial examples, and developed our own similar scripts to automate our experimentation.

Scene-wise we started with the default Macbeth chart, used for obtaining conversion matrices as well as measuring deltaE for a given set of conditions.

Later we also downloaded and used several multispectral scenes. We also leveraged ISET's ability to change the scene's illuminant to simulate photographing the same scenes under different conditions. We used several of the built-in illuminant profiles, including D65, D50, Tungsten, and Fluorescent, as they represent the most popular lighting conditions in real-world use.

We simulated a relatively standard camera with simple optics, but varying many sensor and processing parameters to explore their effect on the color accuracy of resulting images.

We used the macbeth-based CCM calculation method to simulate a "best possible" single-step conversion from sensor space to sRGB. We then used these calculated CCMs when computing other images under each corresponding set of simulated conditions. When exploring the effect of mismatched / unknown illuminant conversion, we also evaluated the built-in auto white balance algorithms.

Finally, we saved the resulting images as .tif files for our subjective evaluation, again leveraging ISET's included examples of code for this purpose.

(Todo)

Describe techniques you used to measure and analyze. Describe the instruments, and experimental procedures in enough detail so that someone could repeat your analysis. What software did you use? What was the idea of the algorithms and data analysis?

Results

(just roughing in possible sections, feel free to change)

(section 1: consistency / upper bounds: variations of the deltaE metric itself vs conditions / camera qualities)

Optimal DeltaE values

Illuminant	D65	D50	Fluorescent	Tungsten
0.1 Lux	17.20	15.54	12.31	13.32
75 Lux	3.95	4.23	2.92	4.88

The illuminant with which we saw the best deltaE values was fluorescent lighting.

 (variations vs. illuminant with "best match" processing: discuss / illustrate best/worst cases)
    for dim light, discussion might hint at different lower limits of sensitivity of sensor elements?
    also I wonder if we could show some illuminant spectrum plots to help explain some of the effects?

 (variations vs. sensor design choices: mention things that had no effect; discuss sensitivity to pixel size)
    
 (variations vs. camera processing: again mention things that had no effect, discuss effectiveness of white balance in correcting "mismatched" processing, focus on deltaE terms.)

(section 2: deltaE vs. viewer opinion of image quality)

 (pixel size: does it matter to "humans" in proportion with deltaE? show examples)

 (illuminant-mismatched processing: do humans care? (yup!)does auto white balance really help? again, does deltaE match opinion)
   good opportunity to show faces, discuss skin tones, indoor illuminants.

   outdoor sunset light vs. noon light

 (small pixels, low light: ugly ugly, but does our judgement match?)

---

Organize your results in a good logical order (not necessarily historical order). Include relevant graphs and/or images. Make sure graph axes are labeled. Make sure you draw the reader's attention to the key element of the figure. The key aspect should be the most visible element of the figure or graph. Help the reader by writing a clear figure caption.

Conclusions

Describe what you learned. What worked? What didn't? Why? What should someone next year try?

Future Work / What We Didn't Try

• LEDs are becoming popular as indoor lighting; Their color accuracy performance should be evaluated.
• Night photography vs. human night vision: what does color accuracy mean in this situation?
• DeltaE after Auto White Balance: content-dependence of "grayworld" and "whiteworld" methods requires a more carefully designed experiment ($\Delta E$ calculation uses macbeth chart scene, but we want to check the white balance transform generated from a different scene).

References

Daxter, Donald; Frederic Cao; Henrik Eliasson; Jonathan Phillips. "Development of the I3A CPIQ spatial metrics." Web. 8 Mar 2014. <http://proceedings.spiedigitallibrary.org/data/Conferences/SPIEP/64097/829302_1.pdf>.

Appendix I

Multispectral Scenes Used

• Included with ISET:
  • Macbeth scene(s)
  • StuffedAnimals_tungsten-hdrs.mat
• LoResFemale6_Cx.mat
• LoResMale4_Cx.mat
• StanfordDish_Cx.mat
• StanfordMemorial_Cx.mat

Upload source code, test images, etc, and give a description of each link. In some cases, your acquired data may be too large to store practically. In this case, use your judgement (or consult one of us) and only link the most relevant data. Be sure to describe the purpose of your code and to edit the code for clarity. The purpose of placing the code online is to allow others to verify your methods and to learn from your ideas.

Appendix II

(for groups only) - Work breakdown. Explain how the project work was divided among group members.