Illuminant estimation and detection using RGB and NIR from HyperSpectral Imaging of Fruits

Introduction

Color constancy is a perpetual inherent ability in the human eye whereas cameras need algorithms.

The human visual system has the ability to sense the color of objects irrespective of the illumination of the object. Usually, for a normal human eye, the color constancy is a perpetual involuntary ability. For example, the human eye perceives a lemon to be yellow in color under day-light and also under an incandescent light bulb. Although, the color signal that is captured (by the human eye or the camera sensor) is a product of the reflective properties of the surface of the object being viewed and the incident light, the eye is able to chromatically adapt (to a certain extent) and thus captures and renders images perfectly. This is not inherently available in a digital camera and thus we need color balancing algorithms to render the images suitable for our vision.

When solving for color constancy, we should first estimate the scene illuminant and then derive its surface reflectance. Estimation of the scene illuminant is very vital for color constancy because only when we have knowledge about the illuminant can we mitigate it's effect on the image captured.

Currently there are different types of color constancy algorithms and can be classified as pixel-based algorithms and edge-based algorithms. Examples of pixel-based algorithms are Gray World and Max RGB and examples of edge-based algorithms are Gray edge and Max edge. Although these algorithms have been accepted widely, they have their own limitations. In this project, I explore the use of near infra-red (NIR) spectrum in estimation of incident illumination and also the probable use in detection of the location of various illuminants in a multiple-lit scene.

An example of how a digital camera captures an image with different illuminants

Current illumination estimation methods

As explained earlier, there are algorithms used for estimation of illuminants. To name a few - Gray World, MaxRGB and Gamut Mapping. The Gray World is one of the simplest algorithms which works on the assumption that the average reflectance from the entire scene is gray. Hence the illuminant color can be estimated by looking at the average color and comparing it with gray.

$Mean(R)=Mean(G)=Mean(B)=1$

The MaxRGB is another extremely simple method of estimating the chromaticity of the scene illumination for color constancy and automatic white balancing based on the assumption that the triple of maxima obtained independently from each of the three color channels represents the color of the illumination.

$Max(R)=Max(G)=Max(B)=1$

In this project, I considered the use of Color by Correlation, apart from the use of NIR, so that this would help us gauge the effectiveness of using NIR and RGB for illuminant estimation [Graham D. Finlayson, Steven D. Hordley, and Paul M. Hubel]. Color by Correlation, like the gamut mapping method, is an algorithm and considered to be more accurate than the above mentioned techniques. But they can only estimate a known illuminant among a set of potential options and are also considered to be too complex. Thus, for color by correlation algorithm, there needs to be prior knowledge about commonly occurring outdoor and indoor illuminants.

Use of the Near-Infrared spectrum

Digital cameras use the visible spectrum (400-700 nm) for capturing images and reproducing them. The sensors of the camera, however, are sensitive to wavelengths ranging from the ultraviolet (200-400 nm) and to the near infrared (700-1100 nm). Clement Fredembach and Sabine Susstrunk from Ecole Polytechnique Federale de Lausanne (EPFL), Switzerland, first suggested the utilization of NIR spectrum in 2009 and predicted that it would enhance the estimation methods greatly.

Use of NIR information literally doubles the "bandwidth", giving us more space to experiment with the scene. Unlike current camera color filters ,which have peak sensitives which are only about 100 nm apart, the near-infrared technique has greater distance and there is a large response variation, with respect to the type of incident light. For example, scattered skylight and fluorescent light do not have emission spectrum in NIR, while incandescent light does. This would help in distinguishing between different light bulbs that have an identical point but different metameric properties.

Single Illuminant Estimation using NIR and RGB values

Since a light source intensity does not alter its color distribution by a significant amount, all algorithms resort to the use of "band-ratios". That is, the algorithms such as Gray World and Max RGB use color ratios for estimation of scene illuminant.

Gray World case:
V_R = ${\frac {Mean(R)}{Mean(G)}}$ ; V_G = ${\frac {Mean(G)}{Mean(B)}}$ ; V_B = ${\frac {Mean(B)}{Mean(R)}}$

MaxRGB case:
V_R = ${\frac {Max(R)}{Max(G)}}$ ; V_G = ${\frac {Max(G)}{Max(B)}}$ ; V_B = ${\frac {Max(B)}{Max(R)}}$

In this project, we use the same ratio principles for the NIR method. The NIR-visible intensity ratios is given by
V_R = ${\frac {R}{NIR}}$ ; V_G = ${\frac {G}{NIR}}$ ; V_B = ${\frac {B}{NIR}}$

Use of B&W 486 and Hoya R72

To use the NIR spectrum from the camera sensor, one has to simply change the filters in it. We can use a B&W 486 UV/IR filter in order to capture the visible wavelengths and a Hoya R72 in order to capture the NIR wavelengths.

Using the color ratios, we can create a 3x3 matrix that is used for a simple transformation of the RGB values. Random points were selected in a scene containing fruits (which has a background of the Macbeth ColorChecker) and creating a direct ratio with the above formula, a transformation was created. With the resulting images, the Δ E values were calculated. This was repeated for different illuminants and for each iteration, we calculate the Δ E values and compare it with the Δ E values that we obtained from the Color by Correlation method (used as a benchmark in this case).

${\begin{bmatrix}R\\G\\B\end{bmatrix}}={\begin{bmatrix}\alpha &0&0\\0&\beta &0\\0&0&\gamma \end{bmatrix}}{\begin{bmatrix}R'\\G'\\B'\end{bmatrix}}$

Also, in this method we try to discriminate among the illuminants instead of estimating them since it allows for a wider range of illuminants to be properly evaluated. In other methods, the ratios are just equated to 1, whereas in the method proposed here, we compare the relationship between these ratios (e.g., is V_G > V_B ?). For a given image, we observe the proportion of ratios that are smaller, greater, much smaller or much greater than 1. With this comparison between the various ratios, the illuminant can be easily confirmed.

Results

The results obtained range from comparison of V_R, V_G and V_B values, comparison of the images obtained by using color by correlation method and also the NIR spectrum alone, change of illuminants for the scenes containing fruits (good ones and rotten ones) and calculating Δ E values for all of them.

Comparison of color ratios - V_R, V_G and V_B

Color ratio comparison

The 3 points in the graph are B/NIR, G/NIR and R/NIR from the lowest power distribution value to the highest with respect to the wavelength. If we draw the normalized spectral power distribution for incandescent light used in these experiments and see the difference between both, we can find that they are constant (around 0.28)

Comparison of images under different illuminations

Normal Fruits

Original picture

Image of fruits

Color correlated

Image of fruits in NIR spectrum

Under D65 daylight illumination

Image of fruits under D65

Color correlated under D65

Image of fruits in NIR spectrum under D65

Under tungsten illumination

Image of fruits under tungsten

Color correlated under tungsten

Image of fruits in NIR spectrum under tungsten

Rotten Fruits

Original picture

Image of fruits

Color correlated

Image of fruits in NIR spectrum

Under D65 daylight illumination

Image of fruits under D65

Color correlated under D65

Image of fruits in NIR spectrum under D65

Under tungsten illumination

Image of fruits under D65

Color correlated under D65

Image of fruits in NIR spectrum under D65

Likewise the experiment was performed on changing the illuminants and then the Δ E values were calculated

Comparison of Δ E values for the 2 methods under different illuminant cases

We see that the Δ E values are very small for the Color by correlation method. This is mainly due to the fact that there is prior knowledge about the illuminant and thus is able to match its results accordingly. But the use of NIR has also produced good images and though the Δ E values are not as low as color by correlation, they are reasonably good, less complex and better than those of gray world and MaxRGB. Also these Δ E values are not the same as the one in the presentation as small changes were made to the script after the presentation.

Δ E Comparison for the 2 methods under different illumination

Multiple Illuminants

Example of Multiple Illuminants in the scene

Although we have derived several algorithms for detecting the illumination, all these cases only assume a single illuminant being used in the scene. Be it gray world or gamut mapping or the current method, the basic assumption is that only one prevailing illuminant is present in the scene. However, this is not the case when we capture the image in our camera.

An example to prove that the algorithm proposed and the current conventional algorithms fail to detect multiple illuminants is given in the adjacent picture. One half of the scene is illuminated by skylight which has a color temperature of 10,000°K while the other half of the scene is illuminated by daylight which has a color temperature of around 5500-6500°K. On using the algorithm, Clement Fredembach and Sabine Susstrunk found that the resultant scene had a color temperature of 8000°K. Although this is the average of the scene, it is wrong for both regions.

To mitigate this error, the 2 researchers propose to dissect the images into blocks and assume each block to have only single illuminants. If these blocks overlap, they propose to take an average of those blocks alone.

For each of these blocks, the average NIR/visible ratios and their magnitude is given by : $mag={\lVert V\lVert }^{2}$ where V = [V_R V_G V_B]

Exceptions in using this method

Although this method seems extremely convenient, it might not work well with all scenes. There are complications when it comes to plants (due to chlorophyll), water and also objects with dyes that do not have emission spectras in the NIR band.

Same experiments with random images

Original Image

Color correlated

Near Infrared

Original Image

Color correlated

Near Infrared

Conclusions

The use of NIR, from the year 2009, has opened several avenues not just for illuminant estimation, but also has thrown other interesting problems. It is shown that by simply calculating the ratios of NIR and RGB channels, the scene illuminant can be accurately determined. Experiments were performed only on illuminants and I believe that more knowledge about the reflectances can give us better insight into color constancy. By knowing the behavior of NIR reflectance, a hybrid illuminant estimation algorithm can be derived that exploits the NIR band completely.

References - Resources and related work

Psych 221 lectures by Prof. Brian Wandell
Illuminant estimation and detection using near-infrared by Clement Fredembach and Sabine Susstrunk : http://infoscience.epfl.ch/record/133341/files/IR_ill_est.pdf
Color by Correlation: A Simple, Unifying Framework for Color Constancy by Graham D. Finlayson, Steven D. Hordley, and Paul M. Hubel : http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=00969113
My sincere thanks to Prof. Joyce Farrell who encouraged me and helped me out throughout this project. I also thank Prof. Brian Wandell profusely for the amazing lectures. My thanks go to Henryk Blasinski for his constant support.

Appendix I - Code and Data

Code

Color by correlation work codes are extensively used from past project done by Srinivasa Rangan Sridharan

File:CodeFile.zip

Appendix II - Work partition (if a group project)

Not a group project