Introduction

A beautifully rendered image on a computer screen or cell phone is the result of complex algorithms, careful measurements, intrinsically elegant machinery, and hard work. Designers must take into account the limitations and brilliance of the human visual system in order to produce an outcome that looks as close to the real scene as possible. Through a variety of processes, accounting for different technical limitations as well as human-related issues, a vivid replica is created for viewing delight. One of these steps is that of creating color constancy (or chromatic adaptation)-- or specifically, mimicking the human visual system’s ability to perceive the color of an object or a scene of objects as identical, no matter what the illumination on the object truly is (Gevers & Gijsenij, 2011). This feature of the human visual system is necessary to correctly identify features of objects. For example, an apple viewed under the fluorescent light of a kitchen is red, but the same apple is also red when viewed in daylight.

Specifically, my project dealt with altering the illumination of a painting (thereby setting up the problem in XYZ space with a known illuminant), and attempting to create color constancy with a variety of methods in order to find the most closely depicted replica to a direct rending of the image under a preferred light source. For instance, my preferred light source was D65, or daylight, and I changed the illumination of the image to fluorescent, and attempted to use two different transforms to perform color balancing on the resulting image. These transformations occurred on a hyperspectral image of a subset of “Virgin, Child and St. John,” a painting by 15th century Italian artist Jacopo del Sellaio, which is currently on display at the Cantor Art Center.

Methods

Visual Representation of the Project Process

Changing the illuminant of an image is relatively simple – one needs only to apply a linear transform of the color matching transforms. A possibly more interesting component is creating color balance once one has an estimate of the illuminant of the scene. All calculations were completed in the XYZ space. In this project, I set the illumination on the Sellaio Face image to one of six different lights (D50, D75, Fluorescent, Fluorescent11, FluorescentOffice, and Tungsten); I then created four different transforms to attempt color constancy/balancing from my known illuminant into my preferred illuminant. Two were created using variations of my naive Simple White Point Scaling, and two were created using the more complex Full Image Scaling. After the transform was completed, the XYZ values were then converted into RGB values for display. My goal was an color balanced image under D65, or sunlight. The transformed image XYZ values were analyzed using the Delta E value to find the best match, both across five points in the image (for my own curiosity) and across the full image.

Matlab and ISET were used for the computations. The image was a small selection from an hyper spectral image received from Dr. Joyce Farrell.

Simple White Point XYZ Scaling

The creation of color constancy is possible through a variety of methods. In a simple, semi-naive, first attempt, I created a very simple transform to design an easy diagonal transform. Taking a cue from the Psych 221 color balancing lecture and a variation on the Von Kries Transform, I sampled the XYZ values of a white point in the image under the current illumination and the illumination into which I wished to convert (Color Balance). These points were selected as one of the mostly like to demonstrate a difference between the two illuminant conditions. I then created a direct ratio between the XYZ values to create the transform:

${\begin{bmatrix}X_{1}/X_{2}&0&0\\0&Y_{1}/Y_{2}&0\\0&0&Z_{1}/Z_{2}\end{bmatrix}}$

where $XYZ_{1}$ refers to the XYZ value of the white point under the current illuminant and $XYZ_{2}$ refers to the XYZ value of the white point under the preferred illuminant. This is labeled the “White Conversion.” For thoroughness, I also created a full 3x3 matrix using a sample of white points to create a more rich transformation. However, after completing the analysis of my results, I feel that my creation technique of the 3x3 matrix in this condition was ultimately flawed. It was created by sampling several white points to create the matrix, but this only served to propagate the error in the original sampling, make the effects more obvious, and resulting is worse Delta E values.

Full Image XYZ Scaling

In addition to the white point conversion, I also used a script (copyright by ImagEval Consultants) that created a transform by using the entirety of the images in relation to one another instead of simply using single points. These transforms were created both in the full 3x3 form and just the diagonal for comparisons. Specifically this script solved for the transformation, L, of one XYZ into another, satisfying:
${\begin{bmatrix}X_{2}\\Y_{2}\\Z_{2}\end{bmatrix}}={\begin{bmatrix}X_{1}\\Y_{1}\\Z_{1}\end{bmatrix}}{\begin{bmatrix}L\end{bmatrix}}$

The transform may be found by taking the inverse of the first XYZ matrix and multiplying against the second. This creates a full 3x3 transformation. One may find a similar diagonal by simply computing the multiplication column by column.

Implementation

In order to complete this project, several functions were written to create the transforms, do the transformation, and evaluate the quality of the resulting image. The transformations were created via four scripts, one for each type of scaling, and within each scaling both full and diagonal transformations. The transformation script itself simply applies the transformation to the XYZ value for each pixel on the screen and assembles a new version. The result was evaluated using a delta E calculation both by comparing various single points across the image and the image as a whole. I also took a look at the predicted ranges of XYZ values according to the transform.

For completeness, the following scripts and functions were created (or altered) and used in the implementation. These are found in the attached code archive:

- - Please note, these functions are specific to the hyperspectral image of SellaioFace1.

PerformXYZTransform.m: Takes the transform, L, and the XYZ of the image that needs to be color balanced. It performs the transformation, and returns the XYZ values of a new balanced image.

createDiagonalInnocentTransform.m: Takes the XYZ values of the image we have and the image that we want (created using the knowledge of both the current illumination and the wanted one). It returns a transform that is created by making a diagonal of the rations of the XYZ values of a white point in the image.

createFullInnocentTransform.m: This function takes the XYZ values of the image we have and the image we want, creating a transform from the inverse multiplication of the XYZ values on one to the other.

CreateFullImageXYZTransform.m: This is ImagEval Consultants’ s_XYZsceneIlluminantTransforms.m slightly altered to be a function that accepts as a parameter the current type of lighting on the painting in the form of a filename. The output is a transform created by using the entire Sellaio image under both lights and solving the

${\begin{bmatrix}X_{2}\\Y_{2}\\Z_{2}\end{bmatrix}}={\begin{bmatrix}L\end{bmatrix}}{\begin{bmatrix}X_{1}\\Y_{1}\\Z_{1}\end{bmatrix}}$
equation to find a full 3x3 transformation.

CreateDiagonalImageXYZTransform.m: This is ImagEval Consultants' s_XYZsceneIlluminantTransforms.m slightly altered to be a function that accepts as a parameter the current type of lighting on the painting in the form of a filename. The output is a transform created by using the entire Sellaio image under both lights and solving the

${\begin{bmatrix}X_{2}\\Y_{2}\\Z_{2}\end{bmatrix}}={\begin{bmatrix}L\end{bmatrix}}{\begin{bmatrix}X_{1}\\Y_{1}\\Z_{1}\end{bmatrix}}$
equation to find a column-by-column transformation in the form of a diagonal.

paintingIllumination.m: This is the function that calls all others. In current form, it takes the parameter of the type of light you want to start with in filename form, and displays an image that starts with that type of light and is color balanced for daylight.

getSampleXYZ.m: This a function that takes the XYZ values for a version of the Sellaio Face image and returns 5 points in an array for a point by point calculation of the Delta E values.

getWhiteXYZ.m: This function take the XYZ values of a version of the Sellaio Face image and returns the XYZ values for the example white point in the particular version.

calculateDeltaE.m: This function takes the XYZ value of the image under the known illuminant, the image under the ideal illuminant and the white point to find and return the Delta E values for five separate points in the images.

calculateAllImageDeltaE.m: This function takes the XYZ value of the image under the known illuminant, the image under the ideal illuminant and the white point to find and return the Delta E values for the entire image.

Please note that this project was built upon ISET functions. These include, but are not limited to: sceneSet and related scene manipulation functions, sceneAdjustLuminance, RGB2XWFormat, XW2RGBFormat, vcReadSpectra, and others.

Results

The following are a few graphical representations of the data, as well as the actual rendered images. They were compared against the larger image of the straight transform into daylight, found at the top right of this page. The straight Delta E values are also reported. For scalability and space issues, not all graphs created are embedded. Instead, the code used to generate them has been left intact in the associated functions in the Matlab Code Archive found in Appendix I.

Rendered Image Results

D50 to D65

Transformation D50 to D65 using Full Image Scaling in a Diagonal

Transformation D50 to D65 using Full Image Scaling in a Full 3x3

Transformation D50 to D65 using White Point Scaling in a Diagonal

Transformation D50 to D65 using White Point Scaling in a Full 3x3

D75 to D65

Transformation D75 to D65 using Full Image Scaling in a Diagonal

Transformation D75 to D65 using Full Image Scaling in a Full 3x3. This was the best transformation.

Transformation D75 to D65 using White Point Scaling in a Diagonal

Transformation D75 to D65 using White Point Scaling in a Full 3x3

Fluorescent to D65

Transformation Fluorescent to D65 using Full Image Scaling in a Diagonal

Transformation Fluorescent to D65 using Full Image Scaling in a Full 3x3

Transformation Fluorescent to D65 using White Point Scaling in a Diagonal

Transformation Fluorescent to D65 using White Point Scaling in a Full 3x3

Fluorescent11 to D65

Transformation Fluorescent11 to D65 using Full Image Scaling in a Diagonal

Transformation Fluorescent11 to D65 using Full Image Scaling in a Full 3x3

Transformation Fluorescent11 to D65 using White Point Scaling in a Diagonal

Transformation Fluorescent11 to D65 using White Point Scaling in a Full 3x3. Note the yellow tinge — this was the worst transformation.

FluorescentOffice to D65

Transformation FluorescentOffice to D65 using Full Image Scaling in a Diagonal

Transformation FluorescentOffice to D65 using Full Image Scaling in a Full 3x3

Transformation FluorescentOffice to D65 using White Point Scaling in a Diagonal

Transformation FluorescentOffice to D65 using White Point Scaling in a Full 3x3

Tungsten to D65

Transformation Tungsten to D65 using Full Image Scaling in a Diagonal

Transformation Tungsten to D65 using Full Image Scaling in a Full 3x3

File:Tung, My, Dia.png Transformation Tungsten to D65 using White Point Scaling in a Diagonal

Transformation Tungsten to D65 using White Point Scaling in a Full 3x3

Delta E Calculations

Full Image

Calculated using the directly rendered image, with one Delta E for the entire image:

White Point, Diagonal

D50 - 1.0277
D75 - 0.6308
Fluorescent - 2.5516
Fluorescent11 - 1.8634
FluorescentOffice - 3.2820
Tungsten - 2.5934

White Point, Full

D50 - 1.1639
D75 - 0.5742
Fluorescent - 2.2295
Fluorescent11 - 4.9412
FluorescentOffice - 3.1519
Tungsten - 3.1640

Full Image, Diagonal

D50 - 1.0277
D75 - 0.5181
Fluorescent - 2.5516
Fluorescent11 - 1.8634
FluorescentOffice - 3.2820
Tungsten - 2.5934

Full Image, Full

D50- 0.2788
D75 - 0.1245
Fluorescent 1.4601
Fluorescent11 - 1.0776
FluorescentOffice - 1.6314
Tungsten - 0.7635

Example of the Five Data Points

An example of the differences between the five points (sampled across the diagonal of the image). This condition was just for my own curiosity, because I wanted to see what the individual effect of the diagonal versus the full truly was:

Full Image, D50, FULL 0.0370 0.3023 0.1412 0.2058 0.4065
Full Image, D50, DIAGONAL 0.8687 0.3923 0.5996 0.3599 2.2083

Graphical Representations

Comparison of the Whole Image Delta E, the Y-axis representing Delta E Value

Conclusions

My computationally simple transform worked reasonably well with lights close to daylight, but it was not close in the realm of fluorescents or tungsten. This is interesting, as the naive transform is so computationally easy. However, the Delta E values do not lie, and the overall champion is obviously the Full Image scaling, as one would expect. Using the full image to create the transform yielded the best results, with Delta E values less than one, making the image undistinguishable from the direct image.

The best image resulted from a transform from D75 to daylight. The worst generally arose from the Fluorescent range. I found it very interesting how close we could bring the image using the Full Image scaling to the true rendering — we can get very close, but without accounting for the sensor differences and the monitor differences, I did not manage a statistically trivial difference.

The differences between using a diagonal versus a full 3x3 matrix offered tradeoffs. The diagonal is more easily obtained, with only three unknowns, but it offered worse scaling across the board in the Full Image scaling. The more complex 3x3 matrix cut down dramatically in the error, but is obviously more difficult to find, with nine unknowns. I have learned, as in all computing, the choice of a transform would rest on what is technologically possible, what capabilities the system needs, and how robust the solution must be.

Curiously, the full transform outperformed the diagonal in the Full Image Scaling, but resulted in a worse result in the Simple White Point Scaling. I believe this problem came from my naive approach at creating the full 3x3 matrix in the Simple White Point condition. My formulation involved sampling several white points, which I believe only served to propagate any error already arising from the sampling of one white point. If I were able to do the project again, I would most certainly formulate this transformation condition in another way.

Above all, I feel I understand color constancy on an entirely different level — plus, I feel a great deal of respect for those that design visual systems in the industry. Before Psych 221 and this project, I never would have been able to conceive of the work that has led to the beauty that is the camera phone. I have definitely learned not to take for granted a well rendered image.

If There Were More Time

If I could keep working on the project, I think it would be fascinating to look at what type of information the hyperspectral data could add to the color constancy effect. I also would love to look at the first step in apply the color constancy process — specifically, identifying an unknown illuminant. Additionally, once this were completed, I think it could be fun to attempt a full color balancing effect by also estimating the differences created by the monitor and the camera sensor.

References

Chromatic Adaptation. In Wikipedia. Retrieved from: http://en.wikipedia.org/wiki/Chromatic_adaptation
Color Balance. In Wikipedia. Retrieved from: http://en.wikipedia.org/wiki/Color_balance#Von_Kries.27s_method
Gevers, T., & Gijsenij, A. (2011) Color Constancy. Retrieved from: http://colorconstancy.com/
Psych 221 Lecture notes (2012)

Additionally, I would like to extend special thanks to Dr. Joyce Farrell for all her help on the project. It was very much appreciated.

All code was built in Matlab upon ISET functions.

Appendix I

The Matlab Code Archive: File:Code-Psych221 2.zip

Note that the image of the Sellaio face data is included in the zip file for convenience purposes only. One would need to adapt the source load code for individual use.

Just In Case (Slides): Image, Illuminant Changes, and Color Constancy

The code in the code archive is described above in the implementation section. In addition, there are comments at the beginning of every function to describe its purpose.