Deep Learning for Illuminant Estimation

Authors: Xuerong Xiao, Jennifer Li

Introduction

Our project aims to estimate the illuminant of a scene using various methods of machine learning with a focus on the deep learning method, convolutional neural networks (CNN). Illumination estimation is an area of interest, because it has applications in topics including image reproduction and image retrieval. For image reproduction, an image may be captured under a certain lighting but rendered under a different lighting. In image retrieval and computer vision, the illumination of the scene needs to be estimated so that it can be accounted for when images of objects are obtained under different lighting.

In past research on illuminant estimation, various methods have been used to recover the illuminant. These methods include gamut mapping and random forest, an ensemble machine learning method, and are described in more detail in the Background section.

Background

In related literature, gamut mapping has been used to recover the illuminant of the scene. [1] In this method, the gamut of each illuminant is precomputed. The gamut is the range of colors that are possible to be displayed. And each gamut of an illuminant is precomputed using a database of measured reflectance spectra. Then, the sensor values obtained for each image are compared to the illuminant gamut using a correlation coefficient. This method was able to classify black body radiator illuminants from 2500K to 8500K correctly to within a few hundred degrees Kelvin.

A drawback of this method is the precomputation of all the illuminant gamuts. This can be resolved by directly comparing feature vectors of images through machine learning. The paper, "Illuminant Classification Based on Random Forest" [2] uses the ensemble machine learning method of random forests to estimate the illuminant. In this method, a random feature vector of the image is used to create decision branches of a decision tree used to classify the illuminant of the image. The results of this research were measured in terms of angular errors, which is the difference between a ground truth color vector of the illuminant and an estimated color vector. These angular errors were compared to those of gamut mapping, and the results were similar.

Another literature of work brings up an interesting application of detecting digital fogery using illuminant estimation [3]. In the method used, the image is segmented into regions of similar chromaticity and the differences in these regions are analyzed. The dissimilarity between objects across region boundaries are compared to the dissimilarty between neighboring objects in that region to detect whether the two regions are under different illuminants. If regions in the image are estimated to be lighted under differing illuminants, this could indicate a digital forgery.

In our method, we have chosen to use machine learning algorithms, like in [2] to estimate the illuminant, so that we can focus on feature extraction and the learning algorithm without the steps of precomputing gamuts or separating the image into regions.

Methods and Results

Image Processing Pipeline

For our machine learning methods, we first need to generate sets of images, which will be used to train and test our learning algorithms. The pipeline for generating our images is shown below.

In the first step, we obtain images from available databases. A database we used is the Caltech101 database, which contains 9142 different images. We then use ISET to render each of these images under the the different illuminants we have chosen to classify. These are the flourescent illuminant and nine black body radiation illuminants of temperatures from 2000K to 10000K. This generates 91420 images in total using the caltech database. These images are also rendered under a camera simulation. The parameters including the optics and sensors of the camera were set in ISET as well, and white balancing was turned off, since we do need the effects of the illuminant to show through. These images are then ready to be split in to the training, validation, and test data for our machine learning algorithm.