Deep Learning for Illuminant Estimation

Authors: Xuerong Xiao, Jennifer Li

Introduction

Our project aims to estimate the illuminant of a scene using various methods of machine learning with a focus on the deep learning method, convolutional neural networks (CNN). Illumination estimation is an area of interest, because it has applications in topics including image reproduction and image retrieval. For image reproduction, an image may be captured under a certain lighting but rendered under a different lighting. In image retrieval and computer vision, the illumination of the scene needs to be estimated so that it can be accounted for when images of objects are obtained under different lighting.

In past research on illuminant estimation, various methods have been used to recover the illuminant. These methods include gamut mapping and random forest, an ensemble machine learning method, and are described in more detail in the Background section.

Background

In related literature, gamut mapping has been used to recover the illuminant of the scene. [1] In this method, the gamut of each illuminant is precomputed. The gamut is the range of colors that are possible to be displayed. And each gamut of an illuminant is precomputed using a database of measured reflectance spectra. Then, the sensor values obtained for each image are compared to the illuminant gamut using a correlation coefficient. This method was able to classify black body radiator illuminants from 2500K to 8500K correctly to within a few hundred degrees Kelvin.

A drawback of this method is the precomputation of all the illuminant gamuts. This can be resolved by directly comparing feature vectors of images through machine learning. The paper, "Illuminant Classification Based on Random Forest" [2] uses the ensemble machine learning method of random forests to estimate the illuminant. In this method, a random feature vector of the image is used to create decision branches of a decision tree used to classify the illuminant of the image. The results of this research were measured in terms of angular errors, which is the difference between a ground truth color vector of the illuminant and an estimated color vector. These angular errors were compared to those of gamut mapping, and the results were similar.

Another literature of work brings up an interesting application of detecting digital fogery using illuminant estimation [3]. In the method used, the image is segmented into regions of similar chromaticity and the differences in these regions are analyzed. The dissimilarity between objects across region boundaries are compared to the dissimilarty between neighboring objects in that region to detect whether the two regions are under different illuminants. If regions in the image are estimated to be lighted under differing illuminants, this could indicate a digital forgery.

In our method, we have chosen to use machine learning algorithms, like in [2] to estimate the illuminant, so that we can focus on feature extraction and the learning algorithm without the steps of precomputing gamuts or separating the image into regions.

Methods and Results

Image Processing Pipeline

For our machine learning methods, we first need to generate sets of images, which will be used to train and test our learning algorithms. The pipeline for generating our images is shown below.

In the first step, we obtain images from available databases. A database we used is the Caltech101 database, which contains 9142 different images. We then use ISET to render each of these images under the the different illuminants we have chosen to classify. These are the flourescent illuminant and nine black body radiation illuminants of temperatures from 2000K to 10000K. This generates 91420 images in total using the caltech database. These images are also rendered under a camera simulation. The parameters including the optics and sensors of the camera were set in ISET as well, and white balancing was turned off, since we do need the effects of the illuminant to show through. These images are then ready to be split in to the training, validation, and test data for our machine learning algorithm.

Support Vector Machine Using Bag of Features Exploration

Besides our main focus of the CNN method, we also wanted to explore other machine learning algorithms including support vector machines (SVM). For this method, we only managed to run a test on a small directory of 350 images just to have general idea of the results.

In the SVM learning method, a point to be classified is represented as a p-dimensional vector. The elements of these vectors are made up of the features of the point, in our case the image. A classifier that can be thought of as a (p-1)-dimensional hyperplane, is used to separate the different points into classes. There are many ways to construct this classifier, and the SVM uses the assumption that the classifier that separates the points of different classes such that the distance from the hyperplane to the points closest to the hyperplane is maximized. These closest points are known as the support vectors. Using this method, a classifier is constructed and used to separate the points into different classes.

The following image (obtained from Wikipedia) illustrates these concepts for classifying points with two dimensional feature vectors into two classes.

Here, H₃ is the best classifier line for SVM, because it separates the two classes and maximizes the distance between the support vectors and the line.

In the case of illuminant classification, we can use the SVM classifier built into the MATLAB machine learning toolbox, and we need to choose which set of features to extract from the image. Some examples include the Histogram of Oriented Gradients (HOG) features of an image. Another that we decided to explore was the Bag of Features (BOF) feature extraction method. The BOF method utilizes the Speeed Up Robust Features (SURF) of the image. We decided to try out these features, since they were readily available in the MATLAB toolbox, but as explained later, they did not produce good results.

To implement the SVM on our image classification issue, we used a small database of images and split them into the illuminant classes. For each class, we extracted the bag of features from the training data set and input these features and the corresponding illuminant class labels of each image into the SVM classifier. We then evaluated the bag of features on the training data set and on the test data set. The training set consisted of 60% of the images and the test consisted of the remaining 40%.

SVM Bag of Features Results

The following show the confusion matrices of the results of the SVM classifier using Bag of Features. (The illuminant class names have been abbreviated). The first matrix shows the result when the classifier is used on the data used to train it. As expected, the accuracy is high. However, the second matrix shows the results of the classifier used on the test data. We see a case of overfitting, since the accuracy of the classifier on the test data is extremely low.

Classifier used on training data.

Classifier used on test data.

This shows that SVM with BOF was not a useful algorithm to use to solve this problem. BOF should be more suited to object recognition in scenes, which is not exactly the same problem as identifying scene illuminants.

Conclusions and Future Work

...

In terms of SVM algorithms, besides increasing the dataset size, further work can be done to explore what types of feature vectors are best used for this application. We found that the Bag of Features method did not work well with illuminant classification. We also believe Histogram of Oriented Gradients would not work very well, since these are mostly good at finding edges of images. Since we are classifying illuminants, the range of colors would be a better feature space to focus on. Perhaps different color features like color histograms could be used along with the SVM to estimate the illuminant.