Joelle Dowling

Introduction

Color Calibration is one piece of simulating the image processing pipeline. Without color calibration, or the camera sensor's spectral sensitivity, we would not know if we are accurately simulating the colors of the sensor image.

In past work, accurate simulated spectral quantum efficiencies were achieved. However, the data used to generate this model is tedious and costly to gather. Previously, the data would be collected by illuminating a surface with many different monochromatic lights. Monochromatic lights are expensive and retaking measurements for each one takes a lot of time. In this project, the data is collected by illuminating the surface with 3 different illuminants. Having this simpler data allows for quicker and cheaper data generation.

The purpose of this project is to model the spectral quantum efficiency (QE) of a camera sensor. Since we do not know the spectral transmittance of the optics and filters in the camera, we take all the wavelength dependency to lie in the sensor. Accurately modelling this can help in validation on other projects. In this project we will model the spectral QE, then test it on new data.

Background

There are a couple of important concepts to understand before going into the setup of the project.

Spectral Quantum Efficiency

Spectral Quantum Efficiency of a sensor is the number of electrons emitted in the sensor per photon absorbed and is typically a function of wavelength. Since we do not know the wavelength characteristics of the optics and filters in the camera, we assume that all the wavelength dependency occurs at the camera sensor. Therefore, we have one spectral sensitivity function.[1] This function relates the surface reflectance (spectral radiance measurements) and sensor response (raw images). To setup the equation, we'll let n be the number of sampled wavelengths and m be the number of patches on the MCC. Let the surface reflectance measurements be in a n x m matrix, M. Let the sensor response, which is the RGB values, be in a m x 3 matrix, R. And let the spectral quantum efficiency be the n x 3 matrix, S. We can relate these matrices by the following linear equation:

R = M'S

We are given R and M. So we need to solve for S.

S = inv(M') R

Overfitting

Overfitting is a phenomena where the model fits the training data so well, that it may not be able to fit to new datasets. In other words, the model will accurately predict results for the training data, but will perform much worse on new datasets. [2]

Singular Value Decomposition (SVD)

Singular Value Decomposition (SVD) is the factorization of a matrix into three different matrices, as shown below:

R = UDV'

U and V are the left and right singular vectors, respectively. D is a diagonal matrix of singular values.[3][4] SVD is a direct way of completing a principal component analysis of a dataset. It is a helpful technique to understand variation in a dataset and to approximate a high-dimensional matrix with lower-dimensional matrices.

Methods

In this section, the data and instruments used in the project are explained, as well as the techniques.

Data and Instrumentation

There are two important types of data that are necessary for this project, spectral radiance measurements and raw camera images. To gather the data, a Macbeth ColorChecker is illuminated by one of three illuminants, Tungsten (A), Cool White Fluorescent (CWF), and Daylight (DAY). The radiance measurements can then be taken using a spectrophotometer (in this case the Photoresearch spectrophotometer, model PR670 was used). This is considered the input of the camera. The Google Pixel 4A is used to capture the camera output (i.e. the images). This phone uses the Sony IMX363 sensor, which is well documented. The sensor's spectral sensitivity data was previously published by Sony.

ISETCAM

The Image Systems Engineering Toolbox for Cameras is an educational tool which can simulate certain aspects of image systems, such as sensor and display models.[1] In this project, we primarily use it to extract and understand the data. Additionally, the Sony IMX363 sensor's spectral QE has already been input into ISETCAM.

Making the Spectral QE Model

This project utilizes multiple methods to improve the spectral quantum efficiency model.

Simple Linear Equation

To initially solve for the spectral quantum efficiency matrix, the linear equation explained in the previous section was used. However, doing this alone is not sufficient to get an accurate model. This is the case, in part, because the radiance data was generated by measuring the radiance of MCC. Our data technically has 72 samples (24 per MCC x 3 illuminants), but the patches are not independent. Singular value decomposition (SVD) can be used to obtain the principal components of our samples, which results in less than 10 independent measurements. Since we are trying to get the spectral QE information for 31 wavelengths (400:10:700 nm), we are heavily under-sampled.

Furthermore, simply solving the linear equation will create an overfitted model. One way of fixing this is using only a few of the most important principal components. This is discussed in the next section. We can use the results of the SVD to make a new basis and represent the spectral QE as a weighted sum of the new basis. Lastly, if we believe that the camera's spectral QE is similar to Sony's characterization of the

Lower Dimensional Model

There are multiple ways of solving for the principal components of the radiance data. In this work, Singular Value Decomposition (SVD) was used. Performing SVD on the radiance data outputs 3 matrices, U, S, V. We can use D to calculate the variance of the dataset. We use the Matlab cumsum function to find the cumulative sum of the elements along the diagonal of D. We then divide the cumulative sum by the sum of D. When this operation equals 1, all of the variation of the dataset is within that subset of the data. When this is close to zero, it means that very little of the variation is represented in that subset of the data. When performing this operation on the radiance data, we get the plot shown in Figure 1. We can see that as more columns of D are included in the sum, the proportion of variation is almost 1 when about 10 columns of D are included in the sum. To prevent overfitting, we want to include the principal components that account for less than 0.95 of the variation. In our case, this is with less than 8 principal components.

Figure 1: Variance Explained

Best Linear Fit

Now that we know how many principal components to include in our low-dimensional model, we can represent it as a weighted sum of those components. We just have to solve for the weights:

spectralQE = Bw

RGB = radiance' * spectralQE = radiance' * Bw

w = pinv(radiance' * B) * RGB

References

1. Digital camera simulation (2012). J. E. Farrell, P. B. Catrysse, B.A. Wandell . Applied Optics Vol. 51 , Iss. 4, pp. A80–A90
2. Overfitting Definition
3. Singular value decomposition. (2020, November 09). Retrieved November 27, 2020
4. Peng, R. (2020, May 01). Dimension Reduction.Retrieved November 27, 2020