Joelle Dowling

Introduction

Color Calibration is one piece of simulating the image processing pipeline. Without color calibration, or the camera sensor's spectral sensitivity, we would not know if we are accurately simulating the colors of the sensor image.

In past work, accurate simulated spectral quantum efficiencies were achieved. However, the data used to generate this model is tedious and costly to gather. Previously, the data would be collected by illuminating a surface with many different monochromatic lights. Monochromatic lights are expensive and retaking measurements for each one takes a lot of time. In this project, the data is collected by illuminating the surface with 3 different illuminants. Having this simpler data allows for quicker and cheaper data generation.

The purpose of this project is to model the spectral quantum efficiency (QE) of a camera sensor. Since we do not know the spectral transmittance of the optics and filters in the camera, we take all the wavelength dependency to lie in the sensor. Accurately modelling this can help in validation on other projects. In this project we will model the spectral QE, then test it on new data.

Background

There are a couple of important concepts to understand before going into the setup of the project.

Spectral Quantum Efficiency

Spectral Quantum Efficiency of a sensor is the number of electrons emitted in the sensor per photon absorbed and is typically a function of wavelength. Since we do not know the wavelength characteristics of the optics and filters in the camera, we assume that all the wavelength dependency occurs at the camera sensor. Therefore, we have one spectral sensitivity function.[1] This function relates the surface reflectance (spectral radiance measurements) and sensor response (raw images). To setup the equation, we'll let n be the number of sampled wavelengths and m be the number of patches on the MCC. Let the surface reflectance measurements be in a n x m matrix, M. Let the sensor response, which is the RGB values, be in a m x 3 matrix, R. And let the spectral quantum efficiency be the n x 3 matrix, S. We can relate these matrices by the following linear equation:

R = M'S

We are given R and M. So we need to solve for S.

S = inv(M') R

Overfitting

Overfitting is a phenomena where the model fits the training data so well, that it may not be able to fit to new datasets. In other words, the model will accurately predict results for the training data, but will perform much worse on new datasets. [2]

Singular Value Decomposition (SVD)

Singular Value Decomposition (SVD) is the factorization of a matrix into three different matrices, as shown below:

R = UDV'

U and V are the left and right singular vectors, respectively. D is a diagonal matrix of singular values.[3][4] SVD is a helpful technique to understand variation in a dataset and to approximate a high-dimensional matrix with lower-dimensional matrices.

Methods

In this section, the data and instruments used in the project are explained, as well as the techniques.

Data and Instrumentation

There are two important types of data that are necessary for this project, spectral radiance measurements and raw camera images. To gather the data, a Macbeth ColorChecker is illuminated by one of three illuminants, Tungsten (A), Cool White Fluorescent (CWF), and Daylight (DAY). The radiance measurements can then be taken using a spectrophotometer (in this case the Photoresearch spectrophotometer, model PR670 was used). This is considered the input of the camera. The Google Pixel 4A is used to capture the camera output (i.e. the images). This phone uses the Sony IMX363 sensor, which is well documented. The sensor's spectral sensitivity data was previously published by Sony.

ISETCAM

The Image Systems Engineering Toolbox for Cameras is an educational tool which can simulate certain aspects of image systems, such as sensor and display models.(1) In this project, we primarily use it to extract and understand the data. Additionally, the Sony IMX363 sensor's spectral QE has already been input into ISETCAM.

Making the Spectral QE Model

This project utilizes multiple methods to improve the spectral quantum efficiency model.

Simple Linear Equation

To initially solve for the spectral quantum efficiency matrix, the linear equation explained in the previous section was used. However, doing this alone is not sufficient to get an accurate model. This is the case, in part, because the radiance data was generated by measuring the radiance of MCC. Our data technically has 72 samples (24 per MCC x 3 illuminants), but the patches are not independent. Singular value decomposition (SVD) can be used to obtain the principal components of our samples, which results in less than 10 independent measurements. Since we are trying to get the spectral QE information for 31 wavelengths (400:10:700 nm), we are heavily under-sampled.

Furthermore, simply solving the linear equation will create an overfitted model. One way of fixing this is using only a few of the most important principal components. This is discussed in the section. We can use the results of the SVD to make a new basis and represent the spectral QE as a weighted sum of the new basis. Lastly, if we believe that the camera's spectral QE is similar to Sony's characterization of the

Lower Dimensional Model

There are multiple ways of solving for the principal components of the radiance data. In this work, Singular Value Decomposition (SVD) was used. Performing SVD on the radiance data outputs 3 matrices, U, S, V. We use U to calculate the variance, as explained in the Background Section.

References

1. Digital camera simulation (2012). J. E. Farrell, P. B. Catrysse, B.A. Wandell . Applied Optics Vol. 51 , Iss. 4, pp. A80–A90
2. Overfitting Definition
3. Singular value decomposition. (2020, November 09). Retrieved November 27, 2020
4. Peng, R. (2020, May 01). Dimension Reduction.Retrieved November 27, 2020