PadmanabanVarma

Introduction

Camera technology has been advancing at a fast pace, from giving everyone access to a high-quality cameras in their smartphones to looking beyond 2D images and being able to capture depth information. However, one of the persistent problems that still remains is being able to capture high quality images in low light conditions. As we often see with the images we capture at night with our smart phones, they are usually poor quality and often corrupted by noise that looks like speckles in the image. Specifically, these images are dominated by Poisson shot noise that is inherent to the distribution of photons that are recorded by camera sensors. This photon noise is usually too small relative to the number of photons captured in daylight/bright-light settings and therefore not noticeable. However as photon count drops in dark settings, this noise begins to dominate image formation.

Image denoising is a popular problem that has been studied extensively in the past as a signal denoising problem. However, one of the major drawbacks of these processes is that they focus almost solely on Gaussian noise in images. Gaussian noise is inherently different from Poisson noise in that it is easier to "average out". As described in the next section, there are some methods that look at removing non-Gaussian noise as well, but these methods are very complex and computationally expensive.

Our objective in this project is to find a simple and inexpensive method to be able to denoise existing and new low-light images. We approach this problem via a machine learning perspective, specifically linear regression, to learn a parameter that maps the noisy values of a patch in the image to its center pixel value. Once this parameter is learned, it is easy to use it to denoise previously captured images. Moreover, since the approach denoises a patch at a time, it can be applied to images of varying sizes. With this formulation, the denoising only relies on vector-vector multiplications, which can can be computed efficiently and quickly.

Background

The first attempts at denoising images began with applying techniques from signal processing such as Wiener filtering. However, this relied on the underlying image being smooth, which is not always true for captured images. A more advanced approach looked at transforming the image to the Fourier or Wavelet domain and removing noise by thresholding the coefficients. The two drawbacks of this approach in context of low light image denoising are that it is computationally expensive and it is targeted towards images with Gaussian, not Poisson, noise. A more recent approach applies independent component analysis (ICA) to images in order to denoise them. This procedure works well with non-Gaussian noise but requires multiple frames of the same scene to be able to denoise the image properly. For previously captured images, this is not a viable option.

TODO: Add other ML paper for image denoising

Dataset

For the base images of our dataset, we start with the Berkeley Segmentation Data Set [1]. This dataset consists mostly of outdoor images, but includes people, nature and monuments. This ensured that we had a good variety of images to learn from and the learned parameter was not biased towards certain colors or spatial frequencies.

We processed these daytime images to simulate nighttime low-light image captures of the same scenes. We do this using the Image Systems Engineering Toolbox~\cite{iset}. The data set does not provide camera information metadata, and so we convert the RGB image to a scene using our monitors (Apple LCD) as the display calibration SPDs. Then we use sceneAdjustIlluminant to normalize the illuminant in all the images to D65. Next, we set the peak photon count in the images to 10 photons because this visually created a reasonable amount of Poisson noise in testing. This is done using the sceneAdjustLuminance function and by querying sceneGet for the photon count. Next, we use scenePhotonNoise to manually change the photon counts in the scene according to the Poisson distribution, and we modify the scenePhotonNoise function to remove the Gaussian approximation. Lastly, when saving the scene as an RGB image using sceneSaveImage, we modify the code such that it produces a dark RGB output (instead of the default, which rescales the XYZ values so that the RGB image is bright). Earlier, we assumed that the daylight is 6500~K blackbody radiation. We justify this choice because it matches a sunny day well, and images taken taken in outdoor night settings can look almost exactly like daylight photos with a long exposure~\cite{moonlight}, which then informs our decision to create low light images by simply scaling down the photon count.

The above procedure details how to create the noisy training examples. The noise-free values corresponding to these will be generated in much the same way, but excluding the Poisson noise addition step. These noise-free values are used as ground truth for the linear regression algorithm.

TODO: Add bright image from dataset and converted noisy, dark image

Methods

All the methods described below were implemented in MATLAB.

Linear Regression

We chose linear regression to learn for denoising low-light images was linear regression where we predict the RGB values of the "denoised" image based on RGB values of other pixels in the noisy image. We use linear regression of the form J($\theta$)= TODO and learn the parameter $\theta$ via batch gradient descent. Specifically, we learn this denoising parameter per patch - that is, one example consists of the pixel values in a 13x13 patch that is used to predict the value of the center pixel/patch. This approach is supported by the assumption that a pixel's color is locally dependent (highly related to and dependent on) the colors of the pixels around it.

Our features for a single pixel/patch are the following:

• Raw RGB values of the 13x13 surrounding patch
• Vertical RGB differences across 13x13 patch (resulting in a 13x12 patch)
• Horizontal RGB differences across 13x13 patch (resulting in a 12x13 patch)

1 Pixel Interpolation

Our first attempt focused on using the features from a 13x13 patch to predict the RGB values of the center pixel of the patch. In this case, the dimensions of the feature vector, x⁽ⁱ⁾ is (13x13 + 12x13 + 13x12) x 3 = 1443. The dimensions of the output vector, y⁽ⁱ⁾ is 3. The $\theta$ that is learned can be visualized as shown in Fig. 1. We can see that the raw pixel values of a specific color channel are highly correlated with values of the same color channel for neighboring pixels. The next two rows, which represent the pixel difference features, show something interesting - each color channel's filter does depend on information from the other two colors. This suggests that training for one RGB pixel instead of each color channel separately will lead to an improvement in denoising compared to learning a separate parameter per color channel. Moreover, this also suggests that the cross-talk in the color channels for pixel difference can be a factor in helping preserve edges in the images.

TODO: Link Image for Thetas

3x3 Pixels Interpolation

We also tried a slightly edited model of the above - instead of predicting the RGB values of a single center pixel, we predicted the RGB values of the center 3x3 patch. This means that the dimensions of the output vector, y⁽ⁱ⁾ is no (3x3)x3 = 27. We learned another theta parameter, but it was difficult to visualize because of its dimensions. Results from the two approaches are described in the Results section.

Appendix II

Nitish Padmanaban: Conversion of dataset to dark and noisy, dark images. Baseline methods (RGB and LAB Gaussian Smoothing, RGB and LAB Bilinear Smoothing).

Paroma Varma: Background Research, Linear regression methods (1x1 and 3x3 versions).

Fig. 1 – Streaming and Augmenting Stereo Camera Images-Pipeline