PadmanabanVarma

Introduction

Camera technology has been advancing at a fast pace, from giving everyone access to a high-quality cameras in their smartphones to looking beyond 2D images and being able to capture depth information. However, one of the persistent problems that still remains is being able to capture high quality images in low light conditions. As we often see with the images we capture at night with our smart phones, they are usually poor quality and often corrupted by noise that looks like speckles in the image. Specifically, these images are dominated by Poisson shot noise that is inherent to the distribution of photons that are recorded by camera sensors. This photon noise is usually too small relative to the number of photons captured in daylight/bright-light settings and therefore not noticeable. However as photon count drops in dark settings, this noise begins to dominate image formation.

Image denoising is a popular problem that has been studied extensively in the past as a signal denoising problem. However, one of the major drawbacks of these processes is that they focus almost solely on Gaussian noise in images. Gaussian noise is inherently different from Poisson noise in that it is easier to "average out". As described in the next section, there are some methods that look at removing non-Gaussian noise as well, but these methods are very complex and computationally expensive.

Our objective in this project is to find a simple and inexpensive method to be able to denoise existing and new low-light images. We approach this problem via a machine learning perspective, specifically linear regression, to learn a parameter that maps the noisy values of a patch in the image to its center pixel value. Once this parameter is learned, it is easy to use it to denoise previously captured images. Moreover, since the approach denoises a patch at a time, it can be applied to images of varying sizes. With this formulation, the denoising only relies on vector-vector multiplications, which can can be computed efficiently and quickly.

Background

The first attempts at denoising images began with applying techniques from signal processing such as Wiener filtering. However, this relied on the underlying image being smooth, which is not always true for captured images. A more advanced approach looked at transforming the image to the Fourier or Wavelet domain and removing noise by thresholding the coefficients. The two drawbacks of this approach in context of low light image denoising are that it is computationally expensive and it is targeted towards images with Gaussian, not Poisson, noise. A more recent approach applies independent component analysis (ICA) to images in order to denoise them. This procedure works well with non-Gaussian noise but requires multiple frames of the same scene to be able to denoise the image properly. For previously captured images, this is not a viable option.

• Insert other ML focused denoising paper*

Dataset

For the base images of our dataset, we start with the Berkeley Segmentation Data Set [1]. This dataset consists mostly of outdoor images, but includes people, nature and monuments. This ensured that we had a good variety of images to learn from and the learned parameter was not biased towards certain colors or spatial frequencies.

TODO: Nitish

We processed these daytime images to simulate nighttime low-light image captures of the same scenes. We do this using the Image Systems Engineering Toolbox~\cite{iset}. The overall process involves converting from RGB to the number of photons, adjusting their relative counts as if the image had been taken in a daylight color spectrum, scaling the photon counts down, adding Poisson noise, and saving the now dark and noisy image as an RGB image. Specifically, when doing this, we assumed that the daylight is 6500~K blackbody radiation, which matches a sunny day well. Images taken taken in outdoor night settings can look almost exactly like daylight photos with a long exposure~\cite{moonlight}, which informs our decision to create low light images by simply scaling down the photon count.

The above procedure details how to create the noisy training examples. The noise-free values corresponding to these will be generated in much the same way, but excluding the Poisson noise addition step. These noise-free values are used as ground truth for the linear regression algorithm.

TODO: Add bright image from dataset and converted noisy, dark image

Methods

Linear Regression

We chose linear regression to learn for denoising low-light images was linear regression where we predict the RGB values of the "denoised" image based on RGB values of other pixels in the noisy image. Specifically, we learn this denoising parameter per patch - that is, one example consists of the pixel values in a 13x13 patch that is used to predict the value of the center patch. This approach is supported by the assumption that a pixel's color is locally dependent (highly related to and dependent on) the colors of the pixels around it.

Specifically, our features for a single pixel are the following:

• Raw RGB values of the 13x13 patch surrounding the pixel
• Vertical RGB differences across 13x13 patch (resulting in a 13x12 patch)
• Horizontal RGB differences across 13x13 patch (resulting in a 12x13 patch)

Something to note is that the image values were integers between 0 and 255. However, our regression model assumes continuous values. To get around this, we round and clip the values of the final "denoised" image in the prediction step.

Thus, our final model is: a training set $\{(x^{(i)},(y^{(i)});i=1\ldots m\}$ where each $x^{(i)}$ is the pixel information for the patch of pixels surrounding the target pixel, and $y^{(i)}$ is the target pixel; and a linear-regression model mapping from $\mathbb R^{1443}$ to $\mathbb R^{3}$. Also note that the parameters our model produces are in the space $\mathbb R^{3\times1443}$, that is, our parameters model the effect each indivudal feature has (such as the R value of an arbitrarity input pixel) on the RGB value of the target pixel.

In addition to this, as a way of making the model better account for edges, we implement a similar model that instead of learning a single target pixel, instead learns the central $3\times3$ region of pixels. When learning patches containing sharp edges, this new model must account for the large jump in pixel intensity in the $3\times3$ region, unlike the $1\times1$, which ignores such spatial information in the optimization. This increases the dimension by a factor of 9 so the output is in $\mathbb R^{27}$, and the model learns a matrix of parameters in $\mathbb R^{27\times1443}$.

Appendix II

Fig. 1 – Streaming and Augmenting Stereo Camera Images-Pipeline