Arthur Alaniz, Tina Mantaring

Introduction

As the use of digital cameras becomes more and more widespread, the imaging algorithms that go into these cameras have also become more sophisticated. The typical digital camera uses a processing pipeline to convert raw sensor values to a final image, and the different parts of the pipeline may involve correcting for hardware defects, interpolating color filter array values, removing noise, and performing color space transformations. As more and more focus is being given to this field, the pipeline algorithms are also becoming more and more complex. However, one must ask the question: do these complex algorithms really outperform their simpler, more straightforward counterparts?

Methods

Denoising and Demosaicking

For our project, the primary algorithm we chose for our imaging pipeline was the one presented in the paper “Denoising and Interpolation of Noisy Bayer Data with Adaptive Cross-Color Filters”, by D. Paliy, A. Foi, R. Bilcu, and V. Katkovnik. Their technique performs simultaneous denoising and demosaicking using directional adaptive filters that are based on the concepts of local polynomial approximation (LPA) and intersection of confidence intervals (ICI). Their paper is summarized in this section.

Local Polynomial Approximation

Local polynomial approximation, or LPA, works on the assumption that the data in some local region can be fitted to a polynomial function. The cross-color filters used in the algorithm are linear combinations of LPA smoothing and difference filters that operate on complimentary color channels.

The above figure shows an example of an LPA smoothing filter (left) and an LPA difference filter (right). The combined LPA cross-color filter is shown below, where the differing colors illustrate that the smoothing and difference components work on different color channels.

If we let $g_{m,s,\theta }^{(k)}$ denote a directional 1D convolution kernel, where $m$ is the polynomial order, $s$ is the scale or size of the kernel (kernel width), $k$ is the order of the derivative, and $\theta$ is the direction, then we can write the interpolation kernels ${\bar {g}}_{s,\theta }$ and the denoising kernels ${\tilde {g}}_{s,\theta }$ as follows:

${\bar {g}}_{s,\theta }=(1-\alpha )g_{0,s,\theta }^{(0)}+\alpha g_{1,s,\theta }^{(0)}+\beta g_{1,s,\theta }^{(1)}$

${\tilde {g}}_{s,\theta }=(1-\alpha )g_{0,s,\theta }^{(0)}+\alpha g_{1,s,\theta }^{(0)}+\beta g_{1,s,\theta }^{(1)}$

Note that there are 4 possible directions for the interpolation filters, while the denoising filters use 8 directions. Aside from this, however, the structures for the interpolation and denoising kernels are very similar. Indeed, the primary difference between them is that their component filters operate on different support channels. This is shown in the figure below.

In the figure, (a) illustrates how the green pixel is interpolated at position g(0). The smoothing components of the interpolation filter operate on the green pixels, while the difference component operates on the red pixel. In (b), we see how denoising of the green pixel at g(0) is done. The smoothing components of the denoising filter operate on the green pixels, while the difference component operate on the red pixels. Similarly, when we are denoising a red pixel, as in (c), the smoothing components operate on the red pixels, while the difference component operates on the green pixels. Finally, since the denoising filters have 8 directions, they can operate along diagonals, which is shown in (d). When denoising green pixels along diagonals, we have only green pixels to deal with, and so there is no difference component. However, when we are denoising red or blue pixels, the smoothing component operates on the same channel while the difference component operates on the complimentary color channel.

Since the filters are directional, the origin of the filter is not in the center, but is located to one side. Obtaining the estimate is done by convolving the filter with the input pixels. That is, if $z$ is our noisy CFA data, then for each scale $s_{1}<s_{2}<s_{3}<...<s_{n}$, we obtain the interpolation and denoising estimates ${\bar {y}}_{s,\theta }(x)$ and ${\tilde {y}}_{s,\theta }(x)$ at each pixel position $x$ as follows:

${\bar {y}}_{s,\theta }(x)=({\bar {g}}_{s,\theta }\ast z)(x)$

${\tilde {y}}_{s,\theta }(x)=({\tilde {g}}_{s,\theta }\ast z)(x)$

Intersection of Confidence Intervals

We created the interpolation and denoising filters ${\bar {g}}_{s,\theta }$ and ${\tilde {g}}_{s,\theta }$ for different scales $s$. The goal of ICI is to choose the largest scale for which the local polynomial approximation is still valid. This is done as follows:

First, we find the standard deviations of our interpolation and denoising estimates using some estimate of the standard deviation of the noise. This is done as follows:

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \sigma_{\bar{y}_{s,\theta}}(x) = \sqrt{(\bar{g}_{s,\theta}^2 \ast \sigma^2)(x)}}

$\sigma _{{\tilde {y}}_{s,\theta }}(x)={\sqrt {({\tilde {g}}_{s,\theta }^{2}\ast \sigma ^{2})(x)}}$

In the above formulas, $\sigma$ is the estimated standard deviation of the noise. The paper gives two examples of signal-dependent noise models, and their corresponding standard deviation estimates:

• Poisson noise: ${\hat {\sigma }}={\sqrt {\frac {z}{\xi }}}$, and $\xi =0.5447$
• Nonstationary Gaussian noise with signal-dependent standard deviation: ${\hat {\sigma }}=|k_{0}+k_{1}z|$, and $k_{0}=10,k_{1}=0.1$