NickNoahMegan

Introduction

The goal of our project was to gain an understanding of the modern state of demosaicing techniques and how to reduce demosaicing artifcats. We explore existing algorithms to discover their strengths and weaknesses, and implement some of them to improve our understanding. We also look into the future possibilities for the ISP by implementing the demosaicing portion of the DeepISP paper.

Background

Digital cameras use Color Filter Arrays (CFAs) such as a Bayer Filter to capture colored image data of the light entering the aperture. The image sensor is equally sensitive to any color of light, so a CFA blocks out all but one of the RGB colors so that we can capture a particular band of wavelengths at each pixel. This means each pixel only contains an R G or B value. We must fill in the blanks so every pixel contains all 3 color values. This is done through some form of interpolation.

The absolute simplest approach is Nearest Neighbor, which just copies the value adjacent pixel of the same color channel. This is one of the less desirable approaches as it doesn’t take into account how colors should change across pixels. The most basic usable approach is Bilinear Interpolation. This method averages the adjacent pixel values of the same color channel.

Artifacts are any mistakes that occur as a result of interpolation. The two most common artifacts for Bayer Filters are false color artifacts and zippering. False color artifacts occur when unnatural colors are seen along edges in images where interpolation occurs across and edge instead of along it, creating colors that were never in the original image. Zippering also occurs along edges, and results in blurry edges at high spatial frequencies. Because edges cause the most artifacts, typically the best algorithms are the ones that are able to correctly interpolate along instead of across edges.

The nearest neighbor and bilinear interpolation algorithms both create a lot of artifacts, but there are various more advanced algorithms to try and reduce the amount of artifacts. Some examples include:

Variable Number of Gradients (VNG): Detects edges along some number of possible angles to figure out the best direction to interpolate from

Patterned Pixel Grouping (PPG): An improvement of VNG that additionally takes into account smooth hue transitions to reduce color artifacts

Adaptive Homogeneity Directed (AHD): Even more advanced algorithm that generates a homogeneity map to pick a direction of interpolation that minimizes color artifacts. Considered by some to be the industry standard

We consider these to be the "classical" approaches to demosaicing because they all follow the same idea of using heuristics to make good decisions on how to interpolate to minimize artifacts. There also exists another class of techniques to reduce the demosaicing artifacts after interpolation. These methods use machine learning to learn good transformations rather than applying good heuristics. Some examples include:

K-Nearest Neighbors: Create image patches to form a dataset, then adding detail within a patch by choosing the the K closest patches (determined by L2 norm) and adding them into the patch.

Linear Regression: We model a missing pixel value within a color channel as being a linear function of the surrounding pixels, then learn the model which minimizes some error metric using an optimization algorithm.

Neural Networks: We can create a non-linear model for our transformation by composing multiple simpler non-linear models in sequence. We can then learn a good model using a loss metric and numerical optimization.

For our project, we used a demosaicing dataset provided by Microsoft Research Cambridge. This data set was designed for machine learning demosaicing, and includes test and verification sets of low resolution images to learn a model and verify results. It includes images with and without noise. The results are verified using the provided ground truth images, which were generated by taking by downsampling the output of the camera in such a way as to reduce aliasing and other unwanted effects. To reintroduce noise into the dataset, they use the commonly accepted noise models that combine a poisson and gaussian distribution. The researches fit the noise model to the input images, and then applied an equal amount of noise to the downsampled images.

Methods

For our classical algorithms, we found the common approaches between them to be as follows. Firstly, they would make the assumption that hue changes in a natural image are subtle, and that we can smooth the hues in the output to try and recreate this expectations. Secondly, they would use luminance gradients to detect edges and then use that information to interpolate along edges instead of across them. Both of these approaches make assumptions about the picture being "natural," meaning that they're pictures we would typically expect to see in consumer photography. Artificially generated images may very well violate all our assumptions, but the importance of these assumptions is that they are very accurate for consumer photography and so can be used heuristically to improve our results.

The first algorithm we attempted to implement is Paired Pixel Grouping (PPG). The details for how this algorithm works can be found at this link