NickNoahMegan

Introduction

The goal of our project was to gain an understanding of the modern state of demosaicing techniques and how to reduce demosaicing artifcats. We explore existing algorithms to discover their strengths and weaknesses, and implement some of them to improve our understanding. We also look into the future possibilities for the ISP by implementing the demosaicing portion of the DeepISP paper.

Background

Digital cameras use Color Filter Arrays (CFAs) such as a Bayer Filter to capture colored image data of the light entering the aperture. The image sensor is equally sensitive to any color of light, so a CFA blocks out all but one of the RGB colors so that we can capture a particular band of wavelengths at each pixel. This means each pixel only contains an R G or B value. We must fill in the blanks so every pixel contains all 3 color values. This is done through some form of interpolation.

The absolute simplest approach is Nearest Neighbor, which just copies the value adjacent pixel of the same color channel. This is one of the less desirable approaches as it doesn’t take into account how colors should change across pixels. The most basic usable approach is Bilinear Interpolation. This method averages the adjacent pixel values of the same color channel.

Artifacts are any mistakes that occur as a result of interpolation. The two most common artifacts for Bayer Filters are false color artifacts and zippering. False color artifacts occur when unnatural colors are seen along edges in images where interpolation occurs across and edge instead of along it, creating colors that were never in the original image. Zippering also occurs along edges, and results in blurry edges at high spatial frequencies. Because edges cause the most artifacts, typically the best algorithms are the ones that are able to correctly interpolate along instead of across edges.

The nearest neighbor and bilinear interpolation algorithms both create a lot of artifacts, but there are various more advanced algorithms to try and reduce the amount of artifacts. Some examples include:

Variable Number of Gradients (VNG): Detects edges along some number of possible angles to figure out the best direction to interpolate from

Patterned Pixel Grouping (PPG): An improvement of VNG that additionally takes into account smooth hue transitions to reduce color artifacts

Adaptive Homogeneity Directed (AHD): Even more advanced algorithm that generates a homogeneity map to pick a direction of interpolation that minimizes color artifacts. Considered by some to be the industry standard

We consider these to be the "classical" approaches to demosaicing because they all follow the same idea of using heuristics to make good decisions on how to interpolate to minimize artifacts. There also exists another class of techniques to reduce the demosaicing artifacts after interpolation. These methods use machine learning to learn good transformations rather than applying good heuristics. Some examples include:

K-Nearest Neighbors: Create image patches to form a dataset, then adding detail within a patch by choosing the the K closest patches (determined by L2 norm) and adding them into the patch.

Linear Regression: We model a missing pixel value within a color channel as being a linear function of the surrounding pixels, then learn the model which minimizes some error metric using an optimization algorithm.

Neural Networks: We can create a non-linear model for our transformation by composing multiple simpler non-linear models in sequence. We can then learn a good model using a loss metric and numerical optimization.

For our project, we used a demosaicing dataset provided by Microsoft Research Cambridge. This data set was designed for machine learning demosaicing, and includes test and verification sets of low resolution images to learn a model and verify results. It includes images with and without noise.