NickNoahMegan

Introduction

The goal of our project was to gain an understanding of the modern state of demosaicing techniques and how to reduce demosaicing artifcats. We explore existing algorithms to discover their strengths and weaknesses, and implement some of them to improve our understanding. We also look into the future possibilities for the ISP by implementing the demosaicing portion of the DeepISP paper.

Background

Digital cameras use Color Filter Arrays (CFAs) such as a Bayer Filter to capture colored image data of the light entering the aperture. The image sensor is equally sensitive to any color of light, so a CFA blocks out all but one of the RGB colors so that we can capture a particular band of wavelengths at each pixel. This means each pixel only contains an R G or B value. We must fill in the blanks so every pixel contains all 3 color values. This is done through some form of interpolation.

The absolute simplest approach is Nearest Neighbor, which just copies the value adjacent pixel of the same color channel. This is one of the less desirable approaches as it doesn’t take into account how colors should change across pixels. The most basic usable approach is Bilinear Interpolation. This method averages the adjacent pixel values of the same color channel.

Artifacts are any mistakes that occur as a result of interpolation. The two most common artifacts for Bayer Filters are false color artifacts and zippering. False color artifacts occur when unnatural colors are seen along edges in images where interpolation occurs across and edge instead of along it, creating colors that were never in the original image. Zippering also occurs along edges, and results in blurry edges at high spatial frequencies. Because edges cause the most artifacts, typically the best algorithms are the ones that are able to correctly interpolate along instead of across edges.