BellMoonVassiliev

Back to Psych 221 Projects 2013

Background

Most digital image sensors rely on a color filter array (CFA) in order to sense color, so demosaicking is an important and near-universal part of the imaging pipeline. Demosiacking is by nature error-prone, since it is an attempt to fill in missing data points with educated guesses. However, if there are multiple images of the same scene - whether captured as individual still frames or from a video - then color information which was lost in one image may be captured in another. By performing joint demosaicking on a set of images, we are able to more accurately reconstruct the color of the scene than with a single image.

A second research area within image processing involves combining a series of images of the same scene in order to acheive a higher signal-to-noise ratio (SNR), higher resolution, or both. Work on super-resolution has focused primarily on grayscale data, since RGB data adds several challenges. In particular, the image channels must remain aligned through any filtering operations, or else the colors may be distorted and leave unpleasant fringes. When these multi-image operations are performed after demosaicking, errors from the demosaicking process are treated as "signal" in the following steps. While denoising is often done on raw Bayer images, super-resolution cannot easily done before demosaicking, since the result. In this project, we treat demosaicking, super-resolution, and noise reduction as different aspects of the same problem, which is to use all of the information in the source images to produce a single result which most accurately represents the scene as it would have been captured by an ideal camera.

Methods

Our approach consists of an image alignment step, which finds the relative horizontal and vertical offsets between the LR images, and a demosaicking step which combines the data from LR images into a single HR image.

Image Alignment

• Used phase correlation
• Fit parabola to get subpixel resolution
• Do least-squares fit to find a globally-consistent set of offsets

Demosaicking

Principles from single-image demosaicing

Before directly working with multiple images for demosaicing, we first apply our demosaicing algorithms to a single image to figure out how effective it is within a single-image framework. Through this experiment, we expected to get some ideas of how to extend this algorithms to multiple-frame demosaicing.

Our algorithm is executed in two steps.

1. Interpolate Green channel by using gradient information of Red and Blue channels.

2. Interpolate Red and Blue by using inter-color correlation parameters, Red-to-Green and Blue-to-Green ratio.

File:Bayer pattern green interpolation.png

Bayer Pattern

Since red and blue channels are down-sampled two times more than the green channel, it's reasonable to interpolate the green channel first then red and blue channel with the interpolated green channel. The simplest approach would be to estimate the unknown pixel values by simple linear interpolation. However, this approach will ignore important information about the correlation between color channels and will cause severe color artifacts. Another critical drawback of simple linear interpolation is that uncorrelated pixels across edges will be averaged and will result in blurred edges. Considering this fact, our algorithm uses second derivatives of the red and blue channel to give different weights for the horizontal and vertical interpolation of the green channel.

,

(The formula above is described for the green value estimation at blue points. The unknown values for the green channel at red points can be estimated by the same formular with replacing the blue channel gradients with red channel gradients.)

The intuition behind this algorithm is that second gradients in the blue or red channels imply possible existence of an edge so that we could use different weight for the horizontal and vertical direction interpolation in the green channel. The large value of second gradients implies existence of an edge while the small value implies either uniform color region or uniformly varying color region. Considering this fact, it's reasonable to giver larger weight to the direction with smaller gradient and smaller weight to the direction with larger gradient when interpolating unknown green pixels. This method will help prevent blurring across edges. If both gradients in the horizontal and vertical directions are zero, equal weights are used (0.5) to equally interpolate in both directions.

Now that unknown pixels in the green channel have been all interpolated, unknown pixels in the red and blue channels could be interpolated with adjacent same colors and the green value weighted by inter-color correlation. coefficient.

Conclusions

Here is where you say what your results mean.

References - Resources and related work

References

[1] S. Farsiu, M. Elad, and P. Milanfar, Multi-Frame Demosaicing and Super-Resolution of Color Images, IEEE Trans. on Image Processing, vol. 15, no. 1, pp. 141-159, January 2006.

[2] E. P. Bennett, M. Uyttendaele, C. L. ZIitnick, R. Szeliski, and S.B. Kang, Video and image Bayesian demosaicing with a two color image prior. In Proceedings of the European Conference on Computer Vision (ECCV’06). Lecture Notes in Computer Science, vol. 3951. 508–521, 2006

Software

Appendix I - Code and Data

Code

All of our code is available on GitHub: [1]

Data

TODO

Appendix II - Work partition

TODO