BellMoonVassiliev

Back to Psych 221 Projects 2013

Background

In many image processing applications, it is desirable to combine a series of images of the same scene in order to acheive a higher signal-to-noise ratio (SNR), higher resolution, or both. This can be done with a set of still frames or from a video sequence.

Work on super-resolution has focused primarily on grayscale data. There are several challenges to working with RGB data. The image channels should remain aligned, so that colors are not distorted. This is typically handled by splitting the image into luminance and chrominance channels, and performing the operations in that space [TODO: cite].

When these operations are performed after demosaicking, errors from the demosaicking process are treated as "signal" in the following steps. By performing demosaicking jointly on a set of images, we are able to reduce the color artifacts introduced by the Bayer pattern.

Methods

Our approach consists of an image alignment step, which finds the relative horizontal and vertical offsets between the LR images, and a demosaicking step which combines the data from LR images into a single HR image.

Image Alignment

• Used phase correlation
• Fit parabola to get subpixel resolution
• Do least-squares fit to find a globally-consistent set of offsets

Demosaicking

Principles from single-image demosaicing

Before directly working with multiple images for demosaicing, we first apply our demosaicing algorithms to a single image to figure out how effective it is within a single-image framework. Through this experiment, we expected to get some ideas of how to extend this algorithms to multiple-frame demosaicing.

Our algorithm is executed in two steps.

1. Interpolate Green channel by using gradient information of Red and Blue channels.

2. Interpolate Red and Blue by using inter-color correlation parameters, Red-to-Green and Blue-to-Green ratio.

File:Bayer pattern.png

Bayer Pattern

Since red and blue channels are down-sampled two times more than the green channel, it's reasonable to interpolate the green channel first then red and blue channel with the interpolated green channel. The simplest approach would be to estimate the unknown pixel values by simple linear interpolation. However, this approach will ignore important information about the correlation between color channels and will cause severe color artifacts. Another critical drawback of simple linear interpolation is that uncorrelated pixels across edges will be averaged and will result in blurred edges. Considering this fact, our algorithm uses second derivatives of the red and blue channel to give different weights for the horizontal and vertical interpolation of the green channel.

G(x,y) = w_horizontal * ( G(x-2,y) + G(x+2,y) )/2 + (1-w_horizontal) * ( G(x, y-2) + G(x,y+2) )/2

w_horizontal = gradient2_v / ( gradient2_h + grandient2_v)

gradient2_h = ( B(x-2,y) + B(x+2,y) - 2*B(x,y) )

gradient2_v = ( B(x,y-2) + B(x,y+2) - 2*B(x,y) )

(The formula above is described for the green value estimation at blue points. The unknown values for the green channel at red points can be estimated by the same formular with replacing the blue channel gradients with red channel gradients.)

The intuition behind this algorithm is that second gradients in the blue or red channels imply possible existence of an edge so that we could use different weight for the horizontal and vertical direction interpolation in the green channel. The large value of second gradients implies existence of an edge while the small value implies either uniform color region or uniformly varying color region. Considering this fact, it's reasonable to giver larger weight to the direction with smaller gradient and smaller weight to the direction with larger gradient when interpolating unknown green pixels. This method will help prevent blurring across edges.

Conclusions

Here is where you say what your results mean.

References - Resources and related work

References

[1] S. Farsiu, M. Elad, and P. Milanfar, Multi-Frame Demosaicing and Super-Resolution of Color Images, IEEE Trans. on Image Processing, vol. 15, no. 1, pp. 141-159, January 2006.

[2] E. P. Bennett, M. Uyttendaele, C. L. ZIitnick, R. Szeliski, and S.B. Kang, Video and image Bayesian demosaicing with a two color image prior. In Proceedings of the European Conference on Computer Vision (ECCV’06). Lecture Notes in Computer Science, vol. 3951. 508–521, 2006

Software

Appendix I - Code and Data

Code

All of our code is available on GitHub: [1]

Data

TODO

Appendix II - Work partition

TODO