A Review of Frequency-selection Demosaicing Algorithm by Shuang Liu
1.Introduction
The modern digital cameras are far more than simply using lens to focus photons onto the onto an image pickup device. In fact, there are various processing performed by the cameras, including auto-focus, white point adjustment, color interpolation, color balancing, image compressing[1]. This project concentrates on one component of the processing pipeline in camera, the demosaicing algorithm, which is needed in every modern consumer-level camera today to render the output image as close to the actual scene as possible from the image captured by the sensor(CFA image).
To capture incident photons of different wavelengths on the optical sensor, i.e., to discriminate and measure the energy of lights of different colors, professional cameras may employ multiple (Usually, three or four) sensors covered by different selective filters to enable each sensor to detect a particular color channel. But in order to reduce cost and size, most consumer-level digital cameras use a single sensor whose surface is overlaid with color filter arrays(CFA) ,which make pixels at the sensor have different types of spectral sensitivities(generally, three or four different types). Since the reconstruction of the full color image requires each pixel must have R, G and B values, however, typically only one color is sampled at a particular spatial location, a subsequent interpolation step, which is commonly called “demosaicing” because of the mosaic pattern of the CFA, is needed for estimating the missing color values at each pixel location.
Professor Farrell et al. classify the demosaicing algorithms into three categories [2]:
(a).Single-channel interpolation:
These algorithms, such as bilinear/bicubic interpolation and spline interpolation, estimate missing color values for each of the three color planes separately. They can be easily implemented, but they totally ignore the inter-channel correlation that can be used to improve the demosaicing performance, for instance, reducing color fringes at sharp edges. As a result, these costeffective methods usually introduce demosaicing artifacts such as blurring and false colors.
(b).Inter-channel interpolation:
These methods exploit the inter-channel dependency and interpolate all color planes to get the missing values .Generally, these algorithms, such as template matching, detect edges in the CFA image first, then average the pixels along the edges rather than across them. They tend to have better performance than those in (a), but their performance varies with the content of image and can have large computational complexity[3].
(c).Frequency selection interpolation:
These methods exploit the characteristics of CFA image, and isolate the luminance and chrominance in spatial frequency domain by means of filtering. These methods seem simple in theory, but the performance highly depends on the design of filters used to select spectral components and the content of input image. As a result, incorporating adaptive filtering[4] in the original method proposed by Alleysson and his colleagues[3], can significantly improve the performance of the algorithm[4][1]. Besides, the trade-off between color rendering and spatial resolution is obvious in these method, as shown in [7], preserving high resolution detains indicating poor color accuracy and accurate color rendering means false luminance patterns.
In general speaking, the demosaicing algorithms require a luminance and chrominance decomposition. Usually, the luminance should contain most of the spatial information in CPA image, so it looks just like a gray-scale rendering of the scene, and should not be applied to the following convolution steps. On other hand, since there’s little spatial information in chrominance, we can average the values of the neighborhoods to get a good color rendering[5].In this project, I first explore the characteristics of CFA image proposed by Alleysson et al. in [3][6], then simulate the general procedure of frequency-selection demosaicing algorithm, in the end briefly discuss several issues encountered in the interpolating step: the choice of CFA pattern, the effect of the filter and the measures used in the comparison of different demosaicing algorithm.
2.Methods
2.1 Luminance and chrominance model
As in [3][4][6], we define lum as the luminance component of the CFA image and chr1+chr2 as the chrominance component of the CFA image. Clearly by this definition, the luminance doesn’t depend on the sampling function, which means it preserves a part of information of the color image I exactly, so we can also call lum the luminance of the color image. But the corresponding chrominance part of the CFA image, chr, is different from that of the original image. Similarly, we can define the chrominance of the original color image as:
There are three points regarding equation(2) that are worth noting:
1.It makes sense that in the frequency-selection demosaicing algorithm we are about to study, we don’t make changes to this part of CFA image, since it preserves the exact luminance. In short, we should not let lum go through the following convolution step.
2.Luminance is just a weighted sum of Ir,Ig and Ib, and coefficients are the possibilities of the R,G,and B samples in CFA pattern we are using, which are also corresponding to the relative sensitivities of human visual system. So a good selection of luminance should preserve as most of the spatial information as possible and look just like a gray scale of the original color image.
3.By further observation, chrominance part of CFA image can be expressed as[3]:
To better understand what the chrominance components chr of the CFA image is, we can do the following transformation to get chrd, which can be referred as ‘demultiplexed’ chrominance:
which indicates demultiplexed chrominance chrd of the CFA image is in fact the sampling of the original chrominance chri of the color image. So in order to get an estimated chrominance chre, we can simply demosaic chrd by some simple interpolation methods, such as bilinear demosaicing algorithm.
2.2 Characteristics of CFA image in spatial frequency domain
Taking the Fourier transform of equation (2b), we get:
So that Fourier transform of the CFA image consists of the baseband component of luminance, and high frequency components which are corresponding to the modulated chr1 and chr2 as shown by Fig.1.
Since the luminance and chrominance are well localized in the frequency domain, in theory, we can use a low pass filter to extract the luminance and a high pass filter to get the chrominance. But in practical, as is also shown in Fig.1. , the luminance and chrominance is highly likely overlapped, which make it impossible to use a filter to extract the luminance without introducing chrominance and vice versa.
2.3 General procedure of the demosaicing algorithm based on frequency selection
(1).Use low pass filter to get estimated luminance lum from CFA image
(2).Use high pass filter or orthogonal filter of low pass filter in (1) to get the chrominance chr.
(3).Use equation (5) to get demultiplexed chr, i.e., chrd.
(4).Use simple interpolation methods, such as bilinear demosaicing algorithm, to get estimated chrominance chre.
(5).Use lum and chre to get reconstructed full-color image: Id= lum + chre
3.Results
3.1 Simulation of the frequency-selection demosaicing algorithm
Fig.2. Shows the general procedure of frequency-selection demosaicing algorithm as described in section 2.3.
In order to show the estimated luminance above is a decent choice, we compare the estimated luminance and gray scale version of the input in Fig.3.. By which we can see, the estimated luminance preserve most of the spatial information.
There are several drawbacks in the simulation model that I must point out here.
1. The original input image is not a good representation of the true world since it is an output of a consumer-level camera which has already been processed by a pipeline inside the camera including optical bluring, demosaicing, color correction and more. So it seems using images of higher quality, such as multispectral or high-spectral images, will be better. But since the project only intends to show the procedures and characteristics of frequency selection algorithm itself (no comparison between different algorithms), and the following steps also make simplification, the input doesn’t affect the simulation too much.
2. In the simulation, I use the R,G or B value of input image as the CFA image values for each pixel. Therefore, the sensitivity function of the filter in the simulation is quite different from that of a true camera.
3. I employed an ideal low pass filter to extract the luminance, which is impossible in real life, since filters can never be ideal.
3.2 Simple demonstration of influence of the low pass filter on the performance of demosaicing
Fig.4. shows a simple demonstration of effects of bandwidth of the low pass filter(for luminance extraction) on performance of the demosaicing method. To one extreme, assume the bandwidth is zero, then the algorithm becomes bilinear demosaicing(since we use blinear method to demultiplex the chrominance),and as mentioned before, it tends to suffer color fringes at sharp edges. To the other extreme, the filter is all-pass filter, the demosaiced image becomes achromatic mosaic image. In short, the design of filters for selecting the luminance/chrominance is of vital importance in determining the performance of the frequency-selection demosaicing algorithm.
3.3 Other points that are worth noting
(1)CFA pattern
Based on the characteristics we discussed in section 2.1, especially equation (6), we can see at least one of the reasons that the Bayer CFA pattern is so popular today. The chr1 and chr2 of Bayer pattern have been modulated to the corner, furthest from the luminance in baseband, which means reducing the region overlapped with luminance.
As shown in Fig.5. (from [8],Figure 5), the manner in which luminance and chrominance overlaps is closely related to the CFA pattern. Bayer pattern is optimal in the terms of using frequency-selection to recover mosaic image. The worst arrangement is non-periodic patterns, such as Fig.5.(e), in which case the chrominance is spread across the whole frequency domain, so the cross talk between luminance and chrominance is severe, in other words, it’s impossible to use frequency-selection demosaicing algorithm to reconstruct the CFA image.
(2)Measures used for estimate the demosaicing algorithm
There is no quantitative measurement of demosacing performance in my simulation, since I made too much simplification and the practical case is much different from here. But I still want to point out the possible measures that we can use to evaluate the performance of a demosaicing algorithm.
The [1][6][8] use mean square error(MSE) or peak signal to noise ratio(PSNR).( They are equivalent , since PSNR=10log(255^2/MSE) (dB) .) And increasing number of papers[1][7][10] are use distances in the perceptually uniform space, such CIELAB or CIELUV, since these metrics, such as S-CLELAB, are better considering that they are consistent with human visual perception .
(3)Improved frequency-selection demosaicing methods
Xin Li et al. compare a variety of demosaicing method in[1], and it seems demosaicing algorithm based on frequency-selection doesn’t perform very well especially in terms of PSNR. But the authors also point out by incorporating adaptive filters(AF) into the method can largely improve the performance. And I believe lots of works[8][9] have been done to study the improvement that AF can contribute to the frequency-selection demosaicing algorithm. In short, the performance of the frequency-selection methods closely depends on the design of filters
4.Conclusions
The frequency-selection demosaicing method is relatively straightforward in theory, just separating the luminance and chrominance in spatial frequency domain by utilizing low pass filter and high pass filter. Then preserve the luminance component, since it represents exactly the spatial information of the original image. Then we can demultiplex the chrominance component and estimated the chromatic information of the input image by applying simple interpolation to the demultiplexed chrominance .The final reconstructed image is the sum of luminance and the estimated chrominance.
There three points about frequency-selection demosaicing algorithm that are worth noting:
(1)The performance depends largely on the content of the image. In the case that the chrominance and luminance are severely overlapped in the frequency domain, which is highly possible for many natural images, it’s simply impossible to obtain a good demosaiced image by using this algorithm.
(2)The performance also depends greatly on the design of filters. Sometimes, getting a better output image may require a very high order filter, which will increase the cost in hardware and computation complexity.
(3)The algorithm require other interpolation methods to estimate chrominance, thus its computation complexity is higher than other simple methods, such as bilinear demosaicing algorithm.
Besides,Bayer CFA pattern is optimal in this context and adaptive filtering can greatly improve the performance.
5.References
[1]Li,X., Gunturk,B., and Zhang, L., “Image demosaicing: A systematic survey,” in [Visual Communications and Image Processing 2008], Pearlman, W. A., Woods, J. W., and Lu, L., eds., Proc. SPIE 6822 (Jan.2008).
[2] Joyce E. Farrell, Peter B. Catrysse, and Brian A. Wandell, "Digital camera simulation," Appl. Opt. 51, A80-A90 (2012)
[3] D.Alleysson, S. Süsstrunk and J. Hérault, “Color Domosaicing by estimating luminance and opponent chromatic signals in the Fourier domain,” Proc. CIC10, 2002.
[4] E. Dubois.” Frequency-domain methods for demosaicking of bayer-sampled color images”, IEEE Signal Proc.Letters, 12(12):847–850, December 2005
[5] Jim Adams , Ken Parulski , Kevin Spaulding, “Color Processing in Digital Cameras”, IEEE Micro, v.18 n.6, p.20-30, November 1998
[6] D.Alleysson, S. Süsstrunk and J. Hérault, ”Influence of Spectral Sensitivity Functions on Color Demosaicing” Proceedings of the eleventh Color Imaging Conference: Color Science and Engineering Systems, Technologies, Applications, pp. 351-357, November 2003.
[7] Manu Parmar, Stanley J. Reeves,” Selection of Optimal Spectral Sensitivity Functions for Color Filter Arrays.” ICIP 2006: 1005-1008
[8] D. Alleysson and B. Chaix de Lavarène, “Frequency selection demosaicking: a review and a look ahead,” Proc. SPIE 6822, 68221M (2008).
[9] Brice Chaix et al. de Lavarene, “Efficient demosaicing through recursive filtering.” In Proc. IEEE Int. Conf.Image Processing, volume 2, pages 189–192, 2007.
[10] Chaix de Lavar`ene, B., Alleysson, D., and H´eraut, J., “Practical implementation of LMMSE demosaicing using luminance and chrominance spaces,” Computer Vision and Image Understanding 107(1-2), 3–13(2007).
6.Appendix I
1.The codes used for simulation: File:1.pdf
2.PPT: File:2.pdf