LiYuXie

Introduction

As the digital camera technology rapidly developed, the camera quality was boosted significantly. Thus the International Standards Organization(ISO) has built up a set of camera image quality metrics to quantify the image quality in terms of resolution, noise level, color accuracy of cameras. However, based on the different content in images, and subjective preference of different people in observing the images, the image quality estimated by even the most commonly used metrics might not be effective in reflecting the observed image quality. As a result, We need to evaluate the metrics by comparing the metrics with the appearance of these images as they are rendered on a display.

In this project, we focus on the effectiveness of metrics for evaluating the noise level of different kinds of images by comparing results generated from metrics, say mainly visual SNR, and observations of human. By taking advantage of Image Systems Engineering Toolbox(ISET), we calculate the metric for simulated camera to evaluate visible noise of a camera – visual SNR. VSNR is the ratio of signal power to noise power, calculated from XYZ value of the processed image. By varying luminance, pixel size, F number, and optical/sensor parameters, we see how the vSNR and images would change so that we evaluate whether the image quality can correctly reflect human observation or not for different scenes, e.g. human faces, texts, landscapes.

Background

Nowadays, a lot of metrics are used to evaluate image quality, such as ISO 15739 visual noise, S-CIELAB and vSNR, which can be found in the paper Development of the I3A CPIQ spatial metrics[1]. Their goals are to predict the visible differences between a pair of images. These metrics are commonly used to measure the important aspects of image quality such as sharpness, noise, contrast and so on. However, the optical non-uniformities prevalent in cell phone cameras and the potentially higher noise level lead to challenges to the ISO 15739 protocal. Thus we would like to introduce vSNR as our main metrics here. Signal-to-noise ratio(SNR) evaluates the level of a signal to the level of background noise. It is defined as the ratio of signal power to the noise power. The SNR used in imaging is a physical measure of the sensitivity of a imaging system. For visual noise evaluation, vSNR is the inverse of the standard deviation of the SCIELAB representation of a uniform field; its units are 1/ΔE[2].The vSNR metric can be used in simulations to predict how imaging system components affect noise visibility[2]. From technology side, PRNU and DSNU can be minimized, but photon noise cannot. VSNR probably is a good metric to measure noise, specifically photon noise here, but the its effectiveness in reflecting human subjective observations is yet to be checked. It is quite important to identify whether it could correlate the human observation and physical measurement, or we are supposed to find better metrics.

Methods

Software

Matlab 7.14

Algorithms

To evaluate the effectiveness of vSNR for measuring image quality, we investigate vSNR effectiveness with different parameters for processing different type of scenes. We use ISET to calculate metrics for different parameters (e.g. luminance, pixel size, exposure time, F-number). We also use ISET to simulate how camera captures and processes different type of scenes (e.g. faces, text and landscapes). For each condition, we compare the metrics with the appearance of these images as they are rendered on a display. We use control variate method to measure and compare the influence of each variable.

Experimental Procedure

Using the script s_Psych221_winter2014_vSNR.m, change the parameters and see what happens to the image displayed and vSNR. Summarize a trend of vSNR effectiveness with different parameters. The focus is to evaluate the effectiveness of vSNR measuring the noise level and overall image quality.

(1) Luminance
(i) For human face,
Control Exposure Time=0.03, f-number=5, viewD=0.2, pixel=[0.5,1,2,4]. Change light level, lightLevels = [5,10,100,200]

(ii) For text,
Control Exposure Time=0.03, f-number=5, viewD=0.2, pixel=[0.5,1,2,4]. Change light level, lightLevels = [5,10,100,200]

(iii) For landscape,
Control Exposure Time=0.03, f-number=5, viewD=0.2, pixel=[0.5,1,2,4]. Change light level, lightLevels = [5,10,100,200]

(2) Pixel Size
(i) For human face,
Control light level = [5,10,100,200], Exposure Time=0.03, f-number=5, viewD=0.2. Change pixel = [0.5, 1, 2, 4].

(ii) For text,
Control light level = [5,10,100,200], Exposure Time=0.03, f-number=5, viewD=0.2. Change pixel = [0.5, 1, 2, 4].

(iii) For landscape,
Control light level = [5,10,100,200], Exposure Time=0.03, f-number=5, viewD=0.2. Change pixel = [0.5, 1, 2, 4].

(3) Exposure Time
(i) For human face,
Control f-number=5, viewD=0.2, light level = [50, 100, 200]. Change Exposure Time=0.01.
Control f-number=5, viewD=0.2, light level = [50, 100, 200]. Change Exposure Time=0.03.
Control f-number=5, viewD=0.2, light level = [50, 100, 200]. Change Exposure Time=0.3.

(ii) For text,
Control Exposure Time=0.03, viewD=0.2, light level = [50, 100, 200]. Change Exposure Time=0.01.
Control Exposure Time=0.03, viewD=0.2, light level = [50, 100, 200]. Change Exposure Time=0.03.
Control Exposure Time=0.03, viewD=0.2, light level = [50, 100, 200]. Change Exposure Time=0.3.

(iii) For landscape,
Control Exposure Time=0.03, viewD=0.2, light level = [50, 100, 200]. Change Exposure Time=0.01.
Control Exposure Time=0.03, viewD=0.2, light level = [50, 100, 200]. Change Exposure Time=0.03.
Control Exposure Time=0.03, viewD=0.2, light level = [50, 100, 200]. Change Exposure Time=0.3.

(4) F-number
(i) For human face,
Control Exposure Time=0.03, viewD=0.2, light level = [50, 100, 200]. Change f-number = 3.
Control Exposure Time=0.03, viewD=0.2, light level = [50, 100, 200]. Change f-number = 5.
Control Exposure Time=0.03, viewD=0.2, light level = [50, 100, 200]. Change f-number = 8.

(ii) For text,
Control Exposure Time=0.03, viewD=0.2, light level = [50, 100, 200]. Change f-number = 3.
Control Exposure Time=0.03, viewD=0.2, light level = [50, 100, 200]. Change f-number = 5.
Control Exposure Time=0.03, viewD=0.2, light level = [50, 100, 200]. Change f-number = 8.

(iii) For landscape,
Control Exposure Time=0.03, viewD=0.2, light level = [50, 100, 200]. Change f-number = 3.
Control Exposure Time=0.03, viewD=0.2, light level = [50, 100, 200]. Change f-number = 5.
Control Exposure Time=0.03, viewD=0.2, light level = [50, 100, 200]. Change f-number = 8.

Results

Luminance

Human Face

exposureTime=0.03, f#=5, viewD=0.2
Horizontally (from left to right): lightLevels = [5,10,100,200]
Vertically (from top to bottom): pixel = [0.5,1,2,4]

From the image displayed, we could see that image quality in terms of noise is getting better with increasing light level in each pixel size condition. We could also see that overall image quality is getting better with increasing light level in each pixel size condition.

vSNR plot:

From the above plot we could see that vSNR increases with increasing light level in each pixel size condition. Thus, we could conclude that vSNR can effectively reflect the noise level and overall image quality of human face, in terms of light level.

Text

exposureTime=0.03, f#=5, viewD=0.2
Horizontally (from left to right): lightLevels = [5,10,100,200]
Vertically (from top to bottom): pixel = [0.5,1,2,4]

From the image displayed, we could see that image quality is getting better with increasing light level in each pixel size condition. We could also see that overall image quality is getting better with increasing light level in each pixel size condition.

vSNR plot:

From the above plot we could see that vSNR increases with increasing light level in each pixel size condition. Thus, we could conclude that vSNR can effectively reflect the noise level and overall image quality of text image, in terms of light level.

Landscape

exposureTime=0.03, f#=5, viewD=0.2
Horizontally (from left to right): lightLevels = [5,10,100,200]
Vertically (from top to bottom): pixel = [0.5,1,2,4]

From the image displayed, we could see that image quality is getting better with increasing light level in each pixel size condition. We could also see that overall image quality is getting better with increasing light level in each pixel size condition.

vSNR plot:

From the above plot we could see that vSNR increases with increasing light level in each pixel size condition. Thus, we could conclude that vSNR can effectively reflect the noise level and overall image quality of landscape image, in terms of light level.

Pixel Size

Human Face

exposureTime=0.03, f#=5, viewD=0.2
Horizontally (from left to right): lightLevels = [5,10,100,200]
Vertically (from top to bottom): pixel = [0.5,1,2,4]

From the image displayed, we could see that image quality in terms of noise is getting better with increasing pixel size in each light level condition. However, the overall image quality from the feeling of human eyes might not be the best at the largest pixel size. Human eyes may sense the best overall image quality at a proper pixel size (might be 2 in this image condition).

vSNR plot:

From the above plot we could see that vSNR increases with increasing pixel size in each light level condition. Thus, we could conclude that vSNR can effectively reflect the noise level of human face, in terms of pixel size. Considering the overall image quality, more resolution factors need to be considered.

Text

exposureTime=0.03, f#=5, viewD=0.2
Horizontally (from left to right): lightLevels = [5,10,100,200]
Vertically (from top to bottom): pixel = [0.5,1,2,4]

From the image displayed, we could see that image quality in terms of noise is getting better with increasing pixel size in each light level condition. However, the overall image quality from the feeling of human eyes might not be the best at the largest pixel size. Human eyes may sense the best overall image quality at a proper pixel size (might be 1 in this image condition).

vSNR plot:

From the above plot we could see that vSNR increases with increasing pixel size in each light level condition. Thus, we could conclude that vSNR can effectively reflect the noise level of text image, in terms of pixel size. Considering the overall image quality, more resolution factors need to be considered.

Landscape

exposureTime=0.03, f#=5, viewD=0.2
Horizontally (from left to right): lightLevels = [5,10,100,200]
Vertically (from top to bottom): pixel = [0.5,1,2,4]

From the image displayed, we could see that image quality in terms of noise is getting better with increasing pixel size in each light level condition. However, the overall image quality from the feeling of human eyes might not be the best at the largest pixel size. Human eyes may sense the best overall image quality at a proper pixel size (might be 1 in this image condition).

vSNR plot:

From the above plot we could see that vSNR increases with increasing pixel size in each light level condition. Thus, we could conclude that vSNR can effectively reflect the noise level of landscape image, in terms of pixel size. Considering the overall image quality, more resolution factors need to be considered.

Exposure Time

Human Face

f#=5, viewD=0.2, pixel = 1
Horizontally (from left to right): lightLevels = [50,100,200]
Vertically (from top to bottom): exposure time = [0.01, 0.03, 0.3]

From the image displayed, we could see that image quality in terms of noise is better when exposure time increasing. We could also see that overall image quality is better exposure time increasing.

vSNR Plot: (from top to bottom, exposure time = 0.01, 0.03, 0.3)

From the above plots we could see that vSNR is higher when exposure time getting longer. Thus, we could conclude that vSNR can effectively reflect the noise level and overall image quality of human face image, in terms of exposure time.

Text

f#=5, viewD=0.2, pixel = 1
Horizontally (from left to right): lightLevels = [50,100,200]
Vertically (from top to bottom): exposure time = [0.01, 0.03, 0.3]

From the image displayed, we could see that image quality in terms of noise is better when exposure time increasing. We could also see that overall image quality is better exposure time increasing.

vSNR Plot: (from top to bottom, exposure time = 0.01, 0.03, 0.3)

From the above plots we could see that vSNR is higher when exposure time getting longer. Thus, we could conclude that vSNR can effectively reflect the noise level and overall image quality of human face image, in terms of exposure time.

Landscape

f#=5, viewD=0.2, pixel = 1
Horizontally (from left to right): lightLevels = [50,100,200]
Vertically (from top to bottom): exposure time = [0.01, 0.03, 0.3]

From the image displayed, we could see that image quality in terms of noise is better when exposure time increasing. We could also see that overall image quality is better exposure time increasing.

vSNR Plot: (from top to bottom, exposure time = 0.01, 0.03, 0.3)

From the above plots we could see that vSNR is higher when exposure time getting longer. Thus, we could conclude that vSNR can effectively reflect the noise level and overall image quality of human face image, in terms of exposure time.

F-number

Human Face

exposureTime=0.03, viewD=0.2, pixel = 1
Horizontally (from left to right): lightLevels = [50,100,200]
Vertically (from top to bottom): fnumber = [3, 5, 8]

From the image displayed, we could see that image quality in terms of noise is better when F-number is smaller. We could also see that overall image quality is better when F-number is smaller.

vSNR Plot: (from top to bottom, fnumber = 3, 5, 8)

From the above plots we could see that vSNR is higher when F-number is smaller. Thus, we could conclude that vSNR can effectively reflect the noise level and overall image quality of human face image, in terms of F-number.

Text

exposureTime=0.03, viewD=0.2, pixel = 1
Horizontally (from left to right): lightLevels = [50,100,200]
Vertically (from top to bottom): fnumber = [3, 5, 8]

From the image displayed, we could see that image quality in terms of noise is better when F-number is smaller. We could also see that overall image quality is better when F-number is smaller.

vSNR Plot: (from top to bottom, fnumber = 3, 5, 8)

From the above plots we could see that vSNR is higher when F-number is smaller. Thus, we could conclude that vSNR can effectively reflect the noise level and overall image quality of text image, in terms of F-number.

Landscape

exposureTime=0.03, viewD=0.2, pixel = 1
Horizontally (from left to right): lightLevels = [50,100,200]
Vertically (from top to bottom): fnumber = [3, 5, 8]

From the image displayed, we could see that image quality in terms of noise is better when F-number is smaller. We could also see that overall image quality is better when F-number is smaller.

vSNR Plot: (from top to bottom, fnumber = 3, 5, 8)

From the above plots we could see that vSNR is higher when F-number is smaller. Thus, we could conclude that vSNR can effectively reflect the noise level and overall image quality of landscape image, in terms of F-number.

Conclusions

Luminance--For all these three kinds of images, we can see from the displayed image that with the increase of light level, image quality in terms of noise is getting better, which matches with what is shown in the vSNR plot. With the increase of light level, the overall image quality is getting better. Thus, we could conclude that this camera image quality metric can effectively reflect the noise level and overall image quality in terms of light level.

Pixel Size--For all these three kinds of images, we can see from the displayed image that with the increase of pixel size, image quality in terms of noise is getting better, which is also indicated in the vSNR plots. However, the overall image quality from human observation might not be the best at the largest pixel size. Due to the influence of resolution factor, human eyes may sense the best overall image quality at a specific pixel size. Thus, we could conclude that this camera image quality metric can effectively reflect the noise level of the images in terms of pixel size, but it does not coincide with human feelings, because of the influence of resolution.

Exposure Time--From the image displayed, we could see that image quality in terms of noise is better when exposure time is increasing, meanwhile the image quality if getting better. This can be related to luminance, since longer exposure time gives more photon cast on the scene, thus reduces the photon noise. VSNR can reflect the noise level of image as well as the observed image quality.

F-number--We can reach a conclusion from the displayed images that the smaller F-number is, the better image quality in terms of noise would be. This is true for all these three kinds of images. When F-number decreases, we can also have better overall image quality. This conclusion could be confirmed by the vSNR plots. In a word, this camera image quality metric could effectively reflect the noise level and overall image quality in terms of F-number, despite of the differences of image kinds.

Tradeoffs--In order to get better image quality, we need to consider the resolution tradeoffs with sensitivity. When pixel size is larger, it will increase the amount of photons a pixel can capture, so the vSNR will be higher. That is, the image quality in terms of noise level will be better. However, larger pixels will decrease the number of sensor pixels sampling the optical irradiance image. This might lead to a low resolution image.

Which to choose, a high resolution noisy image or a low resolution clear image? This depends on many factors such as image type, the display, view distance, and so on. Some image like human faces, we might choose a relatively larger pixel size (higher vSNR) since the noise will affect the image quality remarkably. For example, in our test (4.2 Results/Pixel Size), choose pixel = 2 might lead to better overall image quality for human face. Some image like text, we might choose a relatively smaller pixel size because we need to identify the words and the resolution is more important. For example, in our test (4.2 Results/Pixel Size), choose pixel = 1, a smaller pixel size compared to that of human face, might lead to better overall image quality for text image.

References

[1] Baxter, Donald, et al. "Development of the I3A CPIQ spatial metrics." IS&T/SPIE Electronic Imaging. International Society for Optics and Photonics, 2012.

[2] J. Farrell et al., ”Using visible SNR (vSNR) to compare the image quality of pixel binning and digital resizing”, Proc SPIE 7537, 75370C (2010).

We appreciate guidance and help from Professor Farrell.

Appendix I

s_Psych221_winter2014_vSNR.m [1]

To change light level, pixel size, fnumber, edit in:

   cameraVSNR.m -- "Tuning the camera parameters" part. 

To change exposure time, edit in both these two files:

   cameraVSNR.m -- "Tuning the camera parameters" part. 

   /iset-2014Psych221/camera/cameraVSNR.m -- Line 36. 

Appendix II

Yaoxuan Li: paper study, analyze and generate scripts, do evaluation tests, create presentation PPT
Huijie Yu: paper study, analyze and generate scripts, do evaluation tests, create final wiki page
Xinyi Xie: paper study, analyze and generate scripts, do evaluation tests, create final wiki page