Camera Image Quality Metrics (Sharpness)
Introduction
Image system quality evaluation has been researched for a long time in industry and academia. The International Standards Organization (ISO) is developing a set of camera image quality metrics to quantify the spatial resolution, noise and color accuracy of digital cameras. Several metrics and methods are implemented to measure and calculate the spatial frequency response, which qualifies the sharpness of the image system. More specifically, the metric of MTF50 and acutance in ISO12233 standard are most appropriate to estimate the blurriness quality of the system. By varying some camera setting such as F/# of the lens and pixel size of the sensor, optimized image system could be achieved by plotting the MTF50 and acutance metrics. In this project, we also try to study the correlation between these objective metrics and human viewing experience estimated as the mean opinion score (MOS) which is computed by averaging all team members’ subjective image quality rating of the given image.
This paper is organized as follows. An overview of past work done in the image quality metrics area is given in the background section. The methods section talks about our methodology in metrics data generation and the design-compare flow. The results section encompasses our project results, including sample figures showing photos simulated under different conditions with varying viewing experiences. We will draw our conclusions in the conclusions section and show our references and source codes in the references and appendix sections, respectively.
Background
In a modern camera system, the only widely accepted and marketed camera quality metric is the megapixel count. This metric, however, does not correlate well with the actual image quality. Many image quality metrics have been proposed both in literature and industry, such as ISO12233, MTF and image acuity. These are all objective metrics that try to measure color rendition, sharpness, signal to noise ratio and so on, hoping to achieve a strong correlation with how well an image actually look like. The well studied quality metrics may correspond very poorly between measured and experienced quality, even when the important aspects of the human visual system has been taken into account. It is due to the fact that image quality in the human eye is an aesthetic standard that varies from person to person. Various projects (such as Camera Phone Image Quality (CPIQ)) have been proposed to find the best image quality metrics that aims to model and correct for the human visual system. Nonetheless, they fail to to address the factor of actual human aesthetics.
On the other hand, the university of Texas has compiled a subjective image quality assessment (SIQA) database that collected image quality data based on human subjective feedback. Some research has been done in order to correlated a specific metric with the Mean Opinion Score (MOS), such as PSNR, SSIM and VDP. Yet they mostly focus on the difference between a distorted image and the original image, not on the camera and environmental settings. We aim to estimate the correlation between the device based metrics, such as MTF50, image acutance, pixel size, etc. and the consolidate image viewing experience by computing the metric data and the MOS.
Methods
The Modulation Transfer Function (MTF) is defined as the normalized magnitude of the Fourier Transform of the imaging system’s point spread function. Practically,it is a metric quantifying the sharpness of the image over the range of special frequencies below Nyquist frequency of the system. We measured the system MTF using the ISO standard ISO 12233. This method uses a B/W slanted bar image as the test scene and then measure the response on a rectangular area at the edge to estimate the system MTF. MTF50 is a specific value which indicate the spatial frequency where the amplitude of MTF curve fall to 50% of the maximum amplitude. We also used Acutance, which is calculated using the lens's frequency response and an ISO12233 specific weighting function. Acutance measures the edge sharpness of a picture.
We uses the Image Systems Evaluation Toolkit (ISET) to simulate the image system response in MATLAB. This software set consists of four modules: Scene, Optics, Sensor, and Processor. The Scene module represents the input scene as a multidimensional spectral radiance array at each pixel. The Optics module converts the scene radiance data into an irradiance image through the optics. The Sensor module then transform the irradiance to sensor signal using a model account for both optical and electrical properties of the sensor and pixel. The Processor module finally transforms the electrons on each pixels into a digital image that is rendered for a color display. This module includes algorithms for demosaicing, color conversion to a calibrated color space, and color balancing.
We first evaluate the sensitivities of each parameter of the image system to the final image quality. As a result, the F# and pixel size of the sensor changes the sharpness of image a lot and the effect of other parameters as photon noise, view distance and luminance are insignificant. By sweeping F# and pixel size, we obtain the metric values of MTF50 and acutance for each settings. Then for each specific setting, we simulate the response image using multispectral data of actual scenes. In this project, we choose the image data of face, fruit and landscape to test.
Experiments and Results
To explore whether MTF50 and Acutance correlates well with human viewing experience, we selected 4 different types of scene: landscape, fruit, color checker and human face. As we vary f/# from 1 to 10 and pixel size from 1 to 10 micrometer, we generate 100 groups of data. For each combination we keep the mean illuminance at 100 Candelas/m2, field of view at 5 degree and viewing distance at 1m. Then we measure MTF50 and Acutance against a B/W slanted bar and generate actual images on the above 4 different scenes.
Below is a plot of MTF50 and acutance against F/# and pixel size. As we can see both MTF50 and acutance benefit from lower F/# and smaller pixel size. Also both metrics vary a lot in our sweep range.
Below is a plot of MTF50 against acutance when pixel size is fixed at 1 micrometer and F/# is varied from 1 to 10. These two metrics are correlated.
Now we start to analyze the effect of degrading metrics on real images. In the following scenes we can see that as MTF50 and acutance becomes smaller, all the images become blurrier. One interesting observation is that MTF50 is not a linear metric. In these images, as F/# increases from 1 to 5, MTF50 has degraded by about 250. As F/# increases from 5 to 10, MTF50 has degraded by only 60. Despite the difference in metrics value, the images demonstrated similar sharpness difference when F/# varies from 1 to 5 and 5 to 10.
The other interesting observation is that a lower MTF50 and acutance can actually make human face images look better. This blurriness can make the image look "softer" and provide a better viewing experience.
Finally we keep F/# fixed at 1 and vary pixel size. As we can see from the following image, the increasing of pixel size significantly lowered image quality. The effect is more severe than increasing F/#
One more observation is that when pixel size is large, there are some "bumps" in acutance values. In the following images we vary the pixel size from 7 micrometer to 9 micrometer. Although the image in the middle corresponds to the largest acutance value, it still looks blurrier than the first image on the left. This suggests that acutance might not be a very accurate metric when pixel size is large.
Conclusions
In summary, we can make the following conclusions:
- Lower F/# and smaller pixel size result in higher number for both metrics.
- MTF50 is a nonlinear metric. At lower values, the same difference in MTF50 can translate into more sharpness difference.
- For human face, a smaller MTF50 value makes the image "softer". It actually looks better than a sharper image.
References
- Baxter, Donald, et al. "Development of the I3A CPIQ spatial metrics." IS&T/SPIE Electronic Imaging. International Society for Optics and Photonics, 2012.
- Chen, Ting, et al. "How small should pixel size be?." Electronic Imaging. International Society for Optics and Photonics, 2000.APA
- Farrell, Joyce, Feng Xiao, and Sam Kavusi. "Resolution and light sensitivity tradeoff with pixel size." Electronic Imaging 2006. International Society for Optics and Photonics, 2006.
Appendix I
- blur.m Given F/#, view distance, mean illuminance value, field of view, pixel size and a scene name, this function constructs an imaging system and calculates MTF50 and acutance value against a B/W slanted bar. It also generates an imaged based on the scene name provided. Finally store these data in filesystem.
- tradeoff_viewer.m This function works on the dataset generated by blur.m and plots MTF50 and acutance against F/# and pixel size.
- scene_viewer.m This function plots the images based on an actual scene, using a specified range of F/# and pixel size.
- mtf50_vs_acutance.m This script fixes the pixel size at 1um and plots MTF50 against acutance when F/# varies from 1 to 10.
Appendix ||
Work breakdown:
- Yunqing Hu: metric measurements code, paper study, metric evaluation
- Chengzhong Wu: metric measurements code, paper study, metric evaluation
- Haoxing Zhang: post-processing and visualization code, paper study, metric evaluation