NPorfilio

Back to Psych 204 Projects 2009

Background

Image quality is one of the most important considerations when deciding which camera to purchase. Image quality is affected by a camera’s internal mechanics but also by external factors.

With respect to a camera’s internal factors, image quality is affected by:

Lens: The lens transfers the visual information of the object to be depicted onto the plane of the image sensor.

Sensor: The sensor is an array of millions of tiny pixels that produce the final image. When you press your camera's shutter button and the exposure begins, each of these pixels has a photosite which is uncovered to collect and store photons in a cavity. Once the exposure finishes, the camera closes each of these photosites, and then tries to assess how many photons fell into each (Cambridge in Colour). Sharpness, distortion, vignetting, Lateral Chromatic Aberration, noise, and dynamic range are the principal factors that can be measured at this stage (Imatest).

Image processing pipeline: The image processing pipeline is a set of digital adjustments made to the pre-processed image taken from the lens and sensor, and then transferred into the pipeline. It typically includes demosaicing, color correction, white balance, application of gamma and tonal response curves, sharpening, and noise reduction. The output of the pipeline may be compared to the minimally-processed lens and sensor measurements. However, the effect of the pipeline on subjective image quality can be highly scene and application-dependent, making it difficult to assign objective rankings (Imatest).

Oftentimes, image quality is a function of camera price—many consumers expect a more expensive camera to deliver a higher quality image. But is this always the case? Does an expensive DSLR outperform a compact digital camera that is one-third its price? My project investigates two cameras on the market--the Nikon D5100 and the Sony Cyber-shot DSC-T700—using a subset of the factors discussed above and compares the quality of the image outputted from each.

File:Camerasum.png

Summary of Camera Specs

Methods

I decided I needed both objective and subjective data to appropriately compare the two cameras. After all, even if all the quantitative measures say that one camera produces higher quality photos than another, it’s a moot point if the actual pictures do not look better to humans. Consequently, to get the quantitative data, I took a number of test shots with both cameras using the ISO 12233 and MacBeth ColorChecker and analyzed those in Imatest, a software package for testing digital image quality, on several different metrics discussed below. Then to get the subjective data, I took a couple of pictures of Memorial Church on Stanford University’s campus under a number of different lighting conditions. I then used all this data to compare the cameras and came up with my conclusions.

ISO 12233 Chart

Macbeth ColorChecker

Memorial Church, Stanford University

Modulation Transfer Function

The Modulation Transfer Function is the most important measure of device and image sharpness because it determines the amount of detail an image can convey. It is defined as the contrast at a given spatial frequency relative to low frequencies.

I measured this using the photo from each camera of the ISO 12233 chart and uploading the image to Imatest. Then I chose a region of interest to analyze. I choose a slanted edge in the middle of the chart as the region of interest. In order to compare MTF measurements across photos and cameras, Imatest uses the MTF50 metric, which is the spatial frequency where the MTF is 50% of the low frequency MTF measured in Line Widths/Picture Height. According to Imatest documentation, MTF50 is a good parameter for comparing the sharpness of different cameras and lenses for two reasons: (1) Image contrast is half its low frequency or peak values, hence detail is still quite visible. The eye is relatively insensitive to detail at spatial frequencies where MTF is low: 10% or less. (2) The response of most cameras falls off rapidly in the vicinity of MTF50.

To give this measurement context you divide it by some picture height of interest. For example, if you were interested to see if one camera produced a 5” x 7” photo with excellent sharpness, you would divide the calculated MTF50 by 5”. The resulting metric is then MTF50 measured in Line Widths/inch of the print and has the following breakpoints to determine the image sharpness:

150 and above – Excellent – “Extremely sharp at any viewing distance. About as sharp as most inkjet printers can print.”

110-150 – Very Good – “Large prints (A3 or 13×19 inch) look excellent, though they won’t look perfect under a magnifier. Small prints still look very good”

80-110 – Good – “Large prints look OK when viewed from normal distances, but somewhat soft when examined closely. Small prints look soft— adequate, perhaps, for the average consumer, but definitely not crisp."

Chromatic Aberration

Chromatic aberration (CA) is one of several aberrations that degrade lens performance. (Others include coma, astigmatism, and curvature of field.) It occurs because the index of refraction of glass varies with the wavelength of light, i.e., glass bends different colors by different amounts. This phenomenon is called dispersion. Minimizing chromatic aberration is one of the goals of lens design. It is accomplished by combining glass elements with different dispersion properties (Imatest). Lateral chromatic aberration also called “color fringing” is a lens aberration that causes colors to focus at different distances from the image center. It is most visible near corners of images.

The metric here is: CA (area) as a percentage of the distance to image center. This is the main chromatic aberration metric used in Imatest because it is relatively independent of the region of interest. This is important because CA tends to be proportional to the distance from the image center. Similar to the MTF, I used the ISO 12233 photos from each camera but this time chose a slanted edge away from the center as a region of interest.

CA (area) as a percentage of the distance to image center = 100 * (area between the channels with the highest and lowest CA levels) / (distance from center in pixels), corrected for the angle of the ROI.

The breakpoints are:
Under 0.04: Insignificant
0.04-0.08: Minor
0.08-0.15: Moderate
over 0.15: Serious

Delta-E

Delta-E is a represents the 'distance' between the color in a photo and the standardized CIELAB (L*a*b*) colors.

CIELAB standard color space is specified by the International Commission on Illumination. It describes all the colors visible to the human eye and was created to serve as a device independent model to be used as a reference. The three coordinates of CIELAB represent the lightness of the color (L* = 0 yields black and L* = 100 indicates diffuse white), its position between red/magenta and green (a*, negative values indicate green while positive values indicate magenta) and its position between yellow and blue (b*, negative values indicate blue and positive values indicate yellow).

I measured this using the photo from each camera of the Macbeth ColorCheck and uploading the image to Imatest. The greater away the calculated Delta-E is from 0, the more the color is away from the standard color.

Results

Modulation Transfer Function

Imatest MTF Results. Nikon (left), Sony (right)

The calculated MTF50:
Nikon: 1646
Sony: 1222

Imatest MTF Results as a function of picture size

Chromatic Aberration

Imatest CA Results. Nikon (left), Sony (right)

The calculated CA (area) as a percentage of distance to the center:
Nikon: 0.053% - Low
Sony: 0.031% - Insignificant

Delta-E

Imatest DE Results. Nikon (left), Sony (right)

The mean Delta E:
Nikon 11.5
Sony: 12.1

Noise

Imatest Noise Results. Nikon (left), Sony (right)

Nikon has a higher noise level than the Sony at every exposure value.

Memorial Church

Daytime

Memorial Church, Stanford University during the day. Nikon (left), Sony (right)

Nighttime

Memorial Church, Stanford University at night. Nikon (left), Sony (right)

Conclusions

Summary of Results

Using these metrics, the Sony performed much better than expected. The Nikon had a better MTF by a considerable margin, meaning that it produced a sharper image, and a lower mean Delta-E, meaning that it produced more accurate colors. However, the Sony had a lower CA metric, meaning that it produced images with an insignificant amount of color fringing, and also had a lower noise value, meaning it produced less noisy images. That said, as a subjective measure the Nikon produced higher quality images of Memorial Church during the day and at night.

There are several conclusions to be made. First, the most surprising result was the noise measurement. Noise is a function of pixel size and the Nikon has a larger pixel size than the Sony which means that it can capture more photons and thus detail. This result could mean that the noise measurement calculated in Imatest is not reflective of the resulting image. Second, this could also mean that I made an error in my tests either during taking the images and/or processing them in Imatest. However, I did follow Imatest's instructions for capturing and analyzing the images so it's possible that the software only produces accurate results when used in labs under tightly controlled conditions. Third, of course this could also mean that the Sony actually does produce higher quality images with respect to noise and lateral color aberration. Lastly, the results could be a function of my lens choice for the Nikon. Consumers can purchase a variety of lenses of different quality for the Nikon. The lens I used was a lens of modest quality, so perhaps I chose a poor lens to test with.

If I were to continue with this project in the future, I would reshoot all the test images and analyze them again in Imatest to ensure that my measurements were accurate. Then I would analyze those same images with another software, perhaps ISET. Then I would compare all the data and reassess my results.

References

[1] Dr. Karl Lenhardt, Bad Kreuznach. http://www.schneiderkreuznach.com/knowhow/opt_quali_e.htm
[2] Cambridge in Colour. http://www.cambridgeincolour.com/tutorials/camera-sensors.htm
[3] Wikipedia. http://en.wikipedia.org/wiki/Lab_color_space

Software

Imatest. http://www.imatest.com/

Appendix I

Presentation Slides

"Slides: An Analysis of Camera Image Quality: Nikon D5100 vs Sony Cyber-shot DSC-T700"