Psf analysis and image deblurring using a simulated camera lens

Ilina Mitra, Sam Seyfollahi, Adrian Spanu

Introduction

Point Spread Function

The purpose of this project was to estimate the point spread function (PSF) and geometric distortion of a camera lens. In an ideal world, a camera would not have any non idealities in capturing an image. However, in reality, the image of a point source is blurred due to noise introduced by the camera imaging system. The PSF is a description of this noise, and gives the response of the camera to a point source. The PSF can be thought of as similar to the impulse response of the camera. Thus, knowing the PSF allows one to perform image reconstruction to deblur an image via deconvolution.

Blurring due to an imaging system.

Geometric Distortion

Another known degradation of images comes from geometric distortion. Geometric distortion is a type of optical distortion inherent to lens design that causes straight lines to appear bent or curved in the captured image. The two most common types of distortion are barrel distortion and pincushion distortion. In the case of barrel distortion, the perceived location of a point is farther away from the center than where it actually is located. Pincushion distortion, on the other hand, is where the point is located closer to the center relative to where it actually lies. Both these distortions are radial in nature due to the fact that optical systems have radial symmetry.

Left diagram illustrates the original grid, middle diagram illustrates Barrel Distortion, and right diagram illustrates Pincushion Distortion

This project also investigated geometric distortion by finding the center of the lens using two different algorithms, and determining if the resulting center indicated a high level of geometric distortion.

Background

Background Literature for PSF

Estimates of PSF have been done by several researchers in the past. One methodology is outlined in "Image Enhancement using Calibrated Lens Simulations" by Shih et. al. In this paper, researchers used an iterative method to calculate the PSF of a camera lens. Given a lens prescription, they employed ray tracing to determine the PSF. They tested their estimated PSF by comparing it to a measured PSF using a pinhole array setup, only to find the initial results were unsatisfactory. Thus they used a feedback method by which they updated the lens prescription until it produced a PSF that more closely matched the measured PSF [1].

Iterative method employed by Shih et. al. to estimate PSF

The pinhole array setup used to measure the PSF was the inspiration for this project. These researchers constructed a pin hole array to create a grid of light sources. As is outlined in the method section below, a slightly different approach was taken in this project to create a grid of light points. Nevertheless, the decision to pursue this idea came from the research outlined in this paper.

Pin hole array created to measure PSF by Shih et. al.

Heide et. al. suggested a set of computational photography methods to remove the geometric distortion and chromatic aberration. These techniques would allow one to use simple optics to capture images which can be corrected after the being taken. Another algorithm for local sub-pixel estimation of PSF is suggested in Delbracio et. al., and has a high accuracy and relative errors around 2 to 5%.

Background Literature for Geometric Distortion

Characterization of geometric distortion has also been performed extensively by a number of researchers. In general, because of the radial symmetry exhibited by the most common types of distortion, research efforts are spent estimating the coefficients of the following radial distortion model [4], [5]:

$r_{measured}=r_{u}(1+k_{1}*r^{2}+k_{2}*r^{4}+k_{3}*r^{6}+...)$ where $r_{measured}^{2}=x_{measured}^{2}+y_{measured}^{2}$

In order to remove the radial distortion a parametric model is applied to the sensor image to estimate the distortion coefficients; correction is possible by applying appropriate factors.

The distortion can also be written as $x_{measured}$ = $x.f(r,k)$ and $y_{measured}=y.f(r,k)$

There are some commonly used functions for correcting the radial distortion which are listed in table below:

The exact locations of the grid lines are available from the display image. $(x_{i},y_{i})$ for each grid points on the sensor image could be calculated using MATLAB. Selecting the appropriate polynomial or radial distortion function and replacing the location values the coefficient values (k's) could be calculated.

Methods

Monochrome Camera Lens

Since camera lenses have different channel sensitivities to red, green and blue colors, creation of a monochrome camera lens was required. A white field was placed on the display and a sensor image was captured. The ratio of red vs. green and blue vs. green channel sensitivities was calculated to acquire the scaling factors of red/green and blue/green. Using these two scaling factors, all red and blue pixels in the image were scaled to generate a monochrome camera sampling at the sub pixel resolution.

Next, a sensor image of a grid of white display pixels was captured. The camera was set up far enough so the white grid pixels on the image could all be treated as a single monochromatic point of light (although they were created by displaying red, blue and green sub pixels).

Point Spread Function Estimation

The setup shown above by Shih et al. has the capability of estimating a PSF for all areas of an image very directly. By assuming that each dot in the image represents a point source of light, we can treat the images that they create as localized point spread functions. That is, by assuming that the camera is taking a picture of point light sources, we can see how the camera displays each point source across multiple pixels, with a ‘perfect’ PSF only a single pixel wide.

The underlying assumption behind this PSF estimator, however, is that the camera’s response is shift invariant. That is, if we move our setup in a different direction, the PSF responses should be very similar. If they are not, then we cannot use one picture as an estimator for the PSF as a whole; instead, we will need to find a way to average out the PSF estimators of each region. This process will not be as precise and will introduce errors.

If the PSFs do turn out to be the same in each region when the underlying image is moved, that means that the camera and lens have PSFs that are indeed shift invariant, and we can use any of the estimator images that we have taken to estimate the PSF for the camera itself.

Once the PSF has been estimated and is known, we can use it to reverse its impact on an image and thus sharpen the image. That is, by taking an image that has been taken with this camera and lens, and thus has the same PSF as what has been estimated, we can ‘back out’ of the PSF through de-convolution and thus sharpen the image to simulate what it would have looked like if the camera and lens had no PSF at all.

What should also be noted is that since the camera and lens system have other noise sources outside of the PSF, even two images taken of the exact same setup immediately after each other will also not be exactly the same. Thus, in this project, when comparing two images that have been captured, we will be comparing the entirety of the image and look for ‘close enough’, rather than exactly the same.

Point spread function were estimated using these images and images were tested for shift invariance.

Image of white background for scaling and subtraction purposes

Image of grid of white dots for PSF calculation

Testing for Shift Invariance

Based on the method presented above, our first step to the calculation of the PSF was the checking to ensure that the camera was indeed shift-invariant. To do this, the setup above was created to measure several images that had been shifted by a known amount. Several pictures were captured and comparisons between images of varying shifts could be created to see if this camera did maintain the shift invariance property.

A picture was taken of the original state of the setup - the screen showing the array of point sources of light. This image was then moved by 1 pixel towards the right and down and another photo was captured. This process was repeated once move, to create an image that was shifted by 2 pixels towards the right and down as compared to the original setup. A final picture was taken of purely the background, to serve as background removal.

Characterization of Geometric Distortion of Lens

Geometric distortion is radially symmetric around the center of image. Therefore, a method to estimate the center of the image was developed. The sensor image of the grid of white pixels was loaded into MATLAB and scaled using the red/green and blue/green scaling factors. The exact location of each white pixels was calculated.

Grid of white pixels after loading into MATLAB

Two methods were used to estimate the center of the image: For the first method, the distance from each point to its 4 nearest grid points was calculated ($d_{1}$ , $d_{2}$, $d_{3}$, and $d_{4}$ as shown in the figure below) and $Distance_{total1}=d_{1}+d_{2}+d_{3}+d_{4}$ was calculated. This process was repeated for all the points in the grid. The point with smallest $Distance_{total1}$ was selected as the center of the image.

Distances to adjucent grid points

The second method also involved calculating the distances to the 4 nearest adjacent points for each grid point on the image; however this method placed more emphasis on center of image being equidistant form its adjacent point.

The $Distance_{total2}$was calculated as following:

$Distance_{total2}=|d_{1}-d_{2}|+|d_{1}-d_{3}|+|d_{1}-d_{4}|$

This process were repeated for all the points. The point with the lowest $Distance_{total2}$ is the center of image.

Results

PSF Calculations

The process for shift-invariance undoes the process of shifting the point sources of light. That is, the entire image is shifted by 1 pixel at a time, first towards the left and then up, to a maximum of 5 pixels in each direction, and each resulting image is then compared to the image of the original state. The script that does this is found in Appendix II. As we cannot know the conversion factor - the number of pixels the image moves in the camera when the real thing is moved on screen, we could not simply move that number of pixels. Thus, the range was used and the image with the highest correlation was taken.

There are two different methods used to compare the resulting images. The first is the cross correlation coefficient, used in 2 dimensions, and the second is the structural similarity index. The cross correlation coefficient, r, is given by the following, where A is the mean of the image A. The large the coefficient of r, the more closely related the two images, with 1 being exactly the same image.

The structural similarity, or SSIM, is an index of values between 0 and 1, giving how closely related two images are. Two images that are exactly the same have a value of 1. The SSIM is meant to take into account the perceptual differences of humans in comparing images. More information about it can be found here: http://www.cns.nyu.edu/pub/eero/wang03-reprint.pdf

Results

The PSF response has indeed turned out to be shift invariant. When we performed the analysis above and compared multiple images together, we have been able to achieve correlation values of over 90% between distinct images, as can be seen in the table below. The values described here are the cross correlations. A few notable results need be pointed out about the conclusions at which we have arrived.

Image 1: Original image
Image 2: 1 pixel shift to the right and 1 down
Image 3: 2 pixels shift to the right and 2 down
Image 4: Repeat of 2 pixels shift to the right and 2 down


The final output of the PSF can be seen here, both zoomed in on one specific PSF, and the way that the PSF's are arranged.

PSF Comparison Across the Image

As we can see from this comparison, the PSFs vary drastically across the entirety of the captured image.

First PSF:


Second PSF:

SSIM Index

For our analysis, the SSIM index has proved to be much less reliable than the cross correlation. SSIM is meant to be more useful in calculating visual perception differences between images. That is, the traditional use of the SSIM index is to compare the way in which two images would be viewed by humans, rather than how statistically similar they are. In our case, this method has given very inconsistent results. The SSIM index was never more than 0.69 for any image, with the usual values being lower than 0.2. As well, the most visually similar images reported by the SSIM were the original images as they were taken, before any shifts in pixels had occurred. As a result, when comparing very sparse images accentuated every 90 pixels by small point-spread functions, the statistical similarity has turned out to be more useful.

Background noise

To perform this analysis, we have used a picture of a blank background (that is, with a lack of point sources of light), to remove the effects of the background on the image itself. This has served to both reduce the noise in the image, as well as to add other, localized noise as a result. Taking the PSF in the resulting image and changing its dynamic range to increase the saturation, we can see that the corners of the image have large flare-ups. These sources of noise are most likely caused by inconsistencies in the backlighting of the screen that produced the original image.

Translation Ratio

In our analysis, we were able to determine the translation between screen space and camera space. That is, when creating our methodology, we realized that we had to sweep over multiple pixels, compare the cross correlation of each one before we could make a judgment as to what happened to each of the PSFs when the image space was moved by a single pixel. It appears that the translation ratio in terms of number of pixels moved in camera space for each pixel moved in image space is 1. That is, when we moved the image by 1 pixel in both directions, the resulting photo was also shifted by 1 pixel in both directions. This has been true for 2 pixels as well. When comparing the shifted by 1 pixel, and shifted by 2 pixel images, that is, image 2 and image 3 or image 4, to achieve the best cross correlation, we only need to move by one pixel in each direction – what we would expect with this translation ratio of 1.

Center of Image Calculations

Using the two aforementioned methods, the center of the image was calculated and found to be very close to the actual center of the image for both methods. Additionally, the top five centers for both methods coincided for 4 out of 5 points. Thus the two separate methods confirmed the top results in the estimation of the center of the image.

In calculating the center using the first method, it was found that the percent difference between the center and the point with the worst distortion was 4%. This indicates that overall, the image had very little distortion. The second method confirmed this, with the percent difference between the center and the point with the worst distortion also being less than 4%. Thus it was determined that the camera lens used for these experiments had almost negligible geometric distortion. As a result, it was not necessary to estimate parameters for the radial distortion model, since there was little to no geometric distortion to correct.

Conclusions & Next Steps

Performing an analysis by using the setup described by Shih et al. can be very useful in defining many of the underlying properties of a camera and lens combination. From colour channel scaling to image center and even a PSF, we can calculate many useful properties of this camera system that we can use to improve images after they have been captured.

The next steps would be to take the work described above and actually apply it to images to increase their quality. For example, by knowing the PSF fully, as we have been able to do, we can remove its effects and sharpen the image. We have written a preliminary script that can be used to do just that. However, we have not been able to find test images from this specific camera with which to test out the script and fully understand the impact of the PSF removal.

The next steps that we would like to pursue would be to take a series of test images with the same camera and lens combination and try to understand the level of image quality enhancements we would be able to achieve through the removal of the effects of the PSF.

References

[1] YiChang Shih, Brian Guenter, Neel Joshi, Image Enhancement using Calibrated Lens Simulations. Proc. 12th European Conference on Computer Vision, ECCV 2012.
[2] Heide , Felix, Mushfiqur Rouf, et al. " High-quality computational imaging through simple lenses." ACM Transactions on Graphics (TOG) . 32.5 (2013).
[3] Mauricio Delbracio, Pablo Musé, and Andrés Almansa, Non-parametric Sub-pixel Local Point Spread Function Estimation, Image Processing On Line, vol. 2012, pp. 8–21.
[4] J. Heikkil and O. Silvn, "A four-step camera calibration procedure with implicit image correction," in IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Juan, Puerto Rico, 1997, pp. 1106-1112.
[5] Janne Heikkila and Olli Silven, "Calibration procedure for short focal length off-the-shelf CCD cameras," in Proceedings of 13th International Conference on Pattern Recognition, Vienna, Austria, 1996, pp. 166-170.
[6] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: From error measurement to structural similarity” IEEE Transactions on Image Processing, vol. 13, no. 1, Jan. 2004.
[7] Shalin Mehta. ShiftMatrix.m. http://www.mathworks.com/matlabcentral/fileexchange/31840-shift-matrix/content/shiftmatrix.m

Appendix I

README File:

File:README File.pdf

Source code for this project can be downloaded here:

File:Source Code.zip

Images used for PSF analysis and Geometric Distortion analysis can be downloaded here:

File:Images.zip

The powerpoint presentation can be downloaded here:

File:Powerpoint Presentation.pdf

Appendix II

Adrian:

• Estimation of PSF
• Test for shift invariance of PSF

Sam & Ilina:

• Experimental measurements with Joyce and Haomiao
• Background literature research and review
• Creation of a monochrome camera lens
• Estimation of the geometric center of the lens

Shared Responsibilities:

• Wiki writeup
• Powerpoint slides