Geometric calibration of a camera
Introduction
By calibrating the parameters of the lens and image sensors, the distortion of an image can be calibrated to reconstruct the relative position of a camera and objects in the three-dimensional world. It has been widely used in the areas of remote sensing, measurement, etc. The MATLAB provides a calibration tool based on the calibration method proposed by Zhang et al. In this project, the crema of a Samsung Galaxy A20 was calibrated by using MATLAB toolbox. The results showed high accuracy in both standard mode and fisheye mode.
Background
The camera calibration process is a transformation process from the 3D world coordinate to the image coordinate system. This process can be artificially divided into two steps: 1) from the 3-D world coordinate system to the 3-D camera’s coordinate system (extrinsic); 2) from the 3-D camera’s coordinate system to the image plane coordinate system (intrinsic).
Extrinsic parameter
The extrinsic parameter is a 3D to 3D transformation, it describes the rotation (R) and translation (t) of the relative position of crema and 3D object.
Intrinsic parameter
The camera can be modeled as a pinhole crema as shown in figure 2. Hence, the intrinsic transformation is to transform from a 3D camera coordinate to 2D image plane coordinate, which can be modeled as K = [fx 0 0|s fy 0|cx cy 1], where fx and fy is the normalized focal length, cx and cy is the optical center of the image, and s is Skew coefficient for non-perpendicular axes case.
Therefore, the calibration process can be mathematically described as: P = [R|t]K.
Distortion
The lens distortion contains radical and tangential distortion, which cannot be described by the pinhole camera model. For the radical distortion, the further from the center, the larger the distortion we have, which can be corrected by using Taylor series expansion: xdistorted = x(1 + k1*r2 + k2*r4 + k3*r6); ydistorted= y(1 + k1*r2 + k2*r4 + k3*r6) On the other hand, the tangential distortion origins from the unparallel of the lens and camera sensor plane, which can be corrected by xdistorted = x + [2 * p1 * x * y + p2 * (r2 + 2 * x2)] and ydistorted = y + [p1 * (r2 + 2 *y2) + 2 * p2 * x * y].
Methods and Results
Pattern for calibration
The checkerboard pattern is used for MATLAB Camera Calibrator. The pattern was printed on a letter-size paper and stick on a hard board to prevent curvature on the pattern. The grid size of the checkerboard pattern is 21.2×21.2 mm. The pattern was held ~50 cm from the camera and was focused on the camera. The camera was not in autofocus mode and the zoom kept constant in the process. Totally, 15 images were taken with different orientations relative to the camera. These pictures were shown in figure 2.
Calibration
After the identification of the checkerboard, the standard mode with two coefficients was first tested. The mean reprojection error is shown in figure 3. While the three coefficients give a lower error (mean value 1.39 pixels), nevertheless, the two coefficients is sufficient for the standard pictures. The extrinsic parameters (the relative orientation of the checkerboard with respect to the camera) are then shown in the 3D diagram in figure 4. Figure 4a demonstrates a camera-centric schematic, while figure 4b is a pattern-centric view.
The intrinsic and extrinsic parameters were exported as:
Intrinsics
Focal length (pixels): [ 3310.3793 +/- 10.4656 3309.6850 +/- 10.4052 ] Principal point (pixels):[ 2075.4137 +/- 2.2403 1538.1733 +/- 2.2774 ] Radial distortion: [ 0.3611 +/- 0.0107 -1.7308 +/- 0.1035 2.7642 +/- 0.2904 ]
Extrinsics
Rotation vectors:
[ 0.0494 +/- 0.0019 0.1789 +/- 0.0027 -3.0998 +/- 0.0003 ]
[ 0.1573 +/- 0.0019 0.2086 +/- 0.0024 -2.6450 +/- 0.0003 ]
[ -0.1917 +/- 0.0018 0.5245 +/- 0.0021 -3.0313 +/- 0.0004 ]
[ -0.1036 +/- 0.0019 0.3879 +/- 0.0022 -2.7345 +/- 0.0004 ]
[ -0.3314 +/- 0.0026 -0.1681 +/- 0.0029 2.9346 +/- 0.0004 ]
[ 0.3260 +/- 0.0018 0.0409 +/- 0.0019 -3.0734 +/- 0.0003 ]
[ 0.0438 +/- 0.0019 -0.2061 +/- 0.0015 -0.0696 +/- 0.0003 ]
[ 0.0192 +/- 0.0017 0.1670 +/- 0.0016 0.1131 +/- 0.0003 ]
[ -0.0683 +/- 0.0016 -0.2486 +/- 0.0013 0.0169 +/- 0.0003 ]
[ 0.1402 +/- 0.0022 -0.0764 +/- 0.0026 -2.5605 +/- 0.0003 ]
[ 0.0375 +/- 0.0021 0.2135 +/- 0.0020 -1.2631 +/- 0.0003 ]
[ 0.2354 +/- 0.0019 -0.0809 +/- 0.0020 -2.1068 +/- 0.0003 ]
[ 0.2133 +/- 0.0017 -0.1764 +/- 0.0018 -2.3140 +/- 0.0003 ]
[ 0.4288 +/- 0.0016 0.3009 +/- 0.0018 -2.0037 +/- 0.0004 ]
[ 0.1148 +/- 0.0014 -0.4500 +/- 0.0016 -2.4415 +/- 0.0003 ]
Translation vectors (millimeters):
[ 83.8853 +/- 0.1960 51.8912 +/- 0.2008 300.4381 +/- 0.9506 ]
[ 56.7701 +/- 0.2154 83.4066 +/- 0.2193 333.9287 +/- 1.0382 ]
[ 61.8608 +/- 0.2096 38.1369 +/- 0.2203 324.2352 +/- 0.9936 ]
[ 96.5474 +/- 0.2207 97.1636 +/- 0.2237 340.3009 +/- 1.0644 ]
[ 39.1373 +/- 0.2250 48.6800 +/- 0.2273 340.0936 +/- 1.0491 ]
[ 108.7953 +/- 0.1930 41.8584 +/- 0.2046 306.2588 +/- 0.9483 ]
[ -113.7009 +/- 0.2074 -49.4557 +/- 0.1963 282.3532 +/- 0.9158 ]
[ -124.2760 +/- 0.2455 -75.8827 +/- 0.2545 376.1750 +/- 1.1748 ]
[ -80.5848 +/- 0.2133 -54.0782 +/- 0.2037 298.2958 +/- 0.9469 ]
[ 60.1974 +/- 0.2277 75.5904 +/- 0.2332 342.6287 +/- 1.0913 ]
[ 3.6265 +/- 0.2419 84.3545 +/- 0.2429 369.9493 +/- 1.1358 ]
[ 10.9645 +/- 0.2281 99.2212 +/- 0.2271 338.6736 +/- 1.0737 ]
[ 45.5693 +/- 0.2144 90.2729 +/- 0.2198 322.1885 +/- 1.0307 ]
[ -39.3711 +/- 0.2535 81.1931 +/- 0.2570 380.2468 +/- 1.0889 ]
[ 18.1064 +/- 0.1943 71.1540 +/- 0.2000 284.4594 +/- 0.9456 ]
With the intrinsic parameters, we can calculate the focal length in mm scale, i.e., ~3.45 mm. While according to the information of the Samsung, the focal length is 27mm (35mm equivalent), which is actually 3.63mm, so the estimation error is ~4.9%.
An example of the original image and undistorted image are shown in figure 5.
Measurement of objects
We then took the parameters of the camera to measure the size of a penny. Four coins were put on top of the checkerboard as shown in figure 6a, and the images was taken about 30 cm away from the checkerboard. The undistorted image can be obtained by using the parameters of the calibrated camera as P=[R|t]K, which is demonstrated in figure 6b. Since the penny is shiny and colorful than the background, so it is possible to detect it from the coins in by using the saturation component of the HSV representation of the image. The image was first converted to HSV color space, and then threshold the image to black and white as shown in figure 6c. The coins can be detected as the image in figure 6d.
The image was then transformed into the 3D world coordinate to measure the size of the penny. The diameters of the four pennies were shown in table 1 with a mean value of 19.19 mm and a variance of 0.02. The actual size of a penny should be 19.05 mm, the test results give an error of 0.7%, indicating the high accuracy of the measurement.
[ # Diameter (mm) ]
[ 1 19.30 ]
[ 2 19.40 ]
[ 3 19.12 ]
[ 4 18.95 ]
Fisheye mode calibration
The fisheye lens is an extremely wide-angle lens that the angle of view of is between 100 and 180 degrees. However, to achieve the ultra-wide-angle, the lines of perspective in the images are distorted. The pinhole camera model is no longer suitable for the fisheye lens because the alignment coefficients of the intrinsic parameters are related to the relative position of the sensor plane and the pixel plane. In this project, instead of using a fisheye lens, the fish eye image were taken from a software embedded in the Samsung Galaxy A20 camera as shown in figure 7.
By using the Matlab toolbox in fisheye mode, we can calibrate the fisheye image as shown in figure 8.
Conclusions
In summary, by using the MATLAB camera calibrator, both standard and fisheye mode images were calibrated. The measurements of real planner objects show high accuracy without distortion. The future work will concentrate on the development of calibration software without checkerboard pattern or the calibration of nanoscale imaging cameras like the camera of scan electron microscope (SEM), transmission electron microscope (TEM), etc.
References
1. Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transactions on pattern analysis and machine intelligence, 22.
2. Scaramuzza, D., Martinelli, A., & Siegwart, R. (2006, October). A toolbox for easily calibrating omnidirectional cameras. In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 5695-5701). IEEE.
3. Heikkila, J., & Silven, O. (1997, June). A four-step camera calibration procedure with implicit image correction. In cvpr (Vol. 97, p. 1106).
Appendix
Code for calibration: https://drive.google.com/open?id=1J7sS7lZa3YvK-k24zvQCQDJ07Gi_G3U3
Code for measurement: https://drive.google.com/open?id=1alXlInZ1Tm52Nm6cURok8JRCScKmT1mE