Endoscope
Introduction
Darrell Ford and Michael Mendoza
Our goal for this project was to learn how to appropriately and accurately apply Geometric calibration to an endoscope and also inverse its distortion in real-time so that it resembles the image on the right below.
Initially, we set out to test the various features of the endoscope and become comfortable with the paired user software needed to collect data. Once this was accomplished, our team had to determine the type of distortion found in the system and decide how to go about accounting for its affect on the various collected images. Once this was achieved and our images free of distortion, we would then be able to move on to our next task, algorithm development for two flash depth estimation. Unfortunately, our endoscope seemed to have variable gain on its light source that would change depending on how far it was to its target, so we were unable to fulfill this second part of the project.
Background
Key Terms
Endoscope: An endoscope is a medical device with a camera and light source attached that are used to look inside a body cavity or organ. Typically, an endoscope is inserted through a natural orifice of the body, such as the mouth during a bronchoscopy or the rectum for a sigmoidoscopy. Furthermore, if A medical procedure requires the use of any type of endoscope, it is usually referred to as endoscopy.
Optical Aberration: errors in an image that are caused mostly by imperfect manufacturing and approximations made based on our knowledge and understanding of light. Some of the more common types are Spherical aberration, Coma, Astigmatism, Curvature of Field, Chromatic aberration, and in our case Distortion
Distortion: a commonly used term in modern day photography that refers to any deviation from rectilinear projection. Though distortion comes in a wide range of patterns and irregular forms, the most common types we see are "Barrel Distortion" and "Pincushion distortion" as depicted below.
Barrel: image magnification decreases with distance from the optical axis.
Geometric Calibration: determination of the geometric relation of the imaging process of a camera
Method
Hardware
Procedure
1. Setup Endoscope in stationary position
2. Capture images of test target in varying orientations and distances with endoscope.
3. The image in question should be stationary at each data collection. (30+ images)
Software
Procedure:
1. Download the calibration software at: http://www.vision.caltech.edu/bouguetj/calib_doc/index.html#system
2. Store all of the MATLAB (.m) files in a folder called “toolbox_calib.”
3. Upon opening MATLAB, open the directory where the folder that contains all of the MATLAB files is located. Right click the folder a select “Add to Path.”
4. In the command prompt, type “calib” to start the calibration process.
5. Refer to the following example on the various options to choose and how to go through the calibration process (goes into more details about how to maximize calibration efficiency): http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/example.html
6. Choose the “Standard” option to load the GUI.
7. Store all of the images in a folder called “calib_example.” This can be anywhere, but put it in the “toolbox_calib” folder to avoid complications. Make sure to name all of images with a basename and different number (image01, image02, image03, image04, image05).
8. Click the “Image Name” option to load the images. Enter the basename and image type. The images should now all be loaded.
9. Click the “Extract Grid Corners” option to begin the calibration process. Follow the prompts and specify values when prompted. The checkboard is 7x9 squares. Refer to link above about what each step in this part is doing
10. After extracting the corners of all of the images, click the “Calibration” option. This will generate a .mat file with the calibration results.
11. Click the “Save” option to store the calibration.
12. In order to show the extrinsic properties based off of the calibration, click the “Show Extrinsic” option. This provides two different perspectives of the camera orientation in relation to the grid.
13. At this point there are varying ways to examine the errors and correct the calibration (refer to link above). Click the “Analyse error” option
14. Click the “Recomp. Corners” to automatically extract the corners of images where there may have been errors. Repeat the previous steps to save the calibration.
15. The error should dramatically decrease. Click the “Undistort image” option to create the new images. Compare the before and after to see if the initial calibration worked.
16. If not, refer to the link above for more in depth ways to alter the .m files to optimize the calibration.
Results
Conclusion
As we can see from our before and after pictures, the initial and basic geometric calibration does is not effective. There are some small and distinct differences in each comparison (overall the end result does provide a slightly less distorted image), but there is still much work to be done to fix the calibrations. Ultimately we would like to get the reprojection error less than 1 (we were able to get it from having a maximum error of 6 to a maximum error of around 4). A major error we had was MATLAB crashing when trying to run the software. We could not run various .m files and were not able to calibrate more than a certain amount of times. Of course if these errors were not prevalent, we would have hoped to get better results.
Moving forward (assuming no crashes or bugs), there are several ways to fix the calibrations. Some of those ways include going through each .m files and altering parameters in order to maximize the effectiveness of the calibration. Another way is using our “Analyse Error” option. Using this option will allow us to click on the crosshairs of where the reprojection error occurs for each image. Once we click on a crosshair, information is provided to us in our MATLAB console that we can use to correct the calibration for that specific image. Talking to people who have worked on correcting the images from the endoscope, we understand that the time it takes to fully correct the image is outside of the scope of this project. Regardless it was amazing working with the technology and software. To see how the endoscope technology worked and to see a virtual representation of it was a great experience.
Sources
Background
Aliaga, Daniel . "(Geometric) Camera Calibration." Purdue University, West Lafayette, Indiana. 1 Apr. 2010. Class Lecture.
<https://www.cs.purdue.edu/homes/aliaga/cs635-10/lec-camera-calibration.pdf>
"Endoscope." Medline Plus. A.D.A.M, Inc, 1 Jan. 2015. Web. 14 Mar. 2015.
<http://www.nlm.nih.gov/medlineplus/ency/article/002360.htm>.
Code
http://www.vision.caltech.edu/bouguetj/calib_doc/index.html#system
http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/example.html
Acknowledgements
Andy Lin
Graduate Student
Electrical Engineering
Stanford University
Prof Daniel G. Aliaga
Department of Computer Science
Purdue University
Haomiao Jiang
Psych 221 Mentor
Stanford University
Joyce Farrell
Senior Research Engineer
Electrical Engineering - Information Systems Laboratory
Stanford University
Munenori Fukunishi
Olympus Corporation representative
Project Mentor
Steven Lansel
Olympus Corporation representative
Project Mentor