Image classification with a five band camera - Revision history

imported>Slansel at 19:28, 28 March 2014

2014-03-28T19:28:45Z

imported>Projects221: /* Results */

2014-03-20T21:44:13Z

Results

← Older revision		Revision as of 21:44, 20 March 2014
Line 169:		Line 169:
	My overall best image matching results for this project were surprising and not likely "correct", but relatively explainable. The first and only LOOCV run I attempted (because of computational requirements and time) used no SIFT keypoint thresholding (in an attempt to get as many "hits" as possible) and discarded many-to-one matches in RANSAC (to help combat the increase in false positives). Over all eighteen iterations, average performance for 5-band data, 48.8%, was slightly worse than for 3-band data, 51.2%, a difference of about 2.4%. This finding comes with an important caveat: Accuracy was calculated as the number of iterations where each of the four images in the query set were matched to their corresponding image in the test set, with no errors whatsoever. This is not the typical definition of accuracy for image matching, and is harsher than most standards, hence why my findings are so low. While I didn't calculate accuracy numbers based on the typical equation of number of correct matches subtracted by number of incorrect matches divided by total attempts, I can conservatively say that those numbers are around 25% higher than what I chose to report.		My overall best image matching results for this project were surprising and not likely "correct", but relatively explainable. The first and only LOOCV run I attempted (because of computational requirements and time) used no SIFT keypoint thresholding (in an attempt to get as many "hits" as possible) and discarded many-to-one matches in RANSAC (to help combat the increase in false positives). Over all eighteen iterations, average performance for 5-band data, 48.8%, was slightly worse than for 3-band data, 51.2%, a difference of about 2.4%. This finding comes with an important caveat: Accuracy was calculated as the number of iterations where each of the four images in the query set were matched to their corresponding image in the test set, with no errors whatsoever. This is not the typical definition of accuracy for image matching, and is harsher than most standards, hence why my findings are so low. While I didn't calculate accuracy numbers based on the typical equation of number of correct matches subtracted by number of incorrect matches divided by total attempts, I can conservatively say that those numbers are around 25% higher than what I chose to report.

	[[File:~~Perf~~.png \| 300px \| right]]		[[File:Perf2.png \| 300px \| right]]

	My other attempts varied different parameters such as SIFT thresholding, distance ratio values, and RANSAC thresholds, and although overall success rates varied, none were able to match the matching performance reported above, and none were able to achieve a higher matching performance for the 5-band data over the 3-band data. For example, when I increased SIFT thresholds from zero to 1.5% but included many-to-one matches in RANSAC, the average matching performance dropped to ~~___~~% for 3-band data and to ~~___~~% for 5-band data. This experiment was performed using 10-fold cross validation to keep computation times low and be able to try different parameter settings in a shorter amount of time.		My other attempts varied different parameters such as SIFT thresholding, distance ratio values, and RANSAC thresholds, and although overall success rates varied, none were able to match the matching performance reported above, and none were able to achieve a higher matching performance for the 5-band data over the 3-band data. For example, when I increased SIFT thresholds from zero to 1.5% but included many-to-one matches in RANSAC, the average matching performance dropped to 39.7% for 3-band data and to 31.4% for 5-band data. This experiment was performed using 10-fold cross validation to keep computation times low and be able to try different parameter settings in a shorter amount of time.

	My results have indicated a few important findings. First, these results are in fact contradictory to what has been reported in literature, where all the papers I could find reported increased performance for data with more information in it.		My results have indicated a few important findings. First, these results are in fact contradictory to what has been reported in literature, where all the papers I could find reported increased performance for data with more information in it.

imported>Projects221: /* References */

2014-03-20T21:24:44Z

References

← Older revision		Revision as of 21:24, 20 March 2014
Line 210:		Line 210:

	[11] <http://en.wikipedia.org/wiki/Demosaicing>		[11] <http://en.wikipedia.org/wiki/Demosaicing>

			[12] <http://www.vlfeat.org/>

	== Appendix I ==		== Appendix I ==

imported>Projects221: /* Source Code */

2014-03-20T21:24:23Z

Source Code

← Older revision		Revision as of 21:24, 20 March 2014
Line 213:		Line 213:
	== Appendix I ==		== Appendix I ==
	=== Source Code ===		=== Source Code ===
			The code used for this project is included here in an effort of transparency. A quick read me file is included. All other software needed can be obtained directly from VLFeat's website [12].

			[[File:Source Code Loewke.zip]]

	=== Test Images ===		=== Test Images ===

imported>Projects221: /* Conclusions */

2014-03-20T21:16:01Z

Conclusions

← Older revision		Revision as of 21:16, 20 March 2014
Line 180:		Line 180:

	== Conclusions ==		== Conclusions ==
	In conclusion, I demonstrated image matching performance of a five band camera, but achieved rather inconclusive final results.		In conclusion, I demonstrated image matching performance of a five band camera, but achieved rather inconclusive final results. I’ve discussed how ubiquitous computer vision approaches are becoming, even in consumer-grade electronics where real world advantages are gained from clever software implementation. I gave a quick overview of the groups working on this specific problem, and how their findings were more successful and expected than my own. My experiment used a prototype 5-band camera that was calibrated using Macbeth reflectance data and responsivity data supplied by Olympus. I discussed the importance of image demosaicing and the importance of transparency and retaining accuracy in this operation. I discussed and implemented SIFT keypoint extraction, distance ratio tests, and RANSAC model parameter estimation for eliminating outliers, and k-fold cross-validation for performance estimation. My best image matching/classifying performance was 48.8% accuracy and 51.2% accuracy for 3-band and 5-band data, respectively.

	~~- Describe what you~~ learned. ~~What worked? What~~ didn't~~? Why? What should someone next year~~ try?		I think the biggest lesson I learned is that an experiment such as this one is much like a chain, and that any particular aspect can introduce kinks. This kind of experiment incorporates many different considerations, including image calibration, acquisition, accuracy, consistency, software approaches, and validation schemes. I even found that I didn't initially give the amount of time required for calculations as much respect as it deserves, with each run of LOOCV taking about 6 hours in total.

			If I were to take a second attempt at this project, or if perhaps another student were to try, there would be a number of changes I would make. One, I would spend more time on image aqcuisition and implement some sort of feedback loop for determining the best attainable focus. The focus here would be on consistency. The feedback system could be active or (more likely for a class project) passive, but either would be much better than "eyeballing" it, as I did. Second, I think SIFT keypoint detection was a good choice for this project, but I would try a quicker approach for keypoint matching. Instead, I might try image retrieval using vocabulary trees, which is orders of magnitude faster at comparing images, although slightly less accurate. This improvement in computation time could have allowed me to try much more optimization configurations. Finally, I would definitely try to find a more optimal way of implementing SIFT, taking better advantage of the extra information and being careful not to lose or compress details. This likely could have been done by performing multiple passes of SIFT detection for extra bands of information, and either combining keypoint locations or running band-wise comparisons.

			All in all, I view this project as a success despite findings that don't match what's been reported. I would be excited to tackle this project again in the future using the modified approaches listed above.

	== References ==		== References ==

imported>Projects221: /* K-folds cross-validation */

2014-03-20T20:58:29Z

K-folds cross-validation

← Older revision		Revision as of 20:58, 20 March 2014
Line 151:		Line 151:
	\|}		\|}

	=== K-~~folds~~ cross-validation ===		=== K-fold cross-validation ===
	As is commonly seen in image matching studies, some sort of cross-validation is needed for validating our process and assessing how well specific results might translate to more general cases. In other words, we'd like to determine how accurately our predictive model will perform in practice. Assessing how certain matches do undersamples our data and likely skews perceived results. Instead, we aim to thoroughly sample good and bad cases by assessing performance on all permutations of our data set.		As is commonly seen in image matching studies, some sort of cross-validation is needed for validating our process and assessing how well specific results might translate to more general cases. In other words, we'd like to determine how accurately our predictive model will perform in practice. Assessing how certain matches do undersamples our data and likely skews perceived results. Instead, we aim to thoroughly sample good and bad cases by assessing performance on all permutations of our data set.

imported>Projects221: /* Conclusions */

2014-03-20T20:53:25Z

Conclusions

← Older revision		Revision as of 20:53, 20 March 2014
Line 180:		Line 180:

	== Conclusions ==		== Conclusions ==
			In conclusion, I demonstrated image matching performance of a five band camera, but achieved rather inconclusive final results.

	- Describe what you learned. What worked? What didn't? Why? What should someone next year try?		- Describe what you learned. What worked? What didn't? Why? What should someone next year try?

imported>Projects221: /* Results */

2014-03-20T20:52:25Z

Results

← Older revision		Revision as of 20:52, 20 March 2014
Line 177:		Line 177:
	This leads us to our next finding: I need a better, more transparent method for using 5-band data. As stated before, the demosaicing algorithm used for translating 5-band data into RGB images is unknown to me, and thus cannot be trusted as accurate or ideal. In fact, it is quite likely that because we are trying to combine so much information into such a relatively small representation, we're actually losing data that is critical for SIFT to work properly. As I found before, SIFT is highly sensitive to spatial blurring, but it also seems to be sensitive to spectral resolution. By stuffing 5 channels of information into space enough for 3 channel's worth, we may be shooting ourselves in the foot. It may be that other groups discovered or predicted this finding before publishing results, because many groups tried combining the extra information obtained in their studies in different ways. Some groups even modified their SIFT algorithms to be multispectral, whereas the one implemented here works in grayscale. Therefore by the time data was analyzed by SIFT, it had been demosaiced and down converted. This is far from ideal, but I hadn't anticipated it hurting my results as much as I now expect it did.		This leads us to our next finding: I need a better, more transparent method for using 5-band data. As stated before, the demosaicing algorithm used for translating 5-band data into RGB images is unknown to me, and thus cannot be trusted as accurate or ideal. In fact, it is quite likely that because we are trying to combine so much information into such a relatively small representation, we're actually losing data that is critical for SIFT to work properly. As I found before, SIFT is highly sensitive to spatial blurring, but it also seems to be sensitive to spectral resolution. By stuffing 5 channels of information into space enough for 3 channel's worth, we may be shooting ourselves in the foot. It may be that other groups discovered or predicted this finding before publishing results, because many groups tried combining the extra information obtained in their studies in different ways. Some groups even modified their SIFT algorithms to be multispectral, whereas the one implemented here works in grayscale. Therefore by the time data was analyzed by SIFT, it had been demosaiced and down converted. This is far from ideal, but I hadn't anticipated it hurting my results as much as I now expect it did.

	The last finding that may have hurt results was unintentional, but unavoidable, image blurring in my single round of image acquisitions.		The last finding that may have hurt results was unintentional, but unavoidable, image blurring in my single round of image acquisitions. None of my data looked blurred at the time, and on a macro scale don't seem blurry, but remain convinced that with a little help from autofocusing software I could have achieved consistently sharper images. One may to do this would have been to incorporate an active system in which an infrared signal sent from atop the camera and is bounced off the target and back onto a sensor which records actual distance. A motor could then adjust the focus accordingly. Another method could have been to use a passive approach, imaging a test target with sharp black and white lines and running it through a simple algorithm to search for maximum intensity differences. This again could enable a motorized focus control, or simply be a visual feedback for manual focus control, if operated in real-time. I make this point because SIFT is notoriously sensitive to this parameter, as discussed before. During my image acquisition, I was limited to manual focus with feedback from a macro-scale representation of the image, using the entire field of view. Should this experiment be repeated in the future, implementing a fix for more consistency here would be an important step forward.

	== Conclusions ==		== Conclusions ==

imported>Projects221: /* K-folds cross-validation */

2014-03-20T20:42:04Z

K-folds cross-validation

← Older revision		Revision as of 20:42, 20 March 2014
Line 161:		Line 161:

	The k-fold process is fairly flexible, with the value of k being chosen by the experimenter. If the value for ''k'' matches the number of observations in the original sampling, then it is called leave-one-out cross validation (LOOCV). This approach is very thorough, but iterative and slow. For example, my original experiment used a ''k'' value of 18 such that I was using LOOCV, and each data set (3-band or 5-band) took over three hours to test. When I started experimenting with parameters, I chose a lower value of k for slightly less accurate accuracy numbers, but much improved computation time.		The k-fold process is fairly flexible, with the value of k being chosen by the experimenter. If the value for ''k'' matches the number of observations in the original sampling, then it is called leave-one-out cross validation (LOOCV). This approach is very thorough, but iterative and slow. For example, my original experiment used a ''k'' value of 18 such that I was using LOOCV, and each data set (3-band or 5-band) took over three hours to test. When I started experimenting with parameters, I chose a lower value of k for slightly less accurate accuracy numbers, but much improved computation time.

			Although Matlab some some built-in cross-validation support, I opted to implement this myself because of how my data was organized.

	== Results ==		== Results ==

imported>Projects221: /* Results */

2014-03-20T20:37:25Z

Results

← Older revision		Revision as of 20:37, 20 March 2014
Line 171:		Line 171:
	My other attempts varied different parameters such as SIFT thresholding, distance ratio values, and RANSAC thresholds, and although overall success rates varied, none were able to match the matching performance reported above, and none were able to achieve a higher matching performance for the 5-band data over the 3-band data. For example, when I increased SIFT thresholds from zero to 1.5% but included many-to-one matches in RANSAC, the average matching performance dropped to ___% for 3-band data and to ___% for 5-band data. This experiment was performed using 10-fold cross validation to keep computation times low and be able to try different parameter settings in a shorter amount of time.		My other attempts varied different parameters such as SIFT thresholding, distance ratio values, and RANSAC thresholds, and although overall success rates varied, none were able to match the matching performance reported above, and none were able to achieve a higher matching performance for the 5-band data over the 3-band data. For example, when I increased SIFT thresholds from zero to 1.5% but included many-to-one matches in RANSAC, the average matching performance dropped to ___% for 3-band data and to ___% for 5-band data. This experiment was performed using 10-fold cross validation to keep computation times low and be able to try different parameter settings in a shorter amount of time.

	My results have indicated a few important findings. First, these results are in fact contradictory to what has been reported in literature, where all the papers I could find reported increased performance for data with more information in it. This leads us to our next finding: ~~(2)~~ I need a better, more transparent method for using 5-band data. As stated before, the demosaicing algorithm used for translating 5-band data into RGB images is unknown to me, and thus cannot be trusted as accurate or ideal. In fact, it is quite likely that because we are trying to combine so much information into such a relatively small representation, we're actually losing data that is critical for SIFT to work properly. As I found before, SIFT is highly sensitive to spatial blurring, but it is also sensitive to data		My results have indicated a few important findings. First, these results are in fact contradictory to what has been reported in literature, where all the papers I could find reported increased performance for data with more information in it.

			This leads us to our next finding: I need a better, more transparent method for using 5-band data. As stated before, the demosaicing algorithm used for translating 5-band data into RGB images is unknown to me, and thus cannot be trusted as accurate or ideal. In fact, it is quite likely that because we are trying to combine so much information into such a relatively small representation, we're actually losing data that is critical for SIFT to work properly. As I found before, SIFT is highly sensitive to spatial blurring, but it also seems to be sensitive to spectral resolution. By stuffing 5 channels of information into space enough for 3 channel's worth, we may be shooting ourselves in the foot. It may be that other groups discovered or predicted this finding before publishing results, because many groups tried combining the extra information obtained in their studies in different ways. Some groups even modified their SIFT algorithms to be multispectral, whereas the one implemented here works in grayscale. Therefore by the time data was analyzed by SIFT, it had been demosaiced and down converted. This is far from ideal, but I hadn't anticipated it hurting my results as much as I now expect it did.

			The last finding that may have hurt results was unintentional, but unavoidable, image blurring in my single round of image acquisitions.

	== Conclusions ==		== Conclusions ==

← Older revision		Revision as of 19:28, 28 March 2014
Line 19:		Line 19:
	[[File:Camera Responsivities.png \| right \| 300px]]		[[File:Camera Responsivities.png \| right \| 300px]]

	The camera used in this experiment was a prototype five band camera donated to our lab ~~from Olympus~~. Due to it being a prototype, there are many unknowns about the camera, but we do know a few details. The sensor itself has a size of 1920 x 1080 pixels with a depth of 12 bits. When positioning, aiming, and focusing the camera, we were able to use all of its pixels in "1080P mode" using feedback from the preview software on the connected laptop. However, during data acquisition, the saved images were only 1280 x 720 per channel.		The camera used in this experiment was a prototype five band camera donated to our lab. Due to it being a prototype, there are many unknowns about the camera, but we do know a few details. The sensor itself has a size of 1920 x 1080 pixels with a depth of 12 bits. When positioning, aiming, and focusing the camera, we were able to use all of its pixels in "1080P mode" using feedback from the preview software on the connected laptop. However, during data acquisition, the saved images were only 1280 x 720 per channel.

	Raw files generated from the Matlab acquisition software are linear (meaning they have no gamma transform), 12-bit arrays with five channels per pixel. Each pixel measures one color (red, green, blue, orange, or cyan) based on its position in the color filter array. The demosaicing operation is unknown, but it is known that it can take in 12-bit data and returns 16-bit upscaled data. This data is then scaled down to 8-bits for RGB image formation.		Raw files generated from the Matlab acquisition software are linear (meaning they have no gamma transform), 12-bit arrays with five channels per pixel. Each pixel measures one color (red, green, blue, orange, or cyan) based on its position in the color filter array. The demosaicing operation is unknown, but it is known that it can take in 12-bit data and returns 16-bit upscaled data. This data is then scaled down to 8-bits for RGB image formation.
Line 64:		Line 64:
	The most common commercially used CFA configuration is the Bayer filter shown here [11]. In this case, each 2x2 cell contains one blue short-pass filter, one red long-pass filter, and two green band-pass filters. Other commonly used CFAs are RGBE, CYYM, CYGM, RGBW Bayer, and RGBWs #1-3. The Bayer filter's widespread use is attributed to it being designed to mimic the spectral sensitivity of the human eye, which has higher sensitivity to green wavelengths.		The most common commercially used CFA configuration is the Bayer filter shown here [11]. In this case, each 2x2 cell contains one blue short-pass filter, one red long-pass filter, and two green band-pass filters. Other commonly used CFAs are RGBE, CYYM, CYGM, RGBW Bayer, and RGBWs #1-3. The Bayer filter's widespread use is attributed to it being designed to mimic the spectral sensitivity of the human eye, which has higher sensitivity to green wavelengths.

	Although many consumer-grade digital cameras can now save images in raw format and allow the user to demosaic their data using software other than from the company, we opted to use ~~Olympus'~~ demosaicer, which would be supplied in the form of built-in firmware should a product like this one reach the mass market.		Although many consumer-grade digital cameras can now save images in raw format and allow the user to demosaic their data using software other than from the company, we opted to use an existing demosaicer, which would be supplied in the form of built-in firmware should a product like this one reach the mass market.

	=== Data ===		=== Data ===
	The data used in this experiment is a set of 72 image acquisitions for a total of 144 total images of four pieces of fruit: a ripe banana (mostly yellow appearance), an underripe banana (green and yellow appearance), a red delicious apple, and a gala apple. The set was used to determine not only if we could distinguish between major classes (apple vs. banana), but also between subclasses (ripe vs. underripe or type of apple), indicating further sensitivity to color differences. Images were taken from six different spatial locations (with varied XY location and distance from the subject) and under three different lighting condition possibilities: (1) incandescent room lights on, (2) a diffused and reflected lamp from location left of subject, and (3) a lamp from location right of subject. Both sources of light used tungsten filaments.		The data used in this experiment is a set of 72 image acquisitions for a total of 144 total images of four pieces of fruit: a ripe banana (mostly yellow appearance), an underripe banana (green and yellow appearance), a red delicious apple, and a gala apple. The set was used to determine not only if we could distinguish between major classes (apple vs. banana), but also between subclasses (ripe vs. underripe or type of apple), indicating further sensitivity to color differences. Images were taken from six different spatial locations (with varied XY location and distance from the subject) and under three different lighting condition possibilities: (1) incandescent room lights on, (2) a diffused and reflected lamp from location left of subject, and (3) a lamp from location right of subject. Both sources of light used tungsten filaments.

	Each image was acquired using all five available bands of sensitivity from the camera. Data was then either combined using software provided ~~by Olympus~~ (and thus not readily available to scrutinization or optimization) to produce RGB images from combining (in some unknown way) all five bands of data into three channels, or selectively chosen (i.e. the red, green, and blue channels were preserved while the cyan and orange channels were discarded) to simulate an image taken on a standard three channel camera. These data types are termed 5-band and 3-band, respectively.		Each image was acquired using all five available bands of sensitivity from the camera. Data was then either combined using software provided (and thus not readily available to scrutinization or optimization) to produce RGB images from combining (in some unknown way) all five bands of data into three channels, or selectively chosen (i.e. the red, green, and blue channels were preserved while the cyan and orange channels were discarded) to simulate an image taken on a standard three channel camera. These data types are termed 5-band and 3-band, respectively.

	An example set of each class of data (ripe banana, underripe banana, red delicious apple, and gala apple) for each type of data (3-band on top, 5-band on bottom). It's visually apparent that the three band data is darker, and for good reason: we've essentially thrown away pixels by discarding two of our five channels of data.		An example set of each class of data (ripe banana, underripe banana, red delicious apple, and gala apple) for each type of data (3-band on top, 5-band on bottom). It's visually apparent that the three band data is darker, and for good reason: we've essentially thrown away pixels by discarding two of our five channels of data.
Line 180:		Line 180:

	== Conclusions ==		== Conclusions ==
	In conclusion, I demonstrated image matching performance of a five band camera, but achieved rather inconclusive final results. I’ve discussed how ubiquitous computer vision approaches are becoming, even in consumer-grade electronics where real world advantages are gained from clever software implementation. I gave a quick overview of the groups working on this specific problem, and how their findings were more successful and expected than my own. My experiment used a prototype 5-band camera that was calibrated using Macbeth reflectance data and responsivity data supplied ~~by Olympus~~. I discussed the importance of image demosaicing and the importance of transparency and retaining accuracy in this operation. I discussed and implemented SIFT keypoint extraction, distance ratio tests, and RANSAC model parameter estimation for eliminating outliers, and k-fold cross-validation for performance estimation. My best image matching/classifying performance was 48.8% accuracy and 51.2% accuracy for 3-band and 5-band data, respectively.		In conclusion, I demonstrated image matching performance of a five band camera, but achieved rather inconclusive final results. I’ve discussed how ubiquitous computer vision approaches are becoming, even in consumer-grade electronics where real world advantages are gained from clever software implementation. I gave a quick overview of the groups working on this specific problem, and how their findings were more successful and expected than my own. My experiment used a prototype 5-band camera that was calibrated using Macbeth reflectance data and responsivity data supplied. I discussed the importance of image demosaicing and the importance of transparency and retaining accuracy in this operation. I discussed and implemented SIFT keypoint extraction, distance ratio tests, and RANSAC model parameter estimation for eliminating outliers, and k-fold cross-validation for performance estimation. My best image matching/classifying performance was 48.8% accuracy and 51.2% accuracy for 3-band and 5-band data, respectively.

	I think the biggest lesson I learned is that an experiment such as this one is much like a chain, and that any particular aspect can introduce kinks. This kind of experiment incorporates many different considerations, including image calibration, acquisition, accuracy, consistency, software approaches, and validation schemes. I even found that I didn't initially give the amount of time required for calculations as much respect as it deserves, with each run of LOOCV taking about 6 hours in total.		I think the biggest lesson I learned is that an experiment such as this one is much like a chain, and that any particular aspect can introduce kinks. This kind of experiment incorporates many different considerations, including image calibration, acquisition, accuracy, consistency, software approaches, and validation schemes. I even found that I didn't initially give the amount of time required for calculations as much respect as it deserves, with each run of LOOCV taking about 6 hours in total.