David Knight - Revision history

imported>Knightd: /* Outdoor Images */

2011-03-19T02:37:41Z

Outdoor Images

← Older revision		Revision as of 02:37, 19 March 2011
Line 102:		Line 102:
	=== Outdoor Images ===		=== Outdoor Images ===
	[[File:Outdoor_results_grid.png \|thumb\|128px\|center\| '''Figure 10:''' Outdoor image Comparison matrix]]		[[File:Outdoor_results_grid.png \|thumb\|128px\|center\| '''Figure 10:''' Outdoor image Comparison matrix]]
	''~~Note~~:'' The zig-zag effect along edges (seen previously in Figure 2) is even stronger in the outdoor test images than it is in the indoor test images.		''Update:'' The zig-zag effect along edges (seen previously in Figure 2) is even stronger in the outdoor test images than it is in the indoor test images. Also of note is that the zig-zag is only horizontal, making it more likely that the effect is due to interlacing.

	== Quantitative Results ==		== Quantitative Results ==

imported>Knightd: /* Conclusions */

2011-03-19T02:36:50Z

Conclusions

← Older revision		Revision as of 02:36, 19 March 2011
Line 155:		Line 155:
	Another issue is that the current implementation of the pipeline takes almost one second to execute, which is not fast enough to run in real-time on a video stream. The major obstacle preventing the pipeline from running in real-time is the expense of running the bilateral filter. The current bilateral filter implementation is a non-compiled MATLAB function. Using a compiled MEX implementation might provide a significant speed improvement. Additionally, thresholding with Otsu's method can be performed with fewer computations by working on an equivalent maximization problem and updating parameters incrementally instead of recalculating statistics each iteration. The rest of the pipeline without the bilateral filter still takes roughly 0.08 seconds to complete, which is not fast enough to process incoming video at 30 frames per second. Reimplementing the pipeline in C with OpenCV might decrease the processing time enough to allow for processing to take place on a video stream.		Another issue is that the current implementation of the pipeline takes almost one second to execute, which is not fast enough to run in real-time on a video stream. The major obstacle preventing the pipeline from running in real-time is the expense of running the bilateral filter. The current bilateral filter implementation is a non-compiled MATLAB function. Using a compiled MEX implementation might provide a significant speed improvement. Additionally, thresholding with Otsu's method can be performed with fewer computations by working on an equivalent maximization problem and updating parameters incrementally instead of recalculating statistics each iteration. The rest of the pipeline without the bilateral filter still takes roughly 0.08 seconds to complete, which is not fast enough to process incoming video at 30 frames per second. Reimplementing the pipeline in C with OpenCV might decrease the processing time enough to allow for processing to take place on a video stream.

	~~Finally, broader~~ testing is needed to see if the color space method is robust against additional indoor lighting schemes and against users with different skin tones. Using a probabilistic graphical model to temporally track features such as most skin-like pixel location and threshold value might be beneficial, as well, since it can be assumed that these parameters should not change very quickly from frame to frame. Overall, the color space pipeline produces skin regions with few false positives and false negatives under simplistic conditions, but the method is currently not robust against backgrounds that are more reddish in the red-green continuum.		Other future work encompasses many areas. Broader testing is needed to see if the color space method is robust against additional indoor lighting schemes and against users with different skin tones. Using a probabilistic graphical model to temporally track features such as most skin-like pixel location and threshold value might be beneficial, as well, since it can be assumed that these parameters should not change very quickly from frame to frame. Also, the zig-zag edge effect should be addressed possibly by applying a interlacing algorithm so long as adding this algorithm does not significantly increase the time needed to execute the pipeline.

			Overall, the color space pipeline produces skin regions with few false positives and false negatives under simplistic conditions, but the method is currently not robust against backgrounds that are more reddish in the red-green continuum.

	= References =		= References =

imported>Knightd: /* Outdoor Images */

2011-03-19T02:33:04Z

Outdoor Images

← Older revision		Revision as of 02:33, 19 March 2011
Line 102:		Line 102:
	=== Outdoor Images ===		=== Outdoor Images ===
	[[File:Outdoor_results_grid.png \|thumb\|128px\|center\| '''Figure 10:''' Outdoor image Comparison matrix]]		[[File:Outdoor_results_grid.png \|thumb\|128px\|center\| '''Figure 10:''' Outdoor image Comparison matrix]]
			''Note:'' The zig-zag effect along edges (seen previously in Figure 2) is even stronger in the outdoor test images than it is in the indoor test images.

	== Quantitative Results ==		== Quantitative Results ==

imported>Knightd: /* Appendix I - Files */

2011-03-19T02:21:03Z

Appendix I - Files

← Older revision		Revision as of 02:21, 19 March 2011
Line 172:		Line 172:
	::[[File:Test_data_sets.zip]]		::[[File:Test_data_sets.zip]]
	*Outdoor Test Image Data Set		*Outdoor Test Image Data Set
	::[[File:~~Outdoor_data_sets~~.zip]]		::[[File:Outdoor_test_data_sets.zip]]
	*Aptina MT9M112 Sensor Datasheet		*Aptina MT9M112 Sensor Datasheet
	::[[File:MT9M112_DS_E.pdf]]		::[[File:MT9M112_DS_E.pdf]]

imported>Knightd: /* Appendix I - Files */

2011-03-19T02:20:40Z

Appendix I - Files

← Older revision		Revision as of 02:20, 19 March 2011
Line 169:		Line 169:
	*MATLAB Files		*MATLAB Files
	::[[File:MATLAB_files.zip]]		::[[File:MATLAB_files.zip]]
	*Test Image Data Set		*Indoor Test Image Data Set
	::[[File:~~test_data_sets~~.zip]]		::[[File:Test_data_sets.zip]]
			*Outdoor Test Image Data Set
			::[[File:Outdoor_data_sets.zip]]
	*Aptina MT9M112 Sensor Datasheet		*Aptina MT9M112 Sensor Datasheet
	::[[File:MT9M112_DS_E.pdf]]		::[[File:MT9M112_DS_E.pdf]]

imported>Knightd: /* Test Sets */

2011-03-19T02:18:32Z

Test Sets

← Older revision		Revision as of 02:18, 19 March 2011
Line 94:		Line 94:
	The primary test set of images is 12 hand images taken every 50 frames from the indoor image sequence. A ground truth test set was generated by hand labeling skin regions. A depth-only segmentation results set was generated by thresholding the depth channel to the region [0.2, 0.8]. Finally, a color space segmentation results set was generated using the pipeline described in the Methods section.		The primary test set of images is 12 hand images taken every 50 frames from the indoor image sequence. A ground truth test set was generated by hand labeling skin regions. A depth-only segmentation results set was generated by thresholding the depth channel to the region [0.2, 0.8]. Finally, a color space segmentation results set was generated using the pipeline described in the Methods section.

	'''UPDATE:''' A more challenging test set of 5 hand images taken every 50 frames from an outdoor image sequence has been added. Ground ~~truths were generated by hand labeling skin regions. A~~ depth-only ~~segmentation results set was generated by~~ thresholding ~~the depth channel to the region [0.2~~, ~~0.8]. Finally, a~~ color space segmentation ~~results set was~~ generated ~~using~~ the ~~pipeline described in~~ the ~~Methods section~~.		'''UPDATE:''' A more challenging test set of 5 hand images taken every 50 frames from an outdoor image sequence has been added. In addition to being taken outdoors in overcast weather, the background of these images is more varied than the background of the indoor test set. Ground truth, depth-only thresholding, and color space segmentation result sets were generated with the same methods as the indoor test images.

	== Visual Comparison ==		== Visual Comparison ==

imported>Knightd: /* Test Sets */

2011-03-19T02:16:42Z

Test Sets

← Older revision		Revision as of 02:16, 19 March 2011
Line 92:		Line 92:
	= Results =		= Results =
	== Test Sets ==		== Test Sets ==
	The ~~unprocessed~~ test set of images is 12 hand images taken every 50 frames from the indoor image sequence. A ground truth test set was generated by hand labeling skin regions. A depth-only segmentation results set was generated by thresholding the depth channel to the region [0.2, 0.8]. Finally, a color space segmentation results set was generated using the pipeline described in the Methods section.		The primary test set of images is 12 hand images taken every 50 frames from the indoor image sequence. A ground truth test set was generated by hand labeling skin regions. A depth-only segmentation results set was generated by thresholding the depth channel to the region [0.2, 0.8]. Finally, a color space segmentation results set was generated using the pipeline described in the Methods section.

			'''UPDATE:''' A more challenging test set of 5 hand images taken every 50 frames from an outdoor image sequence has been added. Ground truths were generated by hand labeling skin regions. A depth-only segmentation results set was generated by thresholding the depth channel to the region [0.2, 0.8]. Finally, a color space segmentation results set was generated using the pipeline described in the Methods section.

	== Visual Comparison ==		== Visual Comparison ==

imported>Knightd: /* Conclusions */

2011-03-19T02:12:56Z

Conclusions

← Older revision		Revision as of 02:12, 19 March 2011
Line 152:		Line 152:
	Another issue is that the current implementation of the pipeline takes almost one second to execute, which is not fast enough to run in real-time on a video stream. The major obstacle preventing the pipeline from running in real-time is the expense of running the bilateral filter. The current bilateral filter implementation is a non-compiled MATLAB function. Using a compiled MEX implementation might provide a significant speed improvement. Additionally, thresholding with Otsu's method can be performed with fewer computations by working on an equivalent maximization problem and updating parameters incrementally instead of recalculating statistics each iteration. The rest of the pipeline without the bilateral filter still takes roughly 0.08 seconds to complete, which is not fast enough to process incoming video at 30 frames per second. Reimplementing the pipeline in C with OpenCV might decrease the processing time enough to allow for processing to take place on a video stream.		Another issue is that the current implementation of the pipeline takes almost one second to execute, which is not fast enough to run in real-time on a video stream. The major obstacle preventing the pipeline from running in real-time is the expense of running the bilateral filter. The current bilateral filter implementation is a non-compiled MATLAB function. Using a compiled MEX implementation might provide a significant speed improvement. Additionally, thresholding with Otsu's method can be performed with fewer computations by working on an equivalent maximization problem and updating parameters incrementally instead of recalculating statistics each iteration. The rest of the pipeline without the bilateral filter still takes roughly 0.08 seconds to complete, which is not fast enough to process incoming video at 30 frames per second. Reimplementing the pipeline in C with OpenCV might decrease the processing time enough to allow for processing to take place on a video stream.

	Finally, broader testing is needed to see if the color space method is robust against additional indoor lighting schemes and against users with different skin tones. Using a probabilistic graphical model to temporally track features such as most skin-like pixel location and threshold value might be beneficial, as well, since it can be assumed that these parameters should not change very quickly from frame to frame. Overall, the color space pipeline produces skin regions with few false positives and false negatives under simplistic conditions, but the method is currently not robust against backgrounds that are more reddish in the red-green ~~spectrum~~.		Finally, broader testing is needed to see if the color space method is robust against additional indoor lighting schemes and against users with different skin tones. Using a probabilistic graphical model to temporally track features such as most skin-like pixel location and threshold value might be beneficial, as well, since it can be assumed that these parameters should not change very quickly from frame to frame. Overall, the color space pipeline produces skin regions with few false positives and false negatives under simplistic conditions, but the method is currently not robust against backgrounds that are more reddish in the red-green continuum.

	= References =		= References =

imported>Knightd: /* Conclusions */

2011-03-19T02:12:31Z

Conclusions

← Older revision		Revision as of 02:12, 19 March 2011
Line 148:		Line 148:
	Unfortunately, the color space pipeline produces unrecognizable results on the outdoor test images. Looking at the resulting regions, it is clear that significant portions of reddish background are incorrectly labeled as skin, accounting for the unacceptably high number of false positive skin labels. However, the number of false negative pixels is less than 10% across all the outdoor test images, indicating that highlighting actual skin regions in outdoor lighting is not an issue.		Unfortunately, the color space pipeline produces unrecognizable results on the outdoor test images. Looking at the resulting regions, it is clear that significant portions of reddish background are incorrectly labeled as skin, accounting for the unacceptably high number of false positive skin labels. However, the number of false negative pixels is less than 10% across all the outdoor test images, indicating that highlighting actual skin regions in outdoor lighting is not an issue.

	Currently, highlighting is performed primarily with information from the Cr channel. Multiplying the distance transform image of the depth-thresholded region with the Cr difference layers is not selective enough to fully rejecting reddish objects in scene backgrounds. Additionally, the <math>\Delta_{xy}</math> images are only used to find pixels that closely match the reference skin colors. A separate rejection stage might be needed in order to reject highlighted regions that appear skin-like in color but have a .		Currently, highlighting is performed primarily with information from the Cr channel. Multiplying the distance transform image of the depth-thresholded region with the Cr difference layers is not selective enough to fully rejecting reddish objects in scene backgrounds. Additionally, the <math>\Delta_{xy}</math> images are only used to find pixels that closely match the reference skin colors. A separate rejection stage might be needed in order to reject highlighted regions that appear skin-like in color but have a depth value that is too far or too close to the Kinect.

	Another issue is that the current implementation of the pipeline takes almost one second to execute, which is not fast enough to run in real-time on a video stream. The major obstacle preventing the pipeline from running in real-time is the expense of running the bilateral filter. The current bilateral filter implementation is a non-compiled MATLAB function. Using a compiled MEX implementation might provide a significant speed improvement. Additionally, thresholding with Otsu's method can be performed with fewer computations by working on an equivalent maximization problem and updating parameters incrementally instead of recalculating statistics each iteration. The rest of the pipeline without the bilateral filter still takes roughly 0.08 seconds to complete, which is not fast enough to process incoming video at 30 frames per second. Reimplementing the pipeline in C with OpenCV might decrease the processing time enough to allow for processing to take place on a video stream.		Another issue is that the current implementation of the pipeline takes almost one second to execute, which is not fast enough to run in real-time on a video stream. The major obstacle preventing the pipeline from running in real-time is the expense of running the bilateral filter. The current bilateral filter implementation is a non-compiled MATLAB function. Using a compiled MEX implementation might provide a significant speed improvement. Additionally, thresholding with Otsu's method can be performed with fewer computations by working on an equivalent maximization problem and updating parameters incrementally instead of recalculating statistics each iteration. The rest of the pipeline without the bilateral filter still takes roughly 0.08 seconds to complete, which is not fast enough to process incoming video at 30 frames per second. Reimplementing the pipeline in C with OpenCV might decrease the processing time enough to allow for processing to take place on a video stream.

	Finally, broader testing is needed to see if the color space method is robust against ~~more~~ indoor lighting schemes and against users with different skin tones. Using a probabilistic graphical model to temporally track features such as most skin-like pixel location and threshold value might be beneficial, as well, since it can be assumed that these parameters should not change very quickly from frame to frame. Overall, ~~this method can~~ produces ~~clean segmentation results when scene backgrounds do not have a strong red color component~~, but ~~requires broader testing and~~ the ~~above issues to be addressed before it can be considered~~ robust.		Finally, broader testing is needed to see if the color space method is robust against additional indoor lighting schemes and against users with different skin tones. Using a probabilistic graphical model to temporally track features such as most skin-like pixel location and threshold value might be beneficial, as well, since it can be assumed that these parameters should not change very quickly from frame to frame. Overall, the color space pipeline produces skin regions with few false positives and false negatives under simplistic conditions, but the method is currently not robust against backgrounds that are more reddish in the red-green spectrum.

	= References =		= References =

imported>Knightd: /* Conclusions */

2011-03-19T01:51:09Z

Conclusions

← Older revision		Revision as of 01:51, 19 March 2011
Line 144:		Line 144:

	= Conclusions =		= Conclusions =
	For most of the test images, the color space pipeline ~~produced~~ tighter skin regions than using a fixed threshold on the depth data. Additionally, by counting pixel region labels, the color space pipeline ~~produced~~ labeled false negative and false positive pixels per image than the fixed depth threshold. However, the color space method in two test images ~~did~~ falsely ~~label~~ red patches in the scene background as skin due to the reliance on the Cr channel during the highlighting stage~~. Another issue is that the current implementation of the pipeline takes almost one second to execute, which is not fast enough to run in real-time on a video stream~~.		For most of the indoor test images, the color space pipeline produces tighter skin regions than using a fixed threshold on the depth data. Additionally, by counting pixel region labels, the color space pipeline produces fewer labeled false negative and false positive pixels per image than the fixed depth threshold. However, the color space method in two test images falsely labels red patches in the scene background as skin due to the reliance on the Cr channel during the highlighting stage.

	~~Currently~~, the ~~<math>\Delta_{xy}</math>~~ images are ~~only used to find pixels that closely match~~ the ~~reference~~ skin ~~colors~~. ~~The highlighting is performed in~~ the ~~Cr channel. It might be beneficial to use a mixture~~ of ~~the Cr channel and <math>\Delta_{xy}</math> images to perform highlighting so the stage~~ is ~~more robust against red surfaces in~~ the ~~background being highlighted. Or the <math>\Delta_{xy}</math>~~ images ~~could be used in a later rejection stage~~ that ~~rejects highlighted~~ regions ~~that do~~ not ~~match a reference skin color~~.		Unfortunately, the color space pipeline produces unrecognizable results on the outdoor test images. Looking at the resulting regions, it is clear that significant portions of reddish background are incorrectly labeled as skin, accounting for the unacceptably high number of false positive skin labels. However, the number of false negative pixels is less than 10% across all the outdoor test images, indicating that highlighting actual skin regions in outdoor lighting is not an issue.

	~~The major obstacle preventing~~ the ~~pipeline from running in real-time is~~ the ~~expense~~ of ~~running~~ the ~~bilateral filter. The current bilateral filter implementation is a non~~-~~compiled MATLAB function. Using a compiled MEX implementation might provide a significant speed improvement. Additionally, thresholding~~ with Otsu's method can be performed with fewer computations by working on an equivalent maximization problem and updating parameters incrementally instead of recalculating statistics each iteration. The rest of the pipeline without the ~~bilateral filter still takes roughly 0.08 seconds to complete, which~~ is not ~~fast~~ enough to ~~process incoming video at 30 frames per second~~. ~~Reimplementing~~ the ~~pipeline~~ in ~~C with OpenCV might decrease the processing time enough~~ to ~~allow for processing to take place on~~ a ~~video stream~~.		Currently, highlighting is performed primarily with information from the Cr channel. Multiplying the distance transform image of the depth-thresholded region with the Cr difference layers is not selective enough to fully rejecting reddish objects in scene backgrounds. Additionally, the <math>\Delta_{xy}</math> images are only used to find pixels that closely match the reference skin colors. A separate rejection stage might be needed in order to reject highlighted regions that appear skin-like in color but have a .

	Finally, broader testing is needed to see if the color space method is ~~usable in different~~ lighting ~~conditions~~ and with ~~users of~~ different skin tones. Using a probabilistic graphical model to temporally track features such as most skin-like pixel location and threshold value might be beneficial, as well, since it can be assumed that these parameters should not change very quickly from frame to frame. Overall, this method ~~shows initial promise~~ but requires broader testing and the above issues to be addressed before it can be considered robust.		Another issue is that the current implementation of the pipeline takes almost one second to execute, which is not fast enough to run in real-time on a video stream. The major obstacle preventing the pipeline from running in real-time is the expense of running the bilateral filter. The current bilateral filter implementation is a non-compiled MATLAB function. Using a compiled MEX implementation might provide a significant speed improvement. Additionally, thresholding with Otsu's method can be performed with fewer computations by working on an equivalent maximization problem and updating parameters incrementally instead of recalculating statistics each iteration. The rest of the pipeline without the bilateral filter still takes roughly 0.08 seconds to complete, which is not fast enough to process incoming video at 30 frames per second. Reimplementing the pipeline in C with OpenCV might decrease the processing time enough to allow for processing to take place on a video stream.

			Finally, broader testing is needed to see if the color space method is robust against more indoor lighting schemes and against users with different skin tones. Using a probabilistic graphical model to temporally track features such as most skin-like pixel location and threshold value might be beneficial, as well, since it can be assumed that these parameters should not change very quickly from frame to frame. Overall, this method can produces clean segmentation results when scene backgrounds do not have a strong red color component, but requires broader testing and the above issues to be addressed before it can be considered robust.

	= References =		= References =