LuppescuShah - Revision history

imported>Student2016: /* Conclusion & Future Work */

2016-12-16T19:59:29Z

Conclusion & Future Work

← Older revision		Revision as of 19:59, 16 December 2016
Line 123:		Line 123:

	==Conclusion & Future Work==		==Conclusion & Future Work==
	In this project, we explored different hole filling procedures for depth maps, implemented different segmentation algorithms, and simulated depth of field and tilt-shift photography as post processing effects based on depth maps. Hole filling worked very well for the close range images. For future work, an extension of hole filling could be ~~tonimplement~~ more complex hole filling algorithms like colorization and hierarchical hole filling and to compare the results to the simpler methods presented in this project. We are happy we explored both K-means and mean shift segmentation, as both seemed to work well in different scenarios. For the short range camera, we achieved foreground detection by using K-means with number of clusters = 2. For the longer range depth image, we used mean shift with a radius of 0.04. K-means failed to segment out the body of the person standing in the image from the rest of the clutter in the long range image, but mean shift did reasonably well. The downside to mean shift is that it is much more computationally expensive. For future work, students could explore other types of segmentation algorithms for depth maps like the n-cuts segmentation algorithm.		In this project, we explored different hole filling procedures for depth maps, implemented different segmentation algorithms, and simulated depth of field and tilt-shift photography as post processing effects based on depth maps. Hole filling worked very well for the close range images. For future work, an extension of hole filling could be to implement more complex hole filling algorithms like colorization and hierarchical hole filling and to compare the results to the simpler methods presented in this project. We are happy we explored both K-means and mean shift segmentation, as both seemed to work well in different scenarios. For the short range camera, we achieved foreground detection by using K-means with number of clusters = 2. For the longer range depth image, we used mean shift with a radius of 0.04. K-means failed to segment out the body of the person standing in the image from the rest of the clutter in the long range image, but mean shift did reasonably well. The downside to mean shift is that it is much more computationally expensive. For future work, students could explore other types of segmentation algorithms for depth maps like the n-cuts segmentation algorithm.

	Depth maps allowed us to automatically simulate depth of field given a point of interest, which would be quite tedious given only RGB information. We noticed that there were edge effects when computing depth of field, creating a “glowing” effect around an object in focus. For future work, students could try to improve this depth of field algorithm by accounting for those edge artifacts to reduce the “glowing.” Also, we chose an arbitrary amount to blur based on distance, so another extension of this project could be to incorporate camera information like focal length and f-number to simulate how images would look with different types of cameras.		Depth maps allowed us to automatically simulate depth of field given a point of interest, which would be quite tedious given only RGB information. We noticed that there were edge effects when computing depth of field, creating a “glowing” effect around an object in focus. For future work, students could try to improve this depth of field algorithm by accounting for those edge artifacts to reduce the “glowing.” Also, we chose an arbitrary amount to blur based on distance, so another extension of this project could be to incorporate camera information like focal length and f-number to simulate how images would look with different types of cameras.

imported>Student2016: /* Segmentation */

2016-12-16T19:57:02Z

Segmentation

← Older revision		Revision as of 19:57, 16 December 2016
Line 99:		Line 99:
	<br>		<br>

	K-means segmentation did not work well for long range depth maps. This is because, for scenes which cover longer ranges of distances, there are more objects of interest and hence, a larger number of potential clusters. K-means segmentation was not effective when the number of clusters was large. Hence, for images taken from the long range depth camera i.e. Intel RealSense R200, we tried the mean shift segmentation algorithm ~~resulting~~ in better segmentations of the scene. Given below is the resulting image obtained by segmenting the depth map using the mean shift algorithm. It can be seen that the mean shift algorithm discretizes a continuous depth map and isolates regions of interest, like, for example, the body of the person in the image. Again, the next section shows how segmentation can make depth maps more useful and effective for post-processing effects like depth of field (the difference between colors may not be evident because of the range of values).		K-means segmentation did not work well for long range depth maps. This is because, for scenes which cover longer ranges of distances, there are more objects of interest and hence, a larger number of potential clusters. K-means segmentation was not effective when the number of clusters was large. Hence, for images taken from the long range depth camera i.e. Intel RealSense R200, we tried the mean shift segmentation algorithm, which resulted in better segmentations of the scene. Given below is the resulting image obtained by segmenting the depth map using the mean shift algorithm. It can be seen that the mean shift algorithm discretizes a continuous depth map and isolates regions of interest, like, for example, the body of the person in the image. Again, the next section shows how segmentation can make depth maps more useful and effective for post-processing effects like depth of field (the difference between colors may not be evident because of the range of values).
	[[File:GRMeanShift.png\|frame\|center\|Fig 7. Segmenting depth maps using the mean shift algorithm]]		[[File:GRMeanShift.png\|frame\|center\|Fig 7. Segmenting depth maps using the mean shift algorithm]]

imported>Student2016: /* Mean shift Segmentation */

2016-12-16T19:52:14Z

Mean shift Segmentation

← Older revision		Revision as of 19:52, 16 December 2016
Line 55:		Line 55:
	*Until all pixels have been “seen”:		*Until all pixels have been “seen”:
	**For a given search radius:		**For a given search radius:
	***Choose a starting point that hasn’t “seen” (in our case, the grayscale value of one of the pixels in our depth image)		***Choose a starting point that hasn’t been “seen” (in our case, the grayscale value of one of the pixels in our depth image)
	***Compute the centroid of the data in the search window		***Compute the centroid of the data in the search window
	***Mark all pixels in search radius as “seen”		***Mark all pixels in search radius as “seen”

imported>Student2016: /* K-means Segmentation */

2016-12-16T19:50:22Z

K-means Segmentation

← Older revision		Revision as of 19:50, 16 December 2016
Line 40:		Line 40:
	====K-means Segmentation====		====K-means Segmentation====
	K-means is one of the simplest image segmentation algorithms. The main purpose of this algorithm is to cluster data. In our case, we want to cluster pixels by depth such that pixels at similar depths will be in the same cluster. A theoretical treatment of this problem ultimately leads to a two-step algorithm:		K-means is one of the simplest image segmentation algorithms. The main purpose of this algorithm is to cluster data. In our case, we want to cluster pixels by depth such that pixels at similar depths will be in the same cluster. A theoretical treatment of this problem ultimately leads to a two-step algorithm:
	[[File:Picture11.png\|center]		[[File:Picture11.png\|center]]
	In step 1, each data point is assigned to the cluster with the closest centroid. Step 2 then recalculates the cluster centroids given the new assignments from step 1. These two steps are repeated until convergence, i.e. the change in cluster centers is below a certain threshold.		In step 1, each data point is assigned to the cluster with the closest centroid. Step 2 then recalculates the cluster centroids given the new assignments from step 1. These two steps are repeated until convergence, i.e. the change in cluster centers is below a certain threshold.

imported>Student2016: /* Background */

2016-12-16T19:44:40Z

Background

← Older revision		Revision as of 19:44, 16 December 2016
Line 15:		Line 15:
	Image segmentation is still an active research problem that has been studied extensively. Classical methods like K-means, mean shift, and n-cuts are fairly robust for many types of segmentation problems, but also have many weaknesses when segmenting RGB images. For these methods, problems arise when there is not much color variance, there are complicated shapes, or the image contains multiple occlusions. Depth information can be used along side RGB images to increase the robustness of these segmentation algorithms.		Image segmentation is still an active research problem that has been studied extensively. Classical methods like K-means, mean shift, and n-cuts are fairly robust for many types of segmentation problems, but also have many weaknesses when segmenting RGB images. For these methods, problems arise when there is not much color variance, there are complicated shapes, or the image contains multiple occlusions. Depth information can be used along side RGB images to increase the robustness of these segmentation algorithms.

	Simulating depth of field as an automatic post-processing effect is a very difficult problem without depth information. Usually, depth of field has to be manually simulated using professional software like Photoshop. Automatic depth of field simulation without depth information would require elaborate and robust segmentation and object detection algorithms, and accurate estimations of depth solely based on the RGB image. The inclusion of depth maps significantly simplifies this problem, allowing for efficient and simple algorithms to simulate of depth of field.		Simulating depth of field as an automatic post-processing effect is a very difficult problem without depth information. Usually, depth of field has to be manually simulated using professional software like Photoshop. Automatic depth of field simulation without depth information would require elaborate and robust segmentation and object detection algorithms, and accurate estimations of depth solely based on the RGB image. The inclusion of depth maps significantly simplifies this problem, allowing for efficient and simple algorithms to simulate depth of field.

	==Methods==		==Methods==

imported>Student2016: /* Background */

2016-12-16T19:43:09Z

Background

← Older revision		Revision as of 19:43, 16 December 2016
Line 11:		Line 11:

	==Background==		==Background==
	Given a pixel with missing information, hole filling can be viewed as an interpolation problem, where neighboring pixel information can be used to infer the value of the missing pixel. In this project, we implemented two basic interpolation methods -- mean filtering and median filtering -- as it worked well for most of the cases and allowed us to focus on other algorithms as well. There are more complicated hole filling algorithms such as hierarchical hole filling [2] and colorization [3]. Hierarchical hole filling eliminates the need for any smoothing or filtering of the depth map by implementing a pyramid-like approach to estimate the hole pixels. Colorization using optimization is a well known technique that uses a grayscale image and some color hints as guides to color a grayscale image. This can be extended to hole filling for depth maps by using a corresponding RGB image as a guide to fill up the holes in the depth map based on the known depth values.		Given a pixel with missing information, hole filling can be viewed as an interpolation problem, where neighboring pixel information can be used to infer the value of the missing pixel. In this project, we implemented two basic interpolation methods -- mean filtering and median filtering -- as they worked well for most of the cases and allowed us to focus on other algorithms as well. There are more complicated hole filling algorithms such as hierarchical hole filling [2] and colorization [3]. Hierarchical hole filling eliminates the need for any smoothing or filtering of the depth map by implementing a pyramid-like approach to estimate the hole pixels. Colorization using optimization is a well known technique that uses a grayscale image and some color hints as guides to color a grayscale image. This can be extended to hole filling for depth maps by using a corresponding RGB image as a guide to fill up the holes in the depth map based on the known depth values.

	Image segmentation is still an active research problem that has been studied extensively. Classical methods like K-means, mean shift, and n-cuts are fairly robust for many types of segmentation problems, but also have many weaknesses when segmenting RGB images. For these methods, problems arise when there is not much color variance, there are complicated shapes, or the image contains multiple occlusions. Depth information can be used along side RGB images to increase the robustness of these segmentation algorithms.		Image segmentation is still an active research problem that has been studied extensively. Classical methods like K-means, mean shift, and n-cuts are fairly robust for many types of segmentation problems, but also have many weaknesses when segmenting RGB images. For these methods, problems arise when there is not much color variance, there are complicated shapes, or the image contains multiple occlusions. Depth information can be used along side RGB images to increase the robustness of these segmentation algorithms.

imported>Student2016: /* Median filter-based Hole Filling */

2016-12-16T15:28:07Z

Median filter-based Hole Filling

← Older revision		Revision as of 15:28, 16 December 2016
Line 34:		Line 34:

	In summary, the hole filling pipeline is given below -		In summary, the hole filling pipeline is given below -
	[[File:GRHoleFillingPipeline.png\|frame\|center\|Fig 1. Hole-filling pipeline]]		[[File:GRHoleFillingPipeline.png\|400px\|frame\|center\|Fig 1. Hole-filling pipeline]]

	===Segmentation===		===Segmentation===

imported>Student2016: /* Appendix 1 */

2016-12-16T09:11:36Z

Appendix 1

← Older revision		Revision as of 09:11, 16 December 2016
Line 138:		Line 138:
	==Appendix==		==Appendix==
	===Appendix 1===		===Appendix 1===
	Source Code: [[File:~~SourceCodeGregRaj~~.zip\|Code]]		Source Code: [[File:Source_Code_Greg_Raj.zip\|Code]]
	<br>		<br>
	(Note: "main.cpp" and "example.hpp" are the files used for image acquisition and they require librealsense and OpenCV to be installed. All the other files run on MATLAB.)		(Note: "main.cpp" and "example.hpp" are the files used for image acquisition and they require librealsense and OpenCV to be installed. All the other files run on MATLAB.)

imported>Student2016: /* Appendix 1 */

2016-12-16T09:11:18Z

Appendix 1

← Older revision		Revision as of 09:11, 16 December 2016
Line 138:		Line 138:
	==Appendix==		==Appendix==
	===Appendix 1===		===Appendix 1===
	Source Code: [[File:~~SourceCodeGregRag~~.zip\|Code]]		Source Code: [[File:SourceCodeGregRaj.zip\|Code]]
	<br>		<br>
	(Note: "main.cpp" and "example.hpp" are the files used for image acquisition and they require librealsense and OpenCV to be installed. All the other files run on MATLAB.)		(Note: "main.cpp" and "example.hpp" are the files used for image acquisition and they require librealsense and OpenCV to be installed. All the other files run on MATLAB.)

imported>Student2016: /* Appendix 1 */

2016-12-16T09:10:59Z

Appendix 1

← Older revision		Revision as of 09:10, 16 December 2016
Line 138:		Line 138:
	==Appendix==		==Appendix==
	===Appendix 1===		===Appendix 1===
	Source Code:		Source Code: [[File:SourceCodeGregRag.zip\|Code]]
	<br>		<br>
	(Note: "main.cpp" and "example.hpp" are the files used for image acquisition and they require librealsense and OpenCV to be installed. All the other files run on MATLAB.)		(Note: "main.cpp" and "example.hpp" are the files used for image acquisition and they require librealsense and OpenCV to be installed. All the other files run on MATLAB.)