Bergman - Revision history

imported>Student221: /* Appendix */

2019-12-14T06:23:18Z

Appendix

← Older revision		Revision as of 06:23, 14 December 2019
Line 169:		Line 169:
	Gordon Wetzstein (non-PSYCH 221 student) defined the motivation for the project and desired results to pursue, helped with the literature review, provided guidance and suggestion on the methods, and provided the computing resources for developing the project.		Gordon Wetzstein (non-PSYCH 221 student) defined the motivation for the project and desired results to pursue, helped with the literature review, provided guidance and suggestion on the methods, and provided the computing resources for developing the project.

	I would like to thank my collaborators for their suggestions and guidance in my development of this project - without their input this project would not be where it is now.		I would like to thank my collaborators for their suggestions and guidance in my development of this project - without their input this project would not be where it is now. I'd also like to thank Professor Wandell, Zheng Lyu, and Dr. Farrell for their insightful comments during my presentation which helped guide my writing of this report.

	== Source Code ==		== Source Code ==
	The repository containing the source code for the methods and evaluation of this project is available upon request. Contact awb@stanford.edu. (Don't want to make the code publicly available yet).		The repository containing the source code for the methods and evaluation of this project is available upon request. Contact awb@stanford.edu. (Don't want to make the code publicly available yet).

imported>Student221: /* Adaptive Sampling */

2019-12-14T06:19:04Z

Adaptive Sampling

← Older revision		Revision as of 06:19, 14 December 2019
Line 83:		Line 83:
	=== Adaptive Sampling ===		=== Adaptive Sampling ===

	[[File:Kitti_table.png\|500px\|thumb\|right\|<B> Figure 3</B>]]		[[File:Kitti_table.png\|500px\|thumb\|right\|<B> Figure 3</B>: Comparison of various state of the art depth completion methods on the KITTI dataset. Ours outperforms all other methods at very low sampling rates, and outperforms multiple others at increasing sampling rates.]]

	Figure 3 shows that with adaptive sampling, we improve upon our own network's performance significantly. We observe that at low sampling rates, the adaptive sampling is able to outperform state-of-the-art depth completion using random sampling. Figure 4 shows the improvement of adaptive sampling over random sampling and Poisson-disc sampling as the number of samples decreases. Here we observe that adaptive sampling consistently performs better than other sampling strategies, and the amount of performance increase increases as the number of samples decreases. However, we do observe that choosing clever heuristics for sampling such as furthest point sampling also creates greater increases in performance as the number of samples gets low. This is expected, since at lower number of samples the choice of the sampling locations becomes more important to capture all of the information in the scene. Figure 5 shows the reconstructed dense depth images and predicted sampling masks for the adaptive sampling strategy. Here, we see that the adaptive samples cluster to regions where not sampling enough would produce a large MSE loss, like the regions in the image which are furthest away, since these contribute the most to the MSE loss when they are not accurate.		Figure 3 shows that with adaptive sampling, we improve upon our own network's performance significantly. We observe that at low sampling rates, the adaptive sampling is able to outperform state-of-the-art depth completion using random sampling. Figure 4 shows the improvement of adaptive sampling over random sampling and Poisson-disc sampling as the number of samples decreases. Here we observe that adaptive sampling consistently performs better than other sampling strategies, and the amount of performance increase increases as the number of samples decreases. However, we do observe that choosing clever heuristics for sampling such as furthest point sampling also creates greater increases in performance as the number of samples gets low. This is expected, since at lower number of samples the choice of the sampling locations becomes more important to capture all of the information in the scene. Figure 5 shows the reconstructed dense depth images and predicted sampling masks for the adaptive sampling strategy. Here, we see that the adaptive samples cluster to regions where not sampling enough would produce a large MSE loss, like the regions in the image which are furthest away, since these contribute the most to the MSE loss when they are not accurate.

	[[File:Number_samples_kitti.png\|400px\|thumb\|right\|<B> Figure 4</B>]]		[[File:Number_samples_kitti.png\|400px\|thumb\|right\|<B> Figure 4</B>: Comparison of our method using various sampling strategies as the number of samples decreases. Note that all sampling strategies suffer from the same exponential fall-off in performance as the number of samples decreases, however, because of this our adaptive sampling method results in increasing gains as the number of samples decreases.]]

	Qualitative comparisons of other depth completion methods with out adaptive sampling method at low sampling densities are seen in Figure 6. Here, we see that at lower sampling densities our depth images still preserve the high frequency boundaries of objects in the scene seen in the RGB images. This preservation is not observed in other depth completion methods, which blur the boundaries of depth edges in the scene.		Qualitative comparisons of other depth completion methods with out adaptive sampling method at low sampling densities are seen in Figure 6. Here, we see that at lower sampling densities our depth images still preserve the high frequency boundaries of objects in the scene seen in the RGB images. This preservation is not observed in other depth completion methods, which blur the boundaries of depth edges in the scene.

	[[File:Supplementary_sampling_figure_2.png\|700px\|thumb\|left\|<B> Figure 5</B>]]		[[File:Supplementary_sampling_figure_2.png\|700px\|thumb\|left\|<B> Figure 5</B>: Visualization of the learned sampling pattern and the reconstructed dense depth images using our method. Note that the samples cluster in regions of the depth image which correspond to far away (red). This is because getting these depth values wrong would have the largest impact on the MSE error which we are optimizing for.]]

	[[File:Image5513.png\|1000px\|thumb\|center\|<B> Figure 6</B>]]		[[File:Image5513.png\|1000px\|thumb\|center\|<B> Figure 6</B>: Comparison of various depth completion methods with our adaptive sampling method at the ground truth values that were captured using the velodyne in the KITTI dataset. This figure was generated for 156 samples per image. Red regions of the image correspond to higher depths, while blue regions of the image correspond to shallower depths. In the ground truth depth image, the dark blue background corresponds to 0 depth, or a missing depth value. RMSE is measured in millimeters.]]

	=== Depth Completion ===		=== Depth Completion ===

imported>Student221: /* Source Code */

2019-12-14T06:14:50Z

Source Code

← Older revision		Revision as of 06:14, 14 December 2019
Line 172:		Line 172:

	== Source Code ==		== Source Code ==
	The repository containing the source code for the methods and evaluation of this project is available upon request. Contact awb@stanford.edu.		The repository containing the source code for the methods and evaluation of this project is available upon request. Contact awb@stanford.edu. (Don't want to make the code publicly available yet).

imported>Student221: /* Introduction */

2019-12-14T06:12:10Z

Introduction

← Older revision		Revision as of 06:12, 14 December 2019
Line 1:		Line 1:
	== Introduction ==		== Introduction ==

	[[File:Kitti_sampling_comparison.png‎\|400px\|thumb\|right\|<B>Figure 1</B>: LiDAR systems capture sparse 3D point clouds with high accuracy (bottom). A high-resolution RGB image (top) can be processed with a monocular depth estimation algorithm to compute an initial estimate of a dense depth map of the scene (second row), but depth predictions from monocular images contain ambiguity and are not accurate. We propose an adaptive sampling method which is guided by the RGB image and a depth completion network to resolve these ambiguities. Our method (third row) computes dense depth maps with a significantly higher quality than monocular depth estimators and additionally provides the sparse sample locations which result in the best possible inpainted depth images. Note that the ground truth image is also sparse in this figure - however it is far more dense than the sampling densities that we use for reconstruction. RMSE of the predicted dense depth estimated is calculated against the ground truth depth image at points that have defined depth, and is listed in millimeters. Red regions of the image correspond to higher depths, while blue regions of the image correspond to shallower depths.]]		[[File:Kitti_sampling_comparison.png‎\|400px\|thumb\|right\|<B>Figure 1</B>: LiDAR systems capture sparse 3D point clouds with high accuracy (bottom). A high-resolution RGB image (top) can be processed with a monocular depth estimation algorithm to compute an initial estimate of a dense depth map of the scene (second row), but depth predictions from monocular images contain ambiguity and are not accurate. We propose an adaptive sampling method which is guided by the RGB image and a depth completion network to resolve these ambiguities. Our method (third row) computes dense depth maps with a significantly higher quality than monocular depth estimators and additionally provides the sparse sample locations which result in the best possible inpainted depth images. Note that the ground truth image is also sparse in this figure - however it is far more dense than the sampling densities that we use for reconstruction. RMSE of the predicted dense depth estimated is calculated against the ground truth depth image at points that have defined depth, and is listed in millimeters. Red regions of the image correspond to higher depths, while blue regions of the image correspond to shallower depths. In the ground truth depth image, the dark blue background corresponds to 0 depth, or a missing depth value.]]

	Imaging systems using active illumination and time-resolved detectors are able to make precise depth measurements guided by their own light sources. This capability of capturing 3D information is useful for applications such as autonomous vehicle navigation and robotics [1] and remote sensing [2]. With advances in imaging hardware and processing algorithms, light detection and ranging (LiDAR) systems can capture depth images at extremely long range [3], high speed [4], or high resolution. However, there exists a trade-off between these advances to obtain depth images without sacrificing accuracy. One way to address this trade-off is through depth completion, where dense depth is predicted from a sparse set of initial samples and a single RGB image. This removes the requirement to densely scan a scene for high resolution depth images, requiring a significant amount of time. Recent results in depth completion [5-9] have shown promising results on this task, but performance typically degrades sharply for very low depth sampling rates. This intuitively makes sense, since low sampling rates of high frequency details in the depth image prevent perfect reconstruction as governed by the Nyquist-Shannon sampling theorem. Methods using deep learning for depth completion [5-9] can attempt to hallucinate these details, but performance still degrades with low numbers of samples.		Imaging systems using active illumination and time-resolved detectors are able to make precise depth measurements guided by their own light sources. This capability of capturing 3D information is useful for applications such as autonomous vehicle navigation and robotics [1] and remote sensing [2]. With advances in imaging hardware and processing algorithms, light detection and ranging (LiDAR) systems can capture depth images at extremely long range [3], high speed [4], or high resolution. However, there exists a trade-off between these advances to obtain depth images without sacrificing accuracy. One way to address this trade-off is through depth completion, where dense depth is predicted from a sparse set of initial samples and a single RGB image. This removes the requirement to densely scan a scene for high resolution depth images, requiring a significant amount of time. Recent results in depth completion [5-9] have shown promising results on this task, but performance typically degrades sharply for very low depth sampling rates. This intuitively makes sense, since low sampling rates of high frequency details in the depth image prevent perfect reconstruction as governed by the Nyquist-Shannon sampling theorem. Methods using deep learning for depth completion [5-9] can attempt to hallucinate these details, but performance still degrades with low numbers of samples.

imported>Student221: /* Introduction */

2019-12-14T06:11:42Z

Introduction

← Older revision		Revision as of 06:11, 14 December 2019
Line 1:		Line 1:
	== Introduction ==		== Introduction ==

	[[File:Kitti_sampling_comparison.png‎\|400px\|thumb\|right\|<B>Figure 1</B>: LiDAR systems capture sparse 3D point clouds with high accuracy (bottom). A high-resolution RGB image (top) can be processed with a monocular depth estimation algorithm to compute an initial estimate of a dense depth map of the scene (second row), but depth predictions from monocular images contain ambiguity and are not accurate. We propose an adaptive sampling method which is guided by the RGB image and a depth completion network to resolve these ambiguities. Our method (third row) computes dense depth maps with a significantly higher quality than monocular depth estimators and additionally provides the sparse sample locations which result in the best possible inpainted depth images. Note that the ground truth image is also sparse in this figure - however it is far more dense than the sampling densities that we use for reconstruction. RMSE of the predicted dense depth estimated is calculated against the ground truth depth image at points that have defined depth, and is listed in millimeters.]]		[[File:Kitti_sampling_comparison.png‎\|400px\|thumb\|right\|<B>Figure 1</B>: LiDAR systems capture sparse 3D point clouds with high accuracy (bottom). A high-resolution RGB image (top) can be processed with a monocular depth estimation algorithm to compute an initial estimate of a dense depth map of the scene (second row), but depth predictions from monocular images contain ambiguity and are not accurate. We propose an adaptive sampling method which is guided by the RGB image and a depth completion network to resolve these ambiguities. Our method (third row) computes dense depth maps with a significantly higher quality than monocular depth estimators and additionally provides the sparse sample locations which result in the best possible inpainted depth images. Note that the ground truth image is also sparse in this figure - however it is far more dense than the sampling densities that we use for reconstruction. RMSE of the predicted dense depth estimated is calculated against the ground truth depth image at points that have defined depth, and is listed in millimeters. Red regions of the image correspond to higher depths, while blue regions of the image correspond to shallower depths.]]

	Imaging systems using active illumination and time-resolved detectors are able to make precise depth measurements guided by their own light sources. This capability of capturing 3D information is useful for applications such as autonomous vehicle navigation and robotics [1] and remote sensing [2]. With advances in imaging hardware and processing algorithms, light detection and ranging (LiDAR) systems can capture depth images at extremely long range [3], high speed [4], or high resolution. However, there exists a trade-off between these advances to obtain depth images without sacrificing accuracy. One way to address this trade-off is through depth completion, where dense depth is predicted from a sparse set of initial samples and a single RGB image. This removes the requirement to densely scan a scene for high resolution depth images, requiring a significant amount of time. Recent results in depth completion [5-9] have shown promising results on this task, but performance typically degrades sharply for very low depth sampling rates. This intuitively makes sense, since low sampling rates of high frequency details in the depth image prevent perfect reconstruction as governed by the Nyquist-Shannon sampling theorem. Methods using deep learning for depth completion [5-9] can attempt to hallucinate these details, but performance still degrades with low numbers of samples.		Imaging systems using active illumination and time-resolved detectors are able to make precise depth measurements guided by their own light sources. This capability of capturing 3D information is useful for applications such as autonomous vehicle navigation and robotics [1] and remote sensing [2]. With advances in imaging hardware and processing algorithms, light detection and ranging (LiDAR) systems can capture depth images at extremely long range [3], high speed [4], or high resolution. However, there exists a trade-off between these advances to obtain depth images without sacrificing accuracy. One way to address this trade-off is through depth completion, where dense depth is predicted from a sparse set of initial samples and a single RGB image. This removes the requirement to densely scan a scene for high resolution depth images, requiring a significant amount of time. Recent results in depth completion [5-9] have shown promising results on this task, but performance typically degrades sharply for very low depth sampling rates. This intuitively makes sense, since low sampling rates of high frequency details in the depth image prevent perfect reconstruction as governed by the Nyquist-Shannon sampling theorem. Methods using deep learning for depth completion [5-9] can attempt to hallucinate these details, but performance still degrades with low numbers of samples.

imported>Student221: /* Implementation Details */

2019-12-14T06:08:10Z

Implementation Details

← Older revision		Revision as of 06:08, 14 December 2019
Line 75:		Line 75:
	In order to use the monocular depth estimation network as a preprocessing step, we must first train this network and then freeze its parameters while we train the sampling importance flow field prediction network and depth completion network end-to-end. In order to do this, we must split our training dataset into thirds to ensure that the pre-processing network is not operating on data that is in its training set. So, we use the first third of our dataset in order to train the monocular depth estimation network that is used as a preprocessing step for the RGB image, and we use the remaining two thirds to train the depth completion network and sampling importance flow field prediction network. For training the depth completion network and sampling importance flow field prediction, we first begin by training the depth completion network with random sampling patterns. After 15 epochs, we insert our adaptive sampling method and continue training both the depth completion network and sampling importance flow field network end-to-end.		In order to use the monocular depth estimation network as a preprocessing step, we must first train this network and then freeze its parameters while we train the sampling importance flow field prediction network and depth completion network end-to-end. In order to do this, we must split our training dataset into thirds to ensure that the pre-processing network is not operating on data that is in its training set. So, we use the first third of our dataset in order to train the monocular depth estimation network that is used as a preprocessing step for the RGB image, and we use the remaining two thirds to train the depth completion network and sampling importance flow field prediction network. For training the depth completion network and sampling importance flow field prediction, we first begin by training the depth completion network with random sampling patterns. After 15 epochs, we insert our adaptive sampling method and continue training both the depth completion network and sampling importance flow field network end-to-end.

	For the bilateral filter implementation, we fit a neural network to the output of the fast bilateral solver [14] over our entire dataset. This is done because the neural network is faster to run in the forward and backward pass than the bilateral solver, which we desired. This neural network uses a canned U-Net architecture with 4 down-sampling and 4 up-sampling layers. The monocular depth estimation network was taken from [12] and trained according to the default parameters specified in that paper. Our depth completion networks were trained with a learning rate of <math>0.0003</math> and a batch size of <math>12</math>, while our sampling importance field prediction network was trained using a learning rate of <math>10^{-5}</math>.		For the bilateral filter implementation, we fit a neural network to the output of the fast bilateral solver [14] over our entire dataset. This is done because the neural network is faster to run in the forward and backward pass than the bilateral solver, which we desired. This neural network uses a canned U-Net architecture with 4 down-sampling and 4 up-sampling layers. The monocular depth estimation network was taken from [12] and trained according to the default parameters specified in that paper. Our depth completion networks were trained with a learning rate of <math>0.0003</math> and a batch size of <math>12</math>, while our sampling importance field prediction network was trained using a learning rate of <math>10^{-5}</math> and only using the MSE part of the loss (removing the SSIM).

	In order to train the adaptive sampling networks, we expect there to be a dense ground truth depth image to sample from, since in practice we would be directly measuring these values from the scene. In the case of KITTI, however, the ground truth depth images are not dense since they are collected from a velodyne LiDAR. In order to combat this, we inpaint these ground truth depth images with the optimization based method in [15]. This gives us a plausible ground truth depth map, which we then take as ground truth in our reconstruction task. Since we only train our network and evaluate it for accuracy on the sparse ground truth points presented in the KITTI dataset, the validity of this inpainting only comes into question when training the sampling importance flow field.		In order to train the adaptive sampling networks, we expect there to be a dense ground truth depth image to sample from, since in practice we would be directly measuring these values from the scene. In the case of KITTI, however, the ground truth depth images are not dense since they are collected from a velodyne LiDAR. In order to combat this, we inpaint these ground truth depth images with the optimization based method in [15]. This gives us a plausible ground truth depth map, which we then take as ground truth in our reconstruction task. Since we only train our network and evaluate it for accuracy on the sparse ground truth points presented in the KITTI dataset, the validity of this inpainting only comes into question when training the sampling importance flow field.

imported>Student221: /* Implementation Details */

2019-12-14T06:06:37Z

Implementation Details

← Older revision		Revision as of 06:06, 14 December 2019
Line 73:		Line 73:
	=== Implementation Details ===		=== Implementation Details ===

	In order to use the monocular depth estimation network as a preprocessing step, we must first train this network and then freeze its parameters while we train the sampling importance flow field prediction network and depth completion network end-to-end. In order to do this, we must split our training dataset into thirds to ensure that the pre-processing network is not operating on data that is in its training set. So, we use the first third of our dataset in order to train the monocular depth estimation network as a preprocessing step, and we use the remaining two thirds to train the depth completion network and sampling importance flow field prediction network. For training the depth completion network and sampling importance flow field prediction, we first begin by training the depth completion network with random sampling patterns. After 15 epochs, we insert our adaptive sampling method and continue training both the depth completion network and sampling importance flow field network end-to-end.		In order to use the monocular depth estimation network as a preprocessing step, we must first train this network and then freeze its parameters while we train the sampling importance flow field prediction network and depth completion network end-to-end. In order to do this, we must split our training dataset into thirds to ensure that the pre-processing network is not operating on data that is in its training set. So, we use the first third of our dataset in order to train the monocular depth estimation network that is used as a preprocessing step for the RGB image, and we use the remaining two thirds to train the depth completion network and sampling importance flow field prediction network. For training the depth completion network and sampling importance flow field prediction, we first begin by training the depth completion network with random sampling patterns. After 15 epochs, we insert our adaptive sampling method and continue training both the depth completion network and sampling importance flow field network end-to-end.

	For the bilateral filter implementation, we fit a neural network to the output of the fast bilateral solver [14] over our entire dataset. This is done because the neural network is faster to run in the forward and backward pass than the bilateral solver, which we desired. This neural network uses a canned U-Net architecture with 4 down-sampling and 4 up-sampling layers. The monocular depth estimation network was taken from [12] and trained according to the default parameters specified in that paper. Our depth completion networks were trained with a learning rate of <math>0.0003</math> and a batch size of <math>12</math>, while our sampling importance field prediction network was trained using a learning rate of <math>10^{-5}</math>.		For the bilateral filter implementation, we fit a neural network to the output of the fast bilateral solver [14] over our entire dataset. This is done because the neural network is faster to run in the forward and backward pass than the bilateral solver, which we desired. This neural network uses a canned U-Net architecture with 4 down-sampling and 4 up-sampling layers. The monocular depth estimation network was taken from [12] and trained according to the default parameters specified in that paper. Our depth completion networks were trained with a learning rate of <math>0.0003</math> and a batch size of <math>12</math>, while our sampling importance field prediction network was trained using a learning rate of <math>10^{-5}</math>.

imported>Student221: /* 3D Imaging */

2019-12-14T06:05:53Z

3D Imaging

← Older revision		Revision as of 06:05, 14 December 2019
Line 11:		Line 11:
	=== 3D Imaging ===		=== 3D Imaging ===

	Our method is partially motivated by improvements in scanning LiDAR and emerging optical phased array imaging systems [10]. These phased array imaging systems have the unique capability of rapidly generating arbitrary scan patterns and could facilitate the implementation of adaptive sampling algorithms in 3D imaging applications. This is because our adaptive sampling system is designed with the idea that our system could take arbitrary scan patterns at no additional resolution, speed, or range cost, and phased array imaging systems meet this criterion. Thus, this work opens discussion of the discussion of the development of adaptive sampling algorithms which are sure to become important as imaging systems advance in this direction.		Our method is partially motivated by improvements in scanning LiDAR and emerging optical phased array imaging systems [10]. These phased array imaging systems have the unique capability of rapidly generating arbitrary scan patterns and could facilitate the implementation of adaptive sampling algorithms in 3D imaging applications. This is because our adaptive sampling system is designed with the idea that our system could take arbitrary scan patterns at no additional resolution, speed, or range cost, and phased array imaging systems meet this criterion. Note that for traditional scanning LiDAR systems, solely based on the time-of-flight of light alone, in order to take measurements at 200 meters, 750 measurements could take up to 1 millisecond. This is not including the time required to steer the laser to each of these points. Thus, this work opens discussion of the discussion of the development of adaptive sampling algorithms which are sure to become important as imaging systems advance in this direction.

	=== Depth Estimation ===		=== Depth Estimation ===

imported>Student221: /* Loss functions & Regularization */

2019-12-14T06:02:44Z

Loss functions & Regularization

← Older revision		Revision as of 06:02, 14 December 2019
Line 61:		Line 61:
	<math> \mathcal{L}_{prediction} = \|\|d_{gt} - \hat{d}\|\|^2 + 0.5\cdot SSIM(\hat{d}, \hat{d}_m) </math>		<math> \mathcal{L}_{prediction} = \|\|d_{gt} - \hat{d}\|\|^2 + 0.5\cdot SSIM(\hat{d}, \hat{d}_m) </math>

	In order to use the grid based sampling as a prior, we implement a regularization on the predicted sampling importance flow field <math> V </math> which penalizes large vectors. This intuitively corresponds to penalizing moving the samples too far from their starting positions. This regularization is given by:		In order to use the grid based sampling as a prior, we implement a regularization on the predicted sampling importance flow field <math> V </math> which penalizes large vectors. This intuitively corresponds to penalizing moving the samples too far from their starting grid positions, essentially defining these grid positions as a prior for the image sampling. This regularization is given by:

	<math> \mathcal{L}_{field} = \|\|V\|\|^2 </math>		<math> \mathcal{L}_{field} = \|\|V\|\|^2 </math>

imported>Student221: /* Methods */

2019-12-14T05:56:48Z

Methods

← Older revision		Revision as of 05:56, 14 December 2019
Line 29:		Line 29:
	Our method for depth completion and adaptive sampling is outlined in Figure 2. It takes as input an RGB image, and outputs a reconstructed dense depth image. This is done by determining locations to sample for depth from the RGB image, sampling those locations, and then using those sparse samples and RGB image to reconstruct a dense depth image of the scene.		Our method for depth completion and adaptive sampling is outlined in Figure 2. It takes as input an RGB image, and outputs a reconstructed dense depth image. This is done by determining locations to sample for depth from the RGB image, sampling those locations, and then using those sparse samples and RGB image to reconstruct a dense depth image of the scene.

	[[File:Methods_figure2.png\|1000px\|thumb\|center\|<B> Figure 2</B>: Our adaptive sampling deep network takes as input an RGB image, and predicts optimal sampling pattern and reconstructed dense depth from sampling at these locations. A pre-trained monocular depth estimation network is used to make an initial dense depth estimate of the scene from the RGB image, essentially pre-processing that image. A U-Net is used to extract a sampling importance vector field from the monocular depth estimate. This vector field is integrated in order to decide the locations to sample in this scene. The depth completion network fuses together the coarsely inpainted sparse depth samples with the monocular depth estimate and predicts a final dense depth.]]		[[File:Methods_figure2.png\|1000px\|thumb\|center\|<B> Figure 2</B>: Our adaptive sampling deep network takes as input an RGB image, and predicts optimal sampling pattern and reconstructed dense depth from sampling at these locations. A pre-trained monocular depth estimation network is used to make an initial dense depth estimate of the scene from the RGB image, essentially pre-processing that image. A U-Net is used to extract a sampling importance vector field from the monocular depth estimate. This vector field is integrated in order to decide the locations to sample in this scene. The depth completion network fuses together the coarsely inpainted sparse depth samples with the monocular depth estimate and predicts a final dense depth. The example image is from the NYU-Depth-v2 dataset [18]]]

	=== Preprocessing & Depth Completion ===		=== Preprocessing & Depth Completion ===