Bergman

Introduction

Figure 1: LiDAR systems capture sparse 3D point clouds with high accuracy (bottom). A high-resolution RGB image (top) can be processed with a monocular depth estimation algorithm to compute an initial estimate of a dense depth map of the scene (second row), but depth predictions from monocular images contain ambiguity and are not accurate. We propose an adaptive sampling method which is guided by the RGB image and a depth completion network to resolve these ambiguities. Our method (third row) computes dense depth maps with a significantly higher quality than monocular depth estimators and additionally provides the sparse sample locations which result in the best possible inpainted depth images. Note that the ground truth image is also sparse in this figure - however it is far more dense than the sampling densities that we use for reconstruction. RMSE of the predicted dense depth estimated is calculated against the ground truth depth image at points that have defined depth, and is listed in millimeters.

Imaging systems using active illumination and time-resolved detectors are able to make precise depth measurements guided by their own light sources. This capability of capturing 3D information is useful for applications such as autonomous vehicle navigation and robotics [1] and remote sensing [2]. With advances in imaging hardware and processing algorithms, light detection and ranging (LiDAR) systems can capture depth images at extremely long range [3], high speed [4], or high resolution. However, there exists a trade-off between these advances to obtain depth images without sacrificing accuracy. One way to address this trade-off is through depth completion, where dense depth is predicted from a sparse set of initial samples and a single RGB image. This removes the requirement to densely scan a scene for high resolution depth images, requiring a significant amount of time. Recent results in depth completion [5-9] have shown promising results on this task, but performance typically degrades sharply for very low depth sampling rates. This intuitively makes sense, since low sampling rates of high frequency details in the depth image prevent perfect reconstruction as governed by the Nyquist-Shannon sampling theorem. Methods using deep learning for depth completion [5-9] can attempt to hallucinate these details, but performance still degrades with low numbers of samples.

In this project, we propose an imaging system which obtains dense depth maps from an RGB image and sparse depth measurements generated by a scene-adaptive scanning pattern. Our method is based on a novel deep neural network architecture for solving the depth completion task, and a deep learning architecture for predicting the sampling locations, which can be trained in an end-to-end fashion. An outline of the results of this system is shown in Figure 1. We show that exploiting adaptive sampling by predicting depth sampling locations from an RGB image improves performance of depth completion networks, especially at low sampling rates.

Background

3D Imaging

Our method is partially motivated by improvements in scanning LiDAR and emerging optical phased array imaging systems [10]. These phased array imaging systems have the unique capability of rapidly generating arbitrary scan patterns and could facilitate the implementation of adaptive sampling algorithms in 3D imaging applications. This is because our adaptive sampling system is designed with the idea that our system could take arbitrary scan patterns at no additional resolution, speed, or range cost, and phased array imaging systems meet this criterion. Thus, this work opens discussion of the discussion of the development of adaptive sampling algorithms which are sure to become important as imaging systems advance in this direction.

Depth Estimation

Early methods in monocular depth estimation used hand-crafted features and graphical models [11] to map monocular images to depth given a large repository of RGB-depth data. Since the success of deep learning and convolutional neural networks (CNNs), these tools have been used to directly learn a mapping from monocular images to dense depth [12]. However, due to the ambiguity of absolute depth values in estimation from monocular images, these methods struggle in producing accurate depth estimates. Often times, the resulting estimated depth is off by a scale and bias factor, i.e. these methods produce accurate ordinal depth and edges, but not accurate depth values.

The depth completion task was proposed to help resolve some of the ambiguities in monocular depth estimation by fusing a monocular image and a few sparse depth samples to predict a dense depth images. The reasoning behind this was that if the monocular depth images were off by a scale and bias factor, the few sparse depth samples would be enough to anchor the absolute depth values and produce both accurate ordinal depth and actual depth values. The advent of deep learning brought upon numerous CNN modifications and architectures designed to predict dense depth [5-9]. Still other non-learning based methods have used bilateral filters [14] and optimization [15] to solve the depth completion problem without requiring large datasets to train deep learning models.

Our adaptive sampling system proposed to provide generalization one step further for depth estimation; now our system has the freedom to decide where to sample along with designing a depth completion system. Still, the goal of this system remains to produce the best dense depth estimate of a scene.

Adaptive Sampling

Previous approaches for predicting sample locations proposed sampling heuristics to capture the underlying image signal. For images, furthest point sampling [16] was proposed as a good heuristic to sample images for later image reconstruction. More sophisticated heuristics for progressive image sampling have been proposed based on statistical information in regions of the image or wavelets [17], and come close to adaptive sampling, but do not optimize sample locations for a specific upstream inference task such as dense depth reconstruction. Our formulation allows us to directly optimize these sampling locations using first order optimization methods in order to improve an upstream loss rather than relying on sampling patterns that we already know to be successful for various tasks.

Methods

Our method for depth completion and adaptive sampling is outlined in Figure 2. It takes as input an RGB image, and outputs a reconstructed dense depth image. This is done by determining locations to sample for depth from the RGB image, sampling those locations, and then using those sparse samples and RGB image to reconstruct a dense depth image of the scene.

Figure 2: Our adaptive sampling deep network takes as input an RGB image, and predicts optimal sampling pattern and reconstructed dense depth from sampling at these locations. A pre-trained monocular depth estimation network is used to make an initial dense depth estimate of the scene from the RGB image, essentially pre-processing that image. A U-Net is used to extract a sampling importance vector field from the monocular depth estimate. This vector field is integrated in order to decide the locations to sample in this scene. The depth completion network fuses together the coarsely inpainted sparse depth samples with the monocular depth estimate and predicts a final dense depth.

Preprocessing & Depth Completion

We have observed that many depth completion networks perform much worse with a low number of sparse samples. Prior work [13] has shown that traditional convolution kernels used in CNNs are not well suited for sparse images. Additionally, as we can observe in the comparison results figures, many depth completion networks blur the high frequency details in the depth image such as the boundaries between objects in the depth domain. This results in depth images which are accurate in the MSE sense, but since this loss function doesn't necessarily represent perception and edges well, then the network learns to disregard these details. This is in contrast to many monocular depth estimation networks which attempt to reconstruct depth from only an RGB image. Since these networks are generally trained using SSIM losses or losses on the gradient of the depth images (corresponding to the edges in the depth domain), these networks produce depth images which capture high frequency details well.

Based on these observations, we propose preprocessing the RGB data with a monocular depth estimation network to predict a dense depth map from the input RGB image. This way, we input to our depth completion network an image in the depth domain which accurately captures high frequency edge details. This is shown in Figure 2 as the monocular depth estimation network. We also use a bilateral filter to roughly inpaint the captured sparse depth image, shown in the bilateral filter block of Figure 2. This gets around the problematic sparse images inputted into the neural network, as it estimates a depth value for every pixel. Additionally, we tune the bilateral filter inpainting in such a way that it does not capture high frequency details in the depth domain and instead just blurs the sparse depth out with minor edge awareness. With these two inputs to our depth completion network, our network only needs to learn to apply a high pass filter to the monocular depth estimate and a low pass filter to the bilateral inpainted sparse depth. Additionally, since both images are in the depth domain, we expect our network to have an easier time learning a mapping to dense depth images than if we had fused RGB images with sparse depth like many other depth completion networks do.

For the depth completion network itself, we use an early fusion architecture, meaning that the inputs are concatenated together at the input of the network without gone through deep learning blocks to learn features independently. For the two datasets that we perform experiments on, NYU-Depth-v2 [18] and KITTI Depth [20], we propose two different neural network architectures. For NYU-Depth-v2, we propose a simple U-Net architecture [19] with 4 down-sampling and up-sampling layers, each of whcih contains a convolution, batch normalization, and ReLU. We then modify this simple U-Net structure to concatenate the input into the convolutional layer at each up-sampling step. For the KITTI dataset, we use the fusion network proposed in [9], which is based on ResNet-34 [21] and also has 4 down-sampling and up-sampling residual blocks. This architecture is referred to as the depth completion network in Figure 2.

References

[1] B. Schwarz, “LIDAR: Mapping the world in 3D,” Nature Photonics, vol. 4, pp. 429–430, 2010.

[2] U. Weiss and P. Biber, “Plant detection and mapping for agricultural robots using a 3D LIDAR sensor,” Robotics and Autonomous Systems, vol. 59, pp. 265–273, 2011.

[3] A. M. Pawlikowska, A. Halimi, R. A. Lamb, and G. S. Buller, “Single-photon three-dimensional imaging at up to 10 kilometers range,” Optics Express, vol. 25, no. 10, pp. 11 919–11 931, 2017.

[4] D. B. Lindell, M. OToole, and G. Wetzstein, “Single-Photon 3D Imaging with Deep Sensor Fusion,” ACM Trans. Graph. (SIG548 GRAPH), no. 4, 2018.

[5] F. Ma and S. Karaman, “Sparse-to-dense: Depth prediction from sparse depth samples and a single image,” ICRA, 2018.

[6] A. Eldesokey, M. Felsberg, and F. Khan, “Confidence propagation through cnns for guided sparse depth regression,” IEEE PAMI, 2019.

[7] W. Van Gansbeke, D. Neven, B. De Brabandere, and L. Van Gool, “Sparse and noisy lidar completion with rgb guidance and un578 certainty,” in 2019 16th International Conference on Machine Vision Applications (MVA), 2019.

[8] X. Cheng, P. Wang, and R. Yang, “Depth estimation via affinity learned with convolutional spatial propagation network,” in ECCV, 2018.

[9] F. Ma, G. V. Cavalheiro, and S. Karaman, “Self-supervised sparse-to-dense: Self-supervised depth completion from lidar and monocular camera,” ICRA, 2019.

[10] J. Sun, E. Timurdogan, A. Yaacobi, E. S. Hosseini, and M. R. Watts, “Large-scale nanophotonic phased array,” Nature, vol. 493, pp. 195–199, 2013.

[11] A. Saxena, S. H. Chung, and A. Y. Ng, “Learning depth from single monocular images,” in Advances in Neural Information Processing Systems, 2006.

[12] I. Alhashim and P. Wonka, “High quality monocular depth estimation via transfer learning,” arXiv:1812.11941, 2018.

[13] J. Uhrig, N. Schneider, L. Schneider, U. Franke, T. Brox, and A. Geiger, “Sparsity invariant cnns,” International Conference on 3D Vision (3DV), 2017.

[14] J. T. Barron and B. Poole, “The fast bilateral solver,” in ECCV, 2016.

[15] A. Levin, D. Lischinski, and Y. Weiss, “Colorization using optimization,” in ACM SIGGRAPH, 2004.

[16] Y. Eldar, M. Lindenbaum, M. Porat, and Y. Y. Zeevi, “The Farthest Point Strategy for Progressive Image Sampling,” IEEE TIP, vol. 6, no. 9, pp. 1305–1315, 1997.

[17] V. Saragadam and A. Sankaranarayanan, “Wavelet tree parsing with freeform lensing,” in IEEE ICCP, 2019.

[18] P. K. Nathan Silberman, Derek Hoiem and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in ECCV, 2012.

[19] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” arXiv:1505.04597, 2015.

[20] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The KITTI dataset,” International Journal of Robotics Research (IJRR), 2013.

[21] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” arXiv:1512.03385, 2015.

[22] R. Bridson, “Fast poisson disk sampling in arbitrary dimensions,” in ACM SIGGRAPH 2007 Sketches, ser. SIGGRAPH ’07, 2007.

[23] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in PyTorch,” in NeurIPS Autodiff Workshop, 2017.

[24] M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu, “Spatial transformer networks,” in Advances in Neural Information Processing Systems, 2015.

Appendix

This work is a collaboration with David B. Lindell and Gordon Wetzstein of the Stanford Computational Imaging Group. The team member contribution is broken down as follows:

Alexander W. Bergman (PSYCH 221 student) performed the literature review and analysis, helped brainstorm the solution method, implemented the methods and collected the results, formulated the conclusion drawn from the results, and wrote the report.

David B. Lindell (non-PSYCH 221 student) proposed the idea for the project, helped with the literature review, provided guidance and suggestions on the methods, helped speculate on the interpretation of the results.

Gordon Wetzstein (non-PSYCH 221 student) defined the motivation for the project and desired results to pursue, helped with the literature review, provided guidance and suggestion on the methods, and provided the computing resources for developing the project.

I would like to thank my collaborators for their suggestions and guidance in my development of this project - without their input this project would not be where it is now.

Source Code

The repository containing the source code for the methods and evaluation of this project is available upon request. Contact awb@stanford.edu.

Useful: You can write math equations as follows: $y=x+5$