YueYawenXianzhe

Introduction

Depth map estimation is important in understanding 3D geometric in the scene. The goal of our project is to perform depth prediction from stereo image pairs. In a simple stereo system, given two images taken from a camera at two different locations with a transition matrix $t=[T,0,0]$ , we want to compute the disparity for each pixel on the left image. Disparity is defined as $d=x_l-x_r$ where $x_l$ is the horizontal location of the pixel on the left image, and $x_r$ is the horizontal location of the corresponding pixel on the right image. The depth of the object is then computed following the similar triangle rule with the equation: $z=\frac{f\cdot T}{d}$, where $f$ is the focal length, $T$ is the transition of the camera. In this project, we train a convolutional neural network to predict the depth map from stereo image pairs. The architecture is composed of two deep networks, the first one makes a prediction based on the global information, the second one tries to combine the initial stereo image and the global prediction result as the input and refine the output based on local information. We used the dataset generated by our Course Assistant, Zheng Lyu, from ISET 3D, which contains 238 stereo image pairs. The dataset mainly consists of scenes on the streets.

Background

A popular four-step pipeline has been developed for stereo matching, including matching cost calculation, matching cost aggregation, disparity calculation and disparity refinement[1]. Each step of the pipeline is important to the overall stereo matching performances. Due to the complexity of designing traditional methods, researchers are trying machine learning techniques to improve stereo matching performances and find them outperform traditional methods. Zbontar[2] first introduced CNN for matching cost calculation. They trained a CNN classifier to measure the similarity of two pixels of two images and found that CNN can learn more robust and discriminative features from images, and generates improved stereo matching cost. Kendall[3] integrated matching cost calculation, matching cost aggregation and disparity calculation into a neural network, which achieves higher accuracy and computational efficiency. Jiahao[3] incorporates disparity refinement in a CNN to refine disparity.

Methods

Model Architecture

Enlighted by the model in [4], we designed our model architecture as shown in the diagram. Our model could be divided into 2 phases: CoarseNet and RefineNet.

The top part is the CoarseNet. We replaced the fully convolutional layer in [4] with a residual convolutional block which contains a residual convolutional layer and a fully convolutional layer. We feed the left image and right image of the stereo camera into the model separately. After two layers of ResConv block, we sum their result up. After several other ResConv blocks, we flatten the 2D matrix into a vector. Passing through two fully connected layers and transform it, we could get a coarse depth map.

The bottom part is the RefineNet, we process the concatenated images with one layer of ResConv, concatenate the output with the coarse depth map. After another two ResConv blocks, we could get the final refined depth map prediction.

Data Augmentation

We performed data augmentation on the original dataset. We mainly augmented it by cropping the image. One strategy is to crop the image around the center of the camera. Another strategy is to perform random cropping on the original image. Both strategies aim at maintaining more details of the original dataset.

Loss Function

We use the loss function defined as follows: $L(y,y^{*})=\frac{1}{n}\sum _i{d}_i^2-\frac{\lambda }{n^2}(\sum _i{d}_i{)}^2,$ where $d_i=\log y_i-y_i^{*}$ and $\lambda \in [0,1]$.

Evaluation Matrix

We computed several evaluation matrix defined as follows:

Abs Relative Difference

$\frac{1}{|T|}\sum _{y\in T}|y-y^{*}|/y^{*}$

Squared Relative Difference

$\frac{1}{|T|}\sum _{y\in T}||y-y^{*}|{|}^2/y^{*}$

RMSE(linear)

$\sqrt{\frac{1}{|T|}\sum _{y\in T}||y-y^{*}|{|}^2}$

RMSE(log)

$\sqrt{\frac{1}{|T|}\sum _{y\in T}||y-y^{*}|{|}^2}$

Results

Conclusions

From the result above, we can find that our model learned limited information from the input data and there are more aspects to improve. Especially, it seems that our model tends to learn more about the intensity of the input image instead of the disparity information. The output of our model has less error when it estimates the disparity of a test image with a bright sky. However, since our input data contains a lot of street images with a sky, it tends to predict all brighter input with larger depth, just like the sky. This suggests that with a dataset of relatively similar scenes, our model tends to learn the characteristics of the object itself, such as intensity, instead of its disparity. Moreover, to improve our work, we can continue to refine the architecture of the mode, perform more data augmentation, tune hyperparameters, and test on other datasets for more precise depth estimation.

References

[1]. D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47:742, 2002.

[2]. J. Zbontar and Y. LeCun. Stereo matching by training a convolutional neural network to compare image patches. Journal of Machine Learning Research, 17(1- 32):2, 2016.

[3]. A. Kendall, H. Martirosyan, S. Dasgupta, P. Henry, R. Kennedy, A. Bachrach, and A. Bry. End-to-end learning of geometry and context for deep stereo regression. In IEEE Conference on Computer Vision and Pattern Recognition, 2017

[4] Eigen D, Puhrsch C, Fergus R. Depth map prediction from a single image using a multi-scale deep network. InAdvances in neural information processing systems 2014 (pp. 2366-2374).

Appendix I: Code

Github Page: https://github.com/xianzhez/stereo_depth_map

Appendix II: Work Breakdown

Three of us worked together on implementing the model and solve the problems we met.

Yawen worked on data preprocessing, data augmentation, and the result analysis.

Yue worked on data preprocessing, data augmentation, and result visualization.

Xianzhe worked on designing, training the model, tuning parameters and testing performance.

Appendix III: Acknoledgement

Brian and Joyce gave us excellent lectures and helpful advice on projects.

Zheng helped us on project topic selection and generated the dataset for us with ISET3D.