YueYawenXianzhe

Introduction

Depth map estimation is important in understanding 3D geometric in the scene. The goal of our project is to perform depth prediction from stereo image pairs. In a simple stereo system, given two images taken from a camera at two different locations with a transition matrix $t=[T,0,0]$ , we want to compute the disparity for each pixel on the left image. Disparity is defined as $d=x_l-x_r$ where $x_l$ is the horizontal location of the pixel on the left image, and $x_r$ is the horizontal location of the corresponding pixel on the right image. The depth of the object is then computed following the similar triangle rule with the equation: $z=\frac{f\cdot T}{d}$, where $f$ is the focal length, $T$ is the transition of the camera. In this project, we train a convolutional neural network to predict the depth map from stereo image pairs. The architecture is composed of two deep networks, the first one makes a prediction based on the global information, the second one tries to combine the initial stereo image and the global prediction result as the input and refine the output based on local information. We used the dataset generated by our Course Assistant, Zheng Lyu, from ISET 3D, which contains 238 stereo image pairs. The dataset mainly consists of scenes on the streets.

Background

A popular four-step pipeline has been developed for stereo matching, including matching cost calculation, matching cost aggregation, disparity calculation and disparity refinement[1]. Each step of the pipeline is important to the overall stereo matching performances. Due to the complexity of designing traditional methods, researchers are trying machine learning techniques to improve stereo matching performances and find them outperform traditional methods. Zbontar[2] first introduced CNN for matching cost calculation. They trained a CNN classifier to measure the similarity of two pixels of two images and found that CNN can learn more robust and discriminative features from images, and generates improved stereo matching cost. Kendall[3] integrated matching cost calculation, matching cost aggregation and disparity calculation into a neural network, which achieves higher accuracy and computational efficiency. Jiahao[3] incorporates disparity refinement in a CNN to refine disparity.

Methods

Data Augmentation

We performed data augmentation on the original dataset. We mainly augmented it by cropping the image. One strategy is to crop the image around the center of the camera. Another strategy is to perform random cropping on the original image. Both strategies aim at maintaining more details of the original dataset.

Loss Function

We use the loss function defined as follows: $L(y,y^{*})=\frac{1}{n}\sum _i{d}_i^2-\frac{\lambda }{n^2}(\sum _i{d}_i{)}^2,$ where $d_i=\log y_i-y_i^{*}$ and $\lambda \in [0,1]$.

Evaluation Matrix

We computed several evaluation matrix defined as follows:

Abs Relative Difference

$\frac{1}{|T|}\sum _{y\in T}|y-y^{*}|/y^{*}$

Squared Relative Difference

$\frac{1}{|T|}\sum _{y\in T}||y-y^{*}|{|}^2/y^{*}$

RMSE(linear)

$\sqrt{\frac{1}{|T|}\sum _{y\in T}||y-y^{*}|{|}^2}$

RMSE(log)

$\sqrt{\frac{1}{|T|}\sum _{y\in T}||y-y^{*}|{|}^2}$

Results

Conclusions

From the result above, we can find that our model learned limited information from the input data and there are more aspects to improve. Especially, it seems that our model tends to learn more about the intensity of the input image instead of the disparity information. The output of our model has less error when it estimates the disparity of a test image with a bright sky. However, since our input data contains a lot of street images with a sky, it tends to predict all brighter input with larger depth, just like the sky. This suggests that with a dataset of relatively similar scenes, our model tends to learn the characteristics of the object itself, such as intensity, instead of its disparity. Moreover, to improve our work, we can continue to refine the architecture of the mode, perform more data augmentation, tune hyperparameters, and test on other datasets for more precise depth estimation.

References

[1]. D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47:742, 2002.

[2]. J. Zbontar and Y. LeCun. Stereo matching by training a convolutional neural network to compare image patches. Journal of Machine Learning Research, 17(1- 32):2, 2016.

[3]. A. Kendall, H. Martirosyan, S. Dasgupta, P. Henry, R. Kennedy, A. Bachrach, and A. Bry. End-to-end learning of geometry and context for deep stereo regression. In IEEE Conference on Computer Vision and Pattern Recognition, 2017

Appendix II: Work Breakdown

Three of us worked together on implementing the model. Yawen and Yue worked on data preprocessing and analysis. Xianzhe worked on refining the model and tuning parameters.