YueYawenXianzhe

Introduction

Depth map estimation is important in understanding 3D geometric in the scene. The goal of our project is to perform depth prediction from stereo image pairs. In a simple stereo system, given two images taken from a camera at two different locations with a transition matrix $t=[T,0,0]$ , we want to compute the disparity for each pixel on the left image. Disparity is defined as $d=x_l-x_r$ where $x_l$ is the horizontal location of the pixel on the left image, and $x_r$ is the horizontal location of the corresponding pixel on the right image. The depth of the object is then computed following the similar triangle rule with the equation: $z=\frac{f\cdot T}{d}$, where $f$ is the focal length, $T$ is the transition of the camera. In this project, we train a convolutional neural network to predict the depth map from stereo image pairs. The architecture is composed of two deep networks, the first one makes a prediction based on the global information, the second one tries to combine the initial stereo image and the global prediction result as the input and refine the output based on local information. We used the dataset generated by our Course Assistant, Zheng Lyu, from ISET 3D, which contains 238 stereo image pairs. The dataset mainly consists of scenes on the streets.

Background

A popular four-step pipeline has been developed for stereo matching, including matching cost calculation, matching cost aggregation, disparity calculation and disparity refinement[1]. Each step of the pipeline is important to the overall stereo matching performances. Due to the complexity of designing traditional methods, researchers are trying machine learning techniques to improve stereo matching performances and find them outperform traditional methods. Zbontar[2] first introduced CNN for matching cost calculation. They trained a CNN classifier to measure the similarity of two pixels of two images and found that CNN can learn more robust and discriminative features from images, and generates improved stereo matching cost. Kendall[3] integrated matching cost calculation, matching cost aggregation and disparity calculation into a neural network, which achieves higher accuracy and computational efficiency. Jiahao[3] incorporates disparity refinement in a CNN to refine disparity.

Methods

Loss Function

We use the loss function defined as follows: $L(y,y^{*})=\frac{1}{n}\sum _i{d}_i^2-\frac{\lambda }{n^2}(\sum _i{d}_i{)}^2,$ where $d_i=\log y_i-y_i^{*}$ and $\lambda \in [0,1]$.

Evaluation Matrix

We computed several evaluation matrix defined as follows:

Abs Relative Difference

$\frac{1}{|T|}\sum _{y\in T}|y-y^{*}|/y^{*}$

Squared Relative Difference

$\frac{1}{|T|}\sum _{y\in T}||y-y^{*}|{|}^2/y^{*}$

RMSE(linear)

$\sqrt{\frac{1}{|T|}\sum _{y\in T}||y-y^{*}|{|}^2}$

RMSE(log)

$\sqrt{\frac{1}{|T|}\sum _{y\in T}||y-y^{*}|{|}^2}$

References

[1]. D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47:742, 2002.

[2]. J. Zbontar and Y. LeCun. Stereo matching by training a convolutional neural network to compare image patches. Journal of Machine Learning Research, 17(1- 32):2, 2016.

[3]. A. Kendall, H. Martirosyan, S. Dasgupta, P. Henry, R. Kennedy, A. Bachrach, and A. Bry. End-to-end learning of geometry and context for deep stereo regression. In IEEE Conference on Computer Vision and Pattern Recognition, 2017

Appendix

You can write math equations as follows: $y=x+5$

You can include images as follows (you will need to upload the image first using the toolbox on the left bar, using the "Upload file" link).