HouSetra

Introduction

There are several examples of popular tools for filter creation. A popular one is Instagram, released in 2010 and acquired by Facebook in 2012. Instagram has several filters, most of which involve modifying the image's colors based on its existing RGB values, for example converting an RGB image into greyscale or boosting the red value.

Another one is Prisma, which makes use of convolutional neural networks to transfer artistic qualities of one image onto another [1]. This allows one to transform a sample into to have the style of a famous artist.

Our goal is to create filters as well, but with a focus on applying the depth of an image onto the filter properties.

Methods

Data Capture

We captured RGB and Depth images with two tools: the Intel Realsense R200 world-facing camera and the Google Camera app.

Fig. 1 Intel Realsense R200.

Fig. 2 Intel Realsense R200.

The Intel camera (shown above) captures RGB images at up to 1920x1080 resolution, and depth images at up to 480x360 resolution. In order to keep the sizes consistent, we kept the resolution of both at 480x360. The depth maps themselves are created on-board the camera by first texturing the scene with an infrared projector (Fig. 2). The two infrared cameras then capture the textured scene and a depth map is created from the disparity between the two images. For best results, the depth map is rated at 0.5m-4m indoors, and up to 10m outdoors, although specular objects will produce inaccurate depth values. Examples of captured images are below. We used Visual Studio to program the camera.

Fig. 3 Example of Intel camera RGB image.

Fig. 4 Example of Intel camera depth image.

As can be seen from Fig. 4, there are a large amount of dark black areas. These are referred to as holes, and are areas where the camera could not accurately detect a depth value. We will need to fill these in before using the depth maps.

In order to fully test our filters, we also captured images using the Google Camera application. This application captures several images of a scene by moving the camera slowly. It then estimates depth values through the disparity of these images and outputs a much denser depth map than the Intel camera. The resolution of the resultant RGB and depth image are 1440x1080. Examples of this are below.

Fig. 5 Example of Google Camera RGB image.

Fig. 6 Example of Google Camera depth image.

Hole Filling

As mentioned before, we need some sort of hole filling on the Intel camera's depth images in order to use them more effectively. The hole filling algorithm we used was a content-aware bilateral filter. The plan is to first flag pixels with incorrect depth values (the black pixels in the depth image). Then for each flagged pixel $p$ with depth value $d$ and RGB values $c=(r,g,b)$, we create a neighborhood around the pixel of non-flagged pixels $(p_1,p_2,\mathrm{...},p_N)$ with depth and color values $(d_1,d_2,\mathrm{...},d_N)$ and $(c_1,c_2,\mathrm{...},c_N)$ respectively. In practice we created a $10\times 10$ window centered on $p$ (see Fig. 7). If there are no non-flagged pixels in the chosen neighborhood, then the pixel is skipped.

Fig. 7 Example of a pixel neighborhood. The red dot is a chosen pixel, and the red outline is the neighborhood boundary.

Now for each pixel in the neighborhood, we calculate a weight $w_i=e^{-k||c-c_i|{|}^2}$ with the $L2$ norm in the exponent. For our images we used a value of $k=0.05$. The idea here is that nearby pixels with similar colors should have similar depths, and so those should have a higher weight. Then finally we update the original pixel depth value by setting:

$d=\frac{\sum _i^N{w}_i{d}_i}{\sum _i^N{w}_i}$

This process is then repeated for four iterations, because at each iteration only flagged pixels neighboring non-flagged pixels will be updated. The results are as shown below in Fig. 9. Notice that there are speckles, or small dark dots. These are not black (missing) values, but are simply inaccuracies which where not flagged and not fixed in filtering. To remedy this, we applied one more pass of the bilateral filter, but we made use of all of the pixels, and did not ignore any flagged pixels. The final result is below in Fig. 10.

Fig. 8 Original depth image.

Fig. 9 Depth image after initial bilateral filter.

Fig. 10 Final Depth image after second bilateral filter.

Results

Hole filling

Fig. 8 Original depth image.

Fig. 9 Depth image after initial bilateral filter.

Fig. 10 Final Depth image after second bilateral filter.

References

[1] http://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Gatys_Image_Style_Transfer_CVPR_2016_paper.pdf