HouSetra

Introduction

There are several examples of popular tools for filter creation. A popular one is Instagram, released in 2010 and acquired by Facebook in 2012. Instagram has several filters, most of which involve modifying the image's colors based on its existing RGB values, for example converting an RGB image into greyscale or boosting the red value.

Another one is Prisma, which makes use of convolutional neural networks to transfer artistic qualities of one image onto another [1]. This allows one to transform a sample into to have the style of a famous artist.

Our goal is to create filters as well, but with a focus on applying the depth of an image onto the filter properties.

Methods

Data Capture

We captured RGB and Depth images with two tools: the Intel Realsense R200 world-facing camera and the Google Camera app.

Fig. 1 Intel Realsense R200.

Fig. 2 Intel Realsense R200.

The Intel camera (shown above) captures RGB images at up to 1920x1080 resolution, and depth images at up to 480x360 resolution. In order to keep the sizes consistent, we kept the resolution of both at 480x360. The depth maps themselves are created on-board the camera by first texturing the scene with an infrared projector (Fig. 2). The two infrared cameras then capture the textured scene and a depth map is created from the disparity between the two images. For best results, the depth map is rated at 0.5m-4m indoors, and up to 10m outdoors, although specular objects will produce inaccurate depth values. Examples of captured images are below. We used Visual Studio to program the camera.

Fig. 3 Example of Intel camera RGB image.

Fig. 4 Example of Intel camera depth image.

As can be seen from Fig. 4, there are a large amount of dark black areas. These are referred to as holes, and are areas where the camera could not accurately detect a depth value. We will need to fill these in before using the depth maps.

In order to fully test our filters, we also captured images using the Google Camera application. This application captures several images of a scene by moving the camera slowly. It then estimates depth values through the disparity of these images and outputs a much denser depth map than the Intel camera. The resolution of the resultant RGB and depth image are 1440x1080. Examples of this are below.

Fig. 5 Example of Google Camera RGB image.

Fig. 6 Example of Google Camera depth image.

Hole Filling

As mentioned before, we need some sort of hole filling on the Intel camera's depth images in order to use them more effectively. The hole filling algorithm we used was a content-aware bilateral filter. The plan is to first flag pixels with incorrect depth values (the black pixels in the depth image). Then for each flagged pixel $p$ with depth value $d$ and RGB values $c=(r,g,b)$, we create a neighborhood around the pixel of non-flagged pixels $(p_1,p_2,\mathrm{...},p_N)$ with depth and color values $(d_1,d_2,\mathrm{...},d_N)$ and $(c_1,c_2,\mathrm{...},c_N)$ respectively. In practice we created a $10\times 10$ window centered on $p$ (see Fig. 7). If there are no non-flagged pixels in the chosen neighborhood, then the pixel is skipped.

Fig. 7 Example of a pixel neighborhood. The red dot is a chosen pixel, and the red outline is the neighborhood boundary.

Now for each pixel in the neighborhood, we calculate a weight $w_i=e^{-k||c-c_i|{|}^2}$ with the $L2$ norm in the exponent. For our images we used a value of $k=0.05$. The idea here is that nearby pixels with similar colors should have similar depths, and so those should have a higher weight. Then finally we update the original pixel depth value by setting:

$d=\frac{\sum _i^N{w}_i{d}_i}{\sum _i^N{w}_i}$

This process is then repeated for four iterations, because at each iteration only flagged pixels neighboring non-flagged pixels will be updated. However there will be speckles, or small dark dots. These are not black (missing) values, but are simply inaccuracies which where not flagged and not fixed in filtering. To remedy this, we applied one more pass of the bilateral filter, but we made use of all of the pixels, and did not ignore any flagged pixels. This is a more typical bilateral filter which is a common edge-preserving filter [2].

Results

Hole filling

Below are examples of resultant images from our hole filling algorithm. Notice that Fig. 9 after the initial filter has speckles around image edges. These are areas where the Intel camera had accuracy issues. In Fig.10 these are removed through the second filter.

Fig. 8 Original depth image.

Fig. 9 Depth image after initial bilateral filter.

Fig. 10 Final Depth image after second bilateral filter.

Conclusions

We made use of color and depth images generated from both the Intel Realsense R200 camera and the Google Camera app. The Intel images require hole filling to acquire dense depth maps, and we did this through a bilateral filter. This worked very well for the most part, but has issues with larger holes. In particular the issues are with recreating the depth of a background which may have been too far for the camera. There are more complex image inpainting algorithms out there which could be used for next year. Additionally, the Intel camera outputs both of its IR images used for depth calculations.

With these dense depth maps we created three filters, a depth of field effect, a color mapping effect, and a background warping effect.

References

[1] Leon Gatys, Alexander Ecker, and Matthias Bethge, "Image Style Transfer Using Convolutional Neural Networks," in The IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2414-2423. http://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Gatys_Image_Style_Transfer_CVPR_2016_paper.pdf

[2] Carlo Tomasi and Roberto Manduchi, “Bilateral filtering for gray and color images,” in Computer Vision, 1998. Sixth International Conference on . IEEE, 1998, pp. 839–846. https://users.cs.duke.edu/~tomasi/papers/tomasi/tomasiIccv98.pdf

Appendix I

Here is the Matlab Code used to create our filters and GUI: Code.

Here are test images to apply our filters to: Images.

Here is code that controls the camera and saves images: Code.

Appendix II

Most of the work, such as creating the GUI interface and background reading, was done jointly; the parts that were not were:

• Rafael: Depth of Field filter, Background warping filter
• Austin: Color map filter, hole filling algorithm