Non-line of Sight imaging by SPAD​

Introduction

Silicon single photon avalanche detector (Si SPAD) has become a hot topic today for its single photon sensitivity, pico-second timing resolution and CMOS compatibility and low cost. Because of its unique properties, it becomes a tool to capture transient imaging for computer vision industry. One of applications enabled by transient imaging is non-line of sight problem, usually being referred to "look around the corner", where using Si SPAD, we could locate, track and recognize the shape of objects around the corner without directly seeing it.

There has been a few papers published on the experimenting result of this topic, and the object of our project is to simulate Si SPAD response in non-line of sight imaging using ray tracing and verify its algorithm to track position of "hidden" object.

Background

Difference between APD and SPAD

Traditional avalanche photo detector (APD) and SPAD both use avalanche effect to amplify signal. However, they are very different in circuit configuration and performance. APD is often reversely biased to have a gain around 10-100, while SPAD is biased above breakdown voltage with a gain above $10^6$ or even up to infinity. In timing response, APD usually have a time resolution of sub nanoseconds while SPAD have a time resolution of pico-seconds. The bandwidth of APD is limited by RC delay and timing resolution. On the other hand, SPAD is constrained by dead time, where it has to rest for a while before being sensitive to next arrival photon.

Work principle of SPAD

Because of dead time, the measurement method for using SPAD is time-correlated single photon counting (TCSPC). We would like to use bio-fluorescence as a working example. In these experiments as shown in Fig 1, the fluorescence response with regards to excitation laser is often recorded. The experiment begin by repeating excitation laser pulses with many cycles and sometimes fluorophore will emit photons. For each cycle, the emitted photon may be captured by SPAD and get registered or it may be missing. Si SPAD will also take note in photon arrival time. After repeating a certain times, a histogram of photon arrival time could be drawn and this distribution will be asymptotic to fluorescence emission response, denoted by red curve in Fig 1.

figure 1.SPAD working on Bio-fluorescence

Imaging model of SPAD

With SPAD work principle in mind, SPAD imaging model is built upon Poisson distribution as it comes from discrete nature of photons and SPAD triggering response (trigger or no trigger). In equations below, h is actual measurement result, $A,\mu ,\tau ,d$ are matrix in time domain. $d$ denotes dark count rate (DCR), which is the number of counts without any photon impinging on SPAD due to detector noise. $\mu \tau$ is expected response of how many photons are injected, absorbed and successfully trigger SPAD. $\mu \tau +d$ represents timing response of photon from the environment and this has to convolve with $A$, SPAD timing response itself. The final measurement results are Poisson distribution with expected tming response from $A\times (\mu \tau +d))$.

Imaging example of SPAD

One imaging example of the SPAD can be the object behind the blind. Normal imaging detector would have no way to capture this man behind the blind, and the information they obtain would be like the left hump in the figure below. However some photons pass through the blind and then gets reflected off the object. They form the second smaller hump. And the man can be imaged.

figure 2. photon stistics vs. time

figure 3. men behind the blind

Paper review

We have studied a few papers for their experiment on the SPAD application on the 'hidden' object. The paper draw most of our interest is :“Detection and tracking of moving objects hidden from view”, and we also conducted the simulation analysis based on their experiment result.

The scene they are working on is as described in the graph below. First they have the SPAD and laser light source hang at the wall. Then the laser and SPAD would first hit the ground. And these two regions at the ground are our starting point and the end point.

figure 4.scene

From the data that they measured, they first decide the object valid signal region by estimating the object to SPAD detector distance.

figure 5.background noise area vs. valid object signal region

Methods

Scene introduction

There have been multiple papers published on this topic. The one we studied is “Detection and tracking of moving objects hidden from view”. It was published last year on nature Photonics. The set-up of scene is described in Fig 3. SPAD and laser are spaced 15cm apart and they both are used to locate and track an object (10cm wide and 30cm high) behind the wall.

The working principle starts by laser focus onto the floor where photons are scattered to different directions for first reflection. Some of lucky photons hit on hidden object (as in Fig 3(c) for second reflection. Among second reflection, some photons scatter to the filed of view of SPAD camera and register its time of flight for third reflection. Photons captured in SPAD field of view also has a spacial distribution that is ellipse with focal points one at first reflection location and the other one at hidden object, as illustrated in Fig 3(d).

Reconstruction algorithm

The reconstruction of object location is solely dependent on timing response captured by each SPAD. Fig 4(a) shows measured timing response from both background and from photons reflected from hidden object. By ignoring background signal (which could be done by doing calibration without hidden object behind the wall), we could select target signal associated with time $<t{>}_i$ (Fig 4(b)) and plot spatial distribution for SPAD image sensor pixels who has response at time $<t{>}_i$ in Fig 4(c). Among those pixels at point i, we could tell how far photons travel from laser to object and to point i. Then the trace of an ellipse could be drawn to show all possible positions of an object. We could then repeat this process by drawing another ellipse using pixels who has response in time $<t{>}_i$. The object position could be determined by the most overlapped regions of different ellipses.

Conclusions

This project uses Monte Carlo and ray tracing to simulate the simplest scenario in non-line of sight imaging problem. The code has been optimized to simualte $10^9$ photons in an hour and half. The whole simulation also reveals photon statistics with regards to object dimension, distance, surface reflection property. We also verifies its reconstruction algorithm. This paves way for reconstruction of much more complex scenes, such as complex material surface (occlusion, specular reflection, etc), tracking move object, larger distance and multiple objects. In addition, shape recognition by looking around the corner remains a hot topic at present but need very sophisticated reconstruction processing and much longer time. The research of shape recognition will benefit a lot from developing simulator based on this.

Reference

• 1 http://www.boselec.com/products/sigtcspcwhat.html
• 2 Gariepy, Genevieve, Francesco Tonolini, Robert Henderson, Jonathan Leach, and Daniele Faccio. "Detection and tracking of moving objects hidden from view." Nature Photonics (2015).

Appendix I

Code and datasets could be found here.