Non-line of Sight imaging by SPAD​

Introduction

Silicon single photon avalanche detector (Si SPAD) has become a hot topic today for its single photon sensitivity, pico-second timing resolution and CMOS compatibility and low cost. Because of its unique properties, it becomes a tool to capture transient imaging for computer vision industry. One of applications enabled by transient imaging is non-line of sight problem, usually being referred to "look around the corner", where using Si SPAD, we could locate, track and recognize the shape of objects around the corner without directly seeing it.

There has been a few papers published on the experimenting result of this topic, and the object of our project is to simulate Si SPAD response in non-line of sight imaging using ray tracing and verify its algorithm to track position of "hidden" object.

Background

Difference between APD and SPAD

Traditional avalanche photo detector (APD) and SPAD both use avalanche effect to amplify signal. However, they are very different in circuit configuration and performance. APD is often reversely biased to have a gain around 10-100, while SPAD is biased above breakdown voltage with a gain above $10^{6}$ or even up to infinity. In timing response, APD usually have a time resolution of sub nanoseconds while SPAD have a time resolution of pico-seconds. The bandwidth of APD is limited by RC delay and timing resolution. On the other hand, SPAD is constrained by dead time, where it has to rest for a while before being sensitive to next arrival photon.

Work principle of SPAD

Because of dead time, the measurement method for using SPAD is time-correlated single photon counting (TCSPC). We would like to use bio-fluorescence as a working example. In these experiments as shown in Fig 1, the fluorescence response with regards to excitation laser is often recorded. The experiment begin by repeating excitation laser pulses with many cycles and sometimes fluorophore will emit photons. For each cycle, the emitted photon may be captured by SPAD and get registered or it may be missing. Si SPAD will also take note in photon arrival time. After repeating a certain times, a histogram of photon arrival time could be drawn and this distribution will be asymptotic to fluorescence emission response, denoted by red curve in Fig 1.

figure 1.SPAD working on Bio-fluorescence

Imaging model of SPAD

With SPAD work principle in mind, SPAD imaging model is built upon Poisson distribution as it comes from discrete nature of photons and SPAD triggering response (trigger or no trigger). In equations below, h is actual measurement result, $A,\mu ,\tau ,d$ are matrix in time domain. $d$ denotes dark count rate (DCR), which is the number of counts without any photon impinging on SPAD due to detector noise. $\mu \tau$ is expected response of how many photons are injected, absorbed and successfully trigger SPAD. $\mu \tau +d$ represents timing response of photon from the environment and this has to convolve with $A$, SPAD timing response itself. The final measurement results are Poisson distribution with expected tming response from $A\times (\mu \tau +d))$.

Imaging example of SPAD

One imaging example of the SPAD can be the object behind the blind. Normal imaging detector would have no way to capture this man behind the blind, and the information they obtain would be like the left hump in the figure below. However some photons pass through the blind and then gets reflected off the object. They form the second smaller hump. And the man can be imaged.

figure 2. photon stistics vs. time

figure 3. men behind the blind(gif)

Methods

Scene introduction

There have been multiple papers published on this topic. The one we studied is “Detection and tracking of moving objects hidden from view”. It was published last year on nature Photonics. The set-up of scene is described in Fig 4. SPAD and laser are spaced 15cm apart and they both are used to locate and track an object (10cm wide and 30cm high) behind the wall.

figure 4.scene; 4(a): top view; 4(b): side view; 4(c) top view with laser point source emitting spherical wave; 4(d) top view with hidden object emitting spherical wave

The working principle starts by laser focus onto the floor where photons are scattered to different directions for first reflection. Some of lucky photons hit on hidden object (as in Fig 4(d) for second reflection. Among second reflection, some photons scatter to the filed of view of SPAD camera and register its time of flight for third reflection. Photons captured in SPAD field of view also has a spacial distribution that is ellipse with focal points one at first reflection location and the other one at hidden object, as illustrated in Fig 6(b).

Reconstruction algorithm

figure 5.background noise area vs. valid object signal region

The reconstruction of object location is solely dependent on timing response captured by each SPAD. Fig 5(a) shows measured timing response from both background and from photons reflected from hidden object. By ignoring background signal (which could be done by doing calibration without hidden object behind the wall), we could select target signal associated with time $<t>_{i}$ (Fig 6(b)) and plot spatial distribution for SPAD image sensor pixels who has response at time $<t>_{i}$ in Fig 6(a). Among those pixels at point i, we could tell how far photons travel from laser to object and to point i. Then the trace of an ellipse could be drawn to show all possible positions of an object. We could then repeat this process by drawing another ellipse using pixels who has response in time $<t>_{i}$. The object position could be determined by the most overlapped regions of different ellipses.

figure 6. Spherical wave, ellipses, and estimated object location

Results

Monte Carlo simulation setup ( higher probability)

The first simulation we did is monte-carlo simulation on the scene the paper described, aiming to replicate the result of photon distribution. For better probability, we rotated the object rotation to be horizontal.

To begin with, we simplified the scene provided by the paper: let the wall have infinite height, object have no thickness, and detector area to be square. Besides, we assume all the surface is lambertian reflection, where the reflected ray has largest probability at normal. The layout and lambertian illustration is below:

Fig 7. Simplified layout with rotated object

Fig 8. lambertian reflection illustration

With the above two assumptions we get the simulation result, here we only displaying only the rays that passed the SPAD detector area:

Fig 9. Simulation result with object

Fig 10. Simulation result without object

Among the 43 photons that arrives the SPAD, 18 arrived at the object. So we get the signal ratio of 41.9%. And 4x105 photons were sent for the 43 photons, and so we get the collection efficiency to be 4.5x10-5.. This method also gives us an idea of how to distinguish the detected data. We simulated the system twice, once with the object and once without the object. And plot distance vs. count for both of them. Then subtract the upper curve with the lower curve to get the figure on the right. This tells us the distance of the object.

Fig 11. with object and without object

Fig 12. with object-without object

A full simulation with and without the object would be:

Fig 13. with object

Fig 14. without object

Monte Carlo simulation setup

A second setup is also experimented. We have the object rotated to be facing the first wall, which reduces the probability. Then make the SPAD detection area larger.

Fig 15. scene layout for the second setup

The result came up as: useful data percent: 7 out of 166 arrives at SPAD out of 1E5 total data, Collection efficiency: 7x10-5, Signal ratio: 4.2%

Fig 16. Simulation result with object

Fig 17. Simulation result without object

Attenuation of reflected signals versus distance

We also explored the distance and intensity/attenuation relationship to proof that we get the right collection efficiency. The left graph tells us the relationship between the light source and the object. The right graph tells us the relationship between the object and the SPAD detection area when fixing the distance between the light source and the object. It does validate that e105 is reasonable.

Fig 18. with object-without object

Fig 19. with object-without object

Spatial distribution of reflected photons validation

As we have studied in the paper, this is an very important step lead to object location. So we compared simulated results(left figure) with the expected result(right figure). This is distribution of radius 40cm (with thickness 10cm) from object position in Detection window. From figure 19 when we lay one picture on top of the other one it is quite obvious that they share the same trend, also the red dots all lays at the red region.

Fig 20. simulated results(

Fig 21. expected result

Fig 22. figure17 and figure 18 overlap

Also we tried to vary thickness and eventually get the accurate position of object.Here is the result:

Fig 23. vary thickness

Conclusions

This project uses Monte Carlo and ray tracing to simulate the simplest scenario in non-line of sight imaging problem. The code has been optimized to simualte $10^{9}$ photons in an hour and half. The whole simulation also reveals photon statistics with regards to object dimension, distance, surface reflection property. We also verifies its reconstruction algorithm. This paves way for reconstruction of much more complex scenes, such as complex material surface (occlusion, specular reflection, etc), tracking move object, larger distance and multiple objects. In addition, shape recognition by looking around the corner remains a hot topic at present but need very sophisticated reconstruction processing and much longer time. The research of shape recognition will benefit a lot from developing simulator based on this.

Reference

• 1 http://www.boselec.com/products/sigtcspcwhat.html
• 2 Gariepy, Genevieve, Francesco Tonolini, Robert Henderson, Jonathan Leach, and Daniele Faccio. "Detection and tracking of moving objects hidden from view." Nature Photonics (2015).

Appendix I: code

Attenuation of reflected signals versus distance: https://www.dropbox.com/sh/449p3lwl32ux2kl/AABGc_jeXg_3dQYF7CgH_ppma?dl=0

Monte Carlo simulation: https://www.dropbox.com/sh/n9d34x8jb5l0wmk/AACNTYTf_nJLVaKpkzYEW6SMa?dl=0

Monte Carlo simulation setup ( higher probability): https://www.dropbox.com/sh/t9cftqtuuvwxpk2/AAA4kyJSh4nkOQhQiUuMnj53a?dl=0

Spatial distribution of reflected photons validation: https://www.dropbox.com/sh/kfg5b95kurt4w44/AADlM6qn54A4Kuce5kA-wzVma?dl=0

Full files: https://www.dropbox.com/sh/ev91hw3gsehu0so/AAA_j5qToA1HrfOMirGRqfH7a?dl=0

Appendix II

• Kai Zang: development of simulation code, generate datasets
• Ke Xu: literature review, summary of simulation results
• shared: presentation slides, wiki reports