Attenuation-Based 3D Display Using Stacked LCDs

Introduction

Unlike traditional 2D displays, attenuation-based 3D displays enable the accurate, high-resolution depiction of motion parallax, occlusion, translucency, and specularity. We have implemented iterative tomographic reconstruction for image synthesis on a stack of spatial light modulators (multiple low-cost iPad LCDs). We illuminate these volumetric attenuators with a backlight to recreate a 4D target light field. Although five-layer decomposition generates the optimal tomographic reconstruction, our two-layer display costs less than $100 and requires less computation

Background

Engineers have promulgated designs for 3D displays, and even automultiscopic displays, as early as the turn of the 19th century. In particular, we consider four types of 3D display technologies that stand in contrast to what we have produced: parallax barriers, integral imaging, volumetric displays, and holograms. What relates these technologies is their shared ability to replicate disparity, motion parallax, and binocular depth cues without the need for special eyewear.

A performance summary of these comparative technologies based off the results of [Wetzstein et al. 2011] can be found below:

Multi-layer displays present a fifth class of displays. Differing from volumetric displays with light-emitting layers, overlaid attenuation patterns allow objects to appear beyond the display enclosure. We compute these patterns using tomographic techniques to create a 4D light-field illuminated by a uniform backlight (see below). The benefit of a multi-layered display is that it possess high resolution and contrast with only moderate trade-offs in brightness and complexity. Since our display relies only on two layers and the reconstructed light-fields are precomputed, we mitigate these limitations, although the produced image is merely static. Additionally, the stacked LCD configuration relies on multiplicative light absorption, rather than additive absorption; the benefit of this is that the display can construct occlusion, specularity, and depth without the need for any moving components.

Methods

Layered Attenuation-based Displays

We implemented the computed tomography techniques decsribed in “Layered 3D” by G. Wetzstein, D. Lanman, W. Heidrich, R. Raskar (SIGGRAPH 2011) to produce two 2048x1536x7x7 reconstructed images from many precomputed views of the light field, spanning a 20-degree FOV. We solve these layer decompositions ahead of time, and paint them as static images to the LCDs. The light field images were collected from the Stanford Light Field Archive and The Stanford 3D Scanning Repository. For example, the dice scene was created in POV-Ray and then released under the Creative Commons Attribution-Share Alike 3.0 Unported license.

Tomographic Approximation

We rely on the code produced by [Wetzstein et. al 2011] to synthesize an attenuation map to approximate the chosen target light field, relying on iterative tomographic reconstruction principles to find the optimal solution in the least squares sense, and apply this code to the specific two-layer case.

The 2D volumetric attenuator is defined as a continuously varying attenuation map $\mu (x,y)$. In this model, $\mu (x,y)$ is computed to obey Beer-Lambert's law so that $I=I_{0}*e^{-\int _{c}(\mu (r)dr)}$

By the Weber-Fechner law, however, the human visual system recognizes logarithmic changes in illumination as nearly linear, so the illumination is re-computed as $I^{-}=ln(I/I_{0})$.

In the forward model, the Radon transform $p(u,a)$ can take the attenuation map $\mu$ and the width and height of the layered slabs to encode all possible line integrals through the attenuation map along each ray $(u,a)$. Here, the orientation of ray $(u,a)$ is defined by the slope $a=s-u=d_{r}*tan(\theta )$ where $d_{r}$ is the distance of the s-axis from the u-axis. The oblique light field can then be described as Failed to parse (syntax error): {\displaystyle l^-(u, a) = −p(u, a)} for linear angle $a$.

With parallel beam tomography, an estimate of the attenuation map, $\mu _{approx}*(x,y)$ is recovered from the projections $p(u,a)$ using the inverse Radon transform. In turn, the filtered backprojection algorithm estimates a volumetric attenuator capable of emitting the target light field. The projection matrix $P_{ij}^{k}$ corresponds to line integrals through every basis function k along each ray $(i,j)$ can be expressed as a linear combination of $N_{b}$ non-negative basis functions with coefficients $\alpha$. This system, which models the attenuation as a linear system of equations when considering a discrete light field, is expressed in matrix-vector form as $P*\alpha =-l^{-}+e^{-}$, where $e^{-}$ is the approximation error. As a result, we cast attenuation map synthesis as the following non-negative linear least-squares problem:

arg min $\alpha$ of $||l^{-}+P*\alpha ||^{2}$ for $\alpha >=0$

For multi-layered attenuators, the form of the projection matrix P is modified, now encoding the intersection of every ray with each mask. Thus, a similar optimization solves the inverse problem of constructing an optimal multi-layered attenuator. Practically, however, layers have a finite contrast and the approximation is solved as a constrained least-squares problem.

Assembly

We first procured two 2048x1536, 9.7”, IPS 60Hz, iPad 3 LCD displays. This display model was chosen for its abundance online, which has kept the price low and created a wealth of readily available related-resources such as display drivers. We then carefully disassembled the front LCD, removing its backlight.

The enclosure was then designed such that the stack of LCDs would be well-fastened and close together in approximate alignment. The enclosure also leaves space on the back panel to protect the driver circuitry from obstructing the view. To fabricate the enclosure, we purchased two sheets of Duron, laser cut the sheets to our design specification, and constructed the case.

In this initial prototype, we screwed the back LCD into place and then attached the front LCD with adhesive so that we could manually adjust the display for approximate pixel alignment. Since the two LCDs were linearly polarized, we inserted a quarter-wave polarizing sheet between the LCDs to circularly polarize the light and greatly increase the illumination of the front panel.

Results

Limitations

• Brightness

The most apparent usability issue in our current prototype is the low brightness of the displayed scene. This is a problem inherent to the stacked LCD approach, as each LCD only transmits a fraction of the incident backlighting. Roughly, we measure our 2-stack prototype to have a total light transmissivity of about 10%, as measured with a fixed light source and cheap luxmeter. There exist some LCD panels that achieve higher fill factors and correspondingly higher transmissivity, but for the future of our prototype, we think it simplest to pay off this heavy transmission loss by engineering a high intensity backlight.

• Low Refresh Rate

The original stacked LCD design from Wetzstein et. al called for high refresh rate (120+ Hz) LCDs, such that successive frames could multiplex many light rays in time. We want to first characterize exactly how much bearing this multiplexing has on the perceptual quality of the scene, and if not absolutely critical, explore partial solutions while avoiding perceptual flicker at just 60 Hz.

• Narrow Field of View

Our current field of view limitation comes from the LCD's limited refresh rate, as well as the increased computational cost of rendering more light fields in real time. However, we can reduce the effective field of view requirements by adding a head tracking camera to the system, optimizing for single- or few-viewer cases by rendering narrower fields targeted to the known observers. Put another way, knowing the viewer's position might let us cull many unnecessary light field elements without significantly compromising perceived scene quality.

• Pixel Alignment

Our future prototypes will be designed around a more rigid frame, with better facilities for fine positional adjustments. We think the current prototype suffers from significant misalignment of the two LCDs, resulting in what appears to be a "smeared, blurry" scene.

• Color Crosstalk

Adding more layers to our stack will afford us finer control over the aggregate tomographic mask, helping reduce the crosstalk we currently observe from our lack of an ideally directional backlight.

Performance

• High Contrast Halo Artifacts

When working with few LCD layers, we have limited beam steering control and correspondingly, have difficulty representing high contrast edges edges in the scene. With more layers and higher refresh rates, we can render more complex masks such that these halo-like artifacts are perceptually averaged out.

• Depth of Field

For a two-layer attenuation display, the DoF has an upper bound described by $|f_{\epsilon }|<=[h/(h/2+|d_{o}|)]*f_{0}$ where $|f_{\epsilon }$ is the spatial cutoff frequency and $h$ is the fixed display thickness [Wetzstein et al. 2011].

• Resolution

Thanks to major advances in LCD technology, we achieve much higher layer/mask resolution of 2048x1536 without significant increases in cost. Stacked LCDs are thus attractive as an economical way to achieve high-fidelity control over light rays.

Conclusions

Our goals for this project were modest - we wanted to recreate G. Wetzstein et. al's work from 2012 using low-cost, commodity components, as a rough proof-of-concept for what could eventually become a compelling implementation of 3D display. As in the original paper, we find that the theory is generally sound, but significant challenges remain in engineering a performant system. However, we believe that the resources available to even amateurs in 2017 place us in a significantly better position to iterate and refine our prototype. With time, we see a promising roadmap to a well-engineered, perceptually pleasing product, and the authors intend to develop the work shown here much further.

Improvements

• Strong Backlighting

Our present and near future prototypes target the desktop / fixed display form factor, where we are less constrained by power and form factor (vs. mobile devices). We are developing a much more powerful (10-100x lumen output) directional backlighting system, which we hope will significantly improve usability and enable other improvements that usually reduce light efficiency.

• Dynamic Rendering

Currently, our model is designed for static scenes only, mostly constrained by a lack of compute to render dynamic scenes in real time. Very cursory profiling suggests that significant speedups to the rendering pipeline should be possible, so we intend to write a more optimized, GPU-accelerated renderer targeting 30-60fps scene display.

• Three (or N) Layer Display

Increasing layer count generally affords us better control over light ray steering, reducing artifacts in the light field. Of course, as mentioned above, increasing LCD count also reduces light transmission, which we will initially address with a more advanced, high power lighting scheme. Additionally, a greater number of layers increases the PSNR and improves the MTF of the display, as shown in [Wetzstein et al. 2011]:

• Face-Tracking

Having a good estimate of the observer's position allows aggressive rendering optimization, reducing computational cost and potentially improving solutions to the light field. We initially plan to use a standard web camera and OpenCV software running on the render host, but intend to move to a tightly embedded "RealSense" stereo camera that additionally provides depth annotations, letting us further optimize our target light field for the viewer.

Appendix I

[1] G. Wetzstein, D. Lanman, W. Heidrich, R. Raskar. Layered 3D: Tomographic Image Synthesis for Attenuation-based Light Field and High Dynamic Range Displays. Proc. of SIGGRAPH 2011 (ACM Transactions on Graphics 30, 4), 2011.
[2] G. Wetzstein, D. Lanman, D. Gutierrez, M. Hirsch. Computational Displays. ACM SIGGRAPH 2012 Course, 2012.

Matlab Implementation of Tomographic Light Field Synthesis
Real-Time Implementation of Tomographic Light Field Synthesis

Appendix II

The display was built largely in tandem, as we laser cut and constructed the enclosure as well as rendered the light fields to the display as a pair. In building the display, Neil disassembled the LCDs while Jason worked on the enclosure design.