Simultaneous Color Holographic Display

Introduction

A holographic display is a type of display system that produces 3D or 2D images by manipulating the wavefront of light. With the help of a spatial light modulator (SLM), a holographic display can manipulate the phase of a coherent wavefront at the pixel level. This allows it to reshape the wavefront precisely as it would originally emanate from a real object, creating an image with genuine depth cues.

Holographic displays typically use a laser as a light source, resulting in monochromatic holograms. To achieve color holograms, color holographic displays sequentially switch between RGB lasers at a high rate, leveraging the persistence of vision property of the human eye. This allows the human eye to fuse sequential monochromatic holograms into a perceived color hologram. However, this color scheme sacrifices the refresh rate of the SLM, as displaying one frame of a color target requires displaying three phase patterns, one for each RGB channel.

One potential solution to fully utilize the SLM’s refresh rate is to simultaneously activate the three primary laser lights and have the SLM modulate these three wavefronts with different wavelengths at the same time, using the same phase pattern. This approach could potentially allow for full utilization of the SLM’s refresh rate.

This project aims to investigate the effectiveness of traditional phase retrieval pipelines in this setup and explore potential improvements in reconstruction quality through the use of different loss functions.

Background

How to get light phase from intensity

Unlike conventional displays that directly control light intensity, holographic displays use a Spatial Light Modulator (SLM) to modulate the phase of light on a per-pixel basis. The modulated wavefront then propagates through free space from a starting plane $0$ to the image plane $z$. Our goal is to determine the phase pattern on the SLM such that, at the image plane $z$, the resulting intensity distribution matches a desired target intensity pattern.

Angular Spectrum Method

The Angular Spectrum Method is a computational technique used to model how wavefronts propagate through free space. It can be expressed in the following form:

$u(x,y,z,\lambda )={ℱ}^{-1}\{ℱ\{u(x,y,0,\lambda )\}\cdot \mathscr{H}(k_x,k_y,z,\lambda )\}$

$\mathscr{H}(k_x,k_y,z,\lambda )=\{\begin{array}{cc}{e}^{-\frac{i2\pi }{\lambda }\sqrt{1-(\lambda k_x{)}^2-(\lambda k_y{)}^2}z}& \text{if }\sqrt{k_x^2+k_y^2}<\frac{1}{\lambda },\\ 0& \text{otherwise}.\end{array}$

$u(x,y,0,\lambda )$ is the wavefield at the plane $z=0$. By applying the 2D Fourier transform to it, $ℱ\{u(x,y,0,\lambda )\}$, we decompose the wavefield into a superposition of plane waves traveling in various directions. This continuous distribution of plane waves is known as the angular spectrum. The spatial frequencies $k_x$ and $k_y$ determine the propagation direction of each plane wave component.

As each plane wave propagates through free space, it accumulates a distance-dependent phase shift. This phase shift is described by the transfer function $\mathscr{H}(k_x,k_y,z,\lambda )$. In the Fourier domain, by multiplying the angular spectrum $ℱ\{u(x,y,0,\lambda )\}$ by $\mathscr{H}(k_x,k_y,z,\lambda )$, we effectively propagate all plane wave components over the distance $z$.

Finally, to reconstruct the propagated wavefield at $z$, we apply the inverse Fourier transform: ${ℱ}^{-1}\{ℱ\{u(x,y,0,\lambda )\}\cdot \mathscr{H}(k_x,k_y,z,\lambda )\}$.

Image Formation Model

In our setup of the holographic display, the coherent light source that illuminates the SLM has a source field $u_{src}(x,y,\lambda )$.

The phase phase-only SLM can apply a spatially-varying delay $\varphi (x,y,\lambda )$ on the phase of the field $u_{src}(x,y,\lambda )$, so the wavefield at the SLM becomes to:

$u_{SLM}(x,y,\lambda )=e^{iq(\varphi (x,y,\lambda ))}{u}_{src}(x,y,\lambda )$

The SLM is at plane $z=0$, we can use the Angular Spectrum Method to model what the wavefield will looks like at image plane $z$.

$u_z(x,y,\lambda )=ASM(u_{SLM}(x,y,\lambda ),z)$

At the image plane, what people see is the intensity of light not the wavefield. We can get light intensity by squaring the wavefield.

$I_z(x,y,\lambda )=|u_z(x,y,\lambda ){|}^2$

In combination, the final light intensity distribution is $I_z(x,y,\lambda )=|ASM(e^{iq(\varphi (x,y,\lambda ))}{u}_{src}(x,y,\lambda ),z){|}^2$

For notational convenience, we can write the intensity pattern at the image plane $z$ as:

${𝑰}_{𝒛}\mathit{(}𝒙,𝒚,𝝀\mathit{)}\mathit{=}\mathit{|}𝑨𝑺𝑴\mathit{(}{𝒆}^{𝒊𝒒\mathit{(}𝝓\mathit{)}},𝒛\mathit{)}{\mathit{|}}^2$

This is how we can get the light intensity at the image plane $z$, by showing a phase pattern $\varphi$ at the SLM.

Iterative Method To Get Light Phase from Intensity

Now we know how to calculate light intensity from the phase pattern. However in order to show some images on the holographic display, we need to find a way to calculate phase pattern from the light intensity. People usually use gradient descent to solve this kind of inverse problem.

At iteration 0 we can generate some phase random pattern ${\varphi }^0$, using the image formation model we derive in the previous section, we can get the intensity pattern at the image plane, $|ASM(e^{iq(\varphi )},z){|}^2$ and the light wave amplitude is the square root of the intensity: $|ASM(e^{iq(\varphi )},z)|$

If the target light intensity is $I_{target}$, the target light amplitude is square root of the intensity, $a_{target}=\sqrt{I_{target}}$, we can compare $|ASM(e^{iq(\varphi )},z)|$ and $a_{target}$ with a loss function, $ℒ(|ASM(e^{iq(\varphi )},z)|,a_{target})$.

We can calculate the gradient of $\varphi$ from the loss function, $(\frac{\partial ℒ}{\partial \varphi }{)}^Tℒ(|ASM(e^{iq(\varphi )},z)|,a_{target})$ and iteratively update the value of $\varphi$, ${\varphi }^{(k)}\leftarrow {\varphi }^{(k-1)}-\alpha (\frac{\partial ℒ}{\partial \varphi }{)}^Tℒ(|ASM(e^{iq({\varphi }^{(k-1)})},z)|,a_{target})$

After enough iterations, the value of $\varphi$ should converge. In this way, we can get a phase pattern $\varphi$ that will generate the target intensity $I_{target}$ at the image plane $z$.

Simultaneous Color Scheme Related Questions

Does the laser support to turn all three color on simultaneously?

The laser we used in hardware setup is FISBA RGBeam. It has 3 diodes, and each diode emits red, green and blue light. It is possible to turn all 3 diodes on. All 3 lasers with different wavelength will pass through the same optical fiber. After passing through the collimating optics, the SLM will see a plane wave in white color.

How we can use only one phase pattern to generate three different intensity pattern?

We model how light waves propagate in free space, using the angular spectrum method.

$u(x,y,z,\lambda )={ℱ}^{-1}\{ℱ\{u(x,y,0,\lambda )\}\cdot \mathscr{H}(k_x,k_y,z,\lambda )\}$

$\mathscr{H}(k_x,k_y,z,\lambda )=\{\begin{array}{cc}{e}^{-\frac{i2\pi }{\lambda }\sqrt{1-(\lambda k_x{)}^2-(\lambda k_y{)}^2}z}& \text{if }\sqrt{k_x^2+k_y^2}<\frac{1}{\lambda },\\ 0& \text{otherwise}.\end{array}$

The transfer function in ASM is not only distance dependent but also wavelength dependent. As plane wave propagates through free space, the phase accumulation also differs depending on the wavelength of the plane wave. As a result, even the plane waves with different wavelength being applied the same phase shift at same time at the SLM. The final intensity distribution of these 3 wavelength of light are still different. This phenomenon give us some degree of freedom to use one phase pattern to match 3 different target intensity.

Methods

In the field sequential color scheme, we can generate three phase pattern to match three light intensity pattern of the RGB target. In the simultaneous color scheme we are trying to match three light intensity pattern using just one phase pattern. Even the Angular Spectrum Method indicates we can get three light intensity pattern from one phase pattern, we might not have enough degree of freedom to perfectly to match the three target intensity. There might be always some errors between the reconstruction intensity and the target intensity. A perceptual driven loss function might be useful in this case, other than just try to match wavefield amplitude, we can prioritize the matching of some visual elements that is more important to human perception.

The code base of this project is: Time-multiplexed Neural Holography: A Flexible Framework for Holographic Near-eye Displays with Fast Heavily-quantized Spatial Light Modulators. It provides a good framework to solve for phase pattern from the target intensity, and is flexible to change the loss function in the gradient descent method.

L2 Loss

The mean square loss, also called the L2 loss is the default loss of the code base. It prioritize minimizing the error between reconstruction amplitude and target amplitude.

$MSE=\frac{1}{n}\sum _{i=1}^n(|ASM(e^{iq(\varphi )},z)|-a_{target}{)}^2$

Using L2 loss, after 5000 iterations of gradient descent, the phase pattern, corresponding reconstruction intensity and target intensity are shown below.

CIELAB loss

From the L2 loss reconstruction, we see noticeable color shift between Reconstruction and Target images. These results indicated that using only one phase pattern, we don't have enough degree of freedom to match the target intensity.

From the class, we learnt that CIELAB is a color space that is perceptually uniform, Maybe we can convert the target intensities and reconstruction intensities in to the CIELAB color space and calculate the L2 loss in the CIELAB color space, so we can prioritize the color matching.

$CIELABLoss=MSE(RGB2LAB(|ASM(e^{iq(\varphi )},z){|}^2),RGB2LAB(I_{target}))$

In order to construct RGB2LAB function, we need to get RGB2XYZ matrix for our holographic display first. We assume the wavelength of the three primary laser are 636nm, 518nm and 441nm and power they can achieve are both 0.0035 Watts/sr/nm/m^2. The following plot is the spectral power distributions of our holographic display setup:

Using the ieXYZFromEnergy function from the istcam, we can get the RGB2XYZ for our holographic display setup.

RGB2XYZ = ieXYZFromEnergy(primaries', wavelength(:))'

$RGB2XYZ=\left[\begin{array}{ccc}1.2493& 0.1147& 0.8357\\ 0.4976& 1.6037& 0.0581\\ 0.4976& 1.6037& 0.0581\end{array}\right]$

Once we convert RGB color to XYZ color, the XYZ2LAB are constructed by implementing the following equations we learnt on the class:

$L^{*}=\{\begin{array}{cc}116{\left(\frac{Y}{Y_w}\right)}^{1/3}-16,& \text{if }\frac{Y}{Y_w}>0.00856\\ 903.3\left(\frac{Y}{Y_w}\right),& \text{otherwise}\end{array}$

$a^{*}=500\{{\left(\frac{X}{X_w}\right)}^{1/3}-{\left(\frac{Y}{Y_w}\right)}^{1/3}\}$

$b^{*}=200\{{\left(\frac{Y}{Y_w}\right)}^{1/3}-{\left(\frac{Z}{Z_w}\right)}^{1/3}\}$

The white point we chose is D65, the approximate XYZ values are:

$X_w=95.047,Y_w=100.000,Z_w=108.883$

After switching loss function to CIELAB L2 loss, after 5000 iteration of optimization, the phase pattern, corresponding reconstruction intensity and target intensity are shown below:

Spacial CIELAB loss

From the CIELAB loss reconstruction result we can see, the color matching between reconstruction and target images are better, but there are still some color shift and overall the reconstruction image are even noisier. Maybe we just don't have enough degree of freedom to match color perfectly either.

At this point, I feel this problem is similar to a image compression problem. How to achieve similar visual quality under limited bandwidth.

From the class, we learnt that for high spatial frequency patterns, the human eyes is more sensitive to luminance change than color change. and we talked about Spatial CIELAB metric on the class. Spatial CIELAB applies spatial filters based on the human visual system’s sensitivity to spatial frequencies. Maybe it is good idea to use S-CIELAB as the loss function. By ignoring the high spatial frequency color differences, we can allocate more bandwidth to match what is more important to human vision.

In order to implement the S-CIELAB loss function, We need to first convert RGB color space into the Opponent Color Space. We calculate the RGB2XYZ matrix in the previous section.

$RGB2XYZ=\left[\begin{array}{ccc}1.2493& 0.1147& 0.8357\\ 0.4976& 1.6037& 0.0581\\ 0.4976& 1.6037& 0.0581\end{array}\right]$

I copied the XYZ2OPP matrix from the MATLAB implementation of SCIELAB-1996.

$XYZ2OPP=\left[\begin{array}{ccc}0.2787336& 0.7218031& -0.106552\\ -0.4487736& 0.2898056& 0.0771569\\ 0.0859513& -0.5899859& 0.5011089\end{array}\right]$

By sequentially multiply RGB2XYZ and XYZ2OPP matrix, we can convert reconstruction and target image into the opponent color space.

The opponent color space has three channels: $O1$, $O2$, $O3$. $O1$ represents the luminance $O2$ represents the contrast between red and green, and $O3$ represents the contrast between blue and yellow. Since human has different spatial frequencies sensitivity to these three channels. We will apply low pass filter with different cutoff ratio to each channel.

In order to determine the low pass filter cutoff ratio for each channels, we calculate the effective resolution using the following display model.

The resolution of the SLM is 1280x720. The pixel pitch is 10.8um*10.8um. The physical dimension of the SLM is 1.382cm x 0.78cm. The eyepiece in our hardware setup has a focal length of 50mm. We set d' in the graph as 45mm, so the magnification ratio of the eyepiece is $\frac{50\text{mm}}{50\text{mm}-45\text{mm}}=10$. Then the virtual screen dimension is 13.82cm x 7.8cm. Display diagonal size is around 6.24 inches. d in the above graph is $\frac{1}{(\frac{1}{45\text{mm}}-\frac{1}{50\text{mm}})}=450\text{mm}$ The total viewing distance is $450mm+50mm=50cm$.

For a 6.24 inches screen; resolution is 1280x720; viewing distance is 50cm. Pixels per degree is around 80ppd. The effective resolution is around 40 cpd.

I set the lowpass filter cutoff ratio for each opponent color channel to be 0.8, 0.45, 0.3, so that the highest remaining spatial frequency roughly match the Space-Time-Color graph we learnt on the class.

Effective resolution = 40 cpd

Luminance: 0.8 cutoffs ~ 32 cpd

Red-green: 0.45 cutoffs ~ 18 cpd

Blue-yellow: 0.3 cutoffs ~ 12 cpd

After we applied lowpass filter to each channels of the opponent color space. We can use the OPP2XYZ matrix to convert opponent color space back to the XYZ space. The following steps are the same as the CIELAB loss: calculating the L2 loss in CIELAB color space.

After switching loss function to the S-CIELAB loss, after 5000 iteration of optimization, the phase pattern, corresponding reconstruction intensity and target intensity are shown below:

ColorVideoVDP + S-CIELAB Loss

The reconstruction results using the S-CIELAB loss is better than previous 2 losses. The ColorVideoVDP is a new quality metric designed to evaluate the perceptual quality of color images and videos. It models all spatial vision, temporal vision, color vision, and accounts for display geometry and photometry. The CSF used in S-CIELAB might be overly simplistic. The ColorVideoVDP uses a novel contrast sensitivity model (castleCSF) which accounts for the changes in contrast sensitivity with luminance and accounts for supra-threshold vision (e.g. contrast masking and contrast constancy). The metric can also be used as used as a loss function. By combining ColorVideoVDP Loss with S-CIELAB loss I can achieve even better reconstruction results. The reconstruction results are shown below:

Results

The reconstruction images using four different loss function are evaluated using 2 metric: PSNR and ColorVideoVDP(CVVDP)

PSNR is defined as $\text{PSNR}=10\cdot {\log }_{10}\left(\frac{{\text{MAX}}^2}{\text{MSE}}\right)$, It quantifies how much distortion or noise is present in the reconstruction image by comparing the pixel intensity differences between it and the target image.

ColorVideoVDP reports image/video quality in the JOD (Just-Objectionable-Difference) units. The highest quality (no difference) is reported as 10 and lower values are reported for distorted content. One JOD score difference means that 75% of the people will think the image with higher JOD is better.

Test image 1:

Test image 2:

Test image 3:

Conclusions

In this project, we explored the possibility of just using one phase pattern to generate color hologram. In the traditional field sequential color scheme, we can use three phase pattern to generate three intensity pattern to match each color channel, so we have enough degree of freedom to perfectly match the color target in the simulation. However, since in the simultaneous color scheme, we can only manipulate one phase pattern, there is a chance we cannot fully match the RGB target. There are always some error between the reconstruction images and target images. This project shows by using perceptual driven losses like, S-CIELAB and ColorVideoVDP. we can shift these inevitable error to places that is hard for human to perceive, so the perceptual quality of the reconstructed image is still good.

It is interesting to see for the 3 test images, the reconstructed image using L2 loss always have the highest PSNR, because the L2 loss prioritize minimizing the error of pixel intensity between target and reconstructed image. However, these image have relatively low ColorVideoVDP score and doesn't visually looks good. This is an interesting example to show PSNR is not aligned with human visual perception.

This project explores spatial aspect of human vision, but the purpose to choose simultaneous color scheme over the sequential color scheme is we can fully utilize the refresh of the SLM. The simultaneous color scheme allows a 60hz SLM to show 60hz color video contents, so it is worthwhile to explore the temporal aspect of human vision. Human vision is more sensitive to luminance changes than to color changes in the temporal domain too. The next step is to design a new loss function that accounts for this phenomenon too, so that we can achieve better video quality on the simultaneous color holographic display.