Simulate an Underwater Imaging System and Explore Water Absorption and Scattering Estimation Methods

Introduction

Motivation

There is a growing demand for high-speed wireless underwater data transmission. Optical communication provides one method for achieving this, given its inherently large bandwidth, relatively low attenuation, and ability to form secure point-to-point connections. However, underwater wireless optical communication (UWOC) is impacted significantly by distortions due to optical turbulence, caused by small changes in the refractive index of the water due to different medium conditions.

What has been done in the past

In the past, researchers have superimposed Laguerre-Gaussian (LG) beams to create an alphabet of spatially multiplexed symbols that can be encoded with information and decoded at the receiver by a Convolutional Neural Network (CNN). LG beams can be superimposed due to their orthogonality due to the fact they carry orbital angular momentum (OAM), and they have been shown to have inherent resilience to optical turbulence. Through signal processing and machine learning methods, researchers can reduce a certain amount of impact of oceanic optical turbulence on the received signal. However, the challenges in the oceanic environment motivate investigation into understanding the water medium better to further improve optical communication.

Goal

The goal of this project is to develop experimental designs aimed at characterizing a participating medium using camera captures and estimate the properties using convex optimization algorithm. The absorption and scattering properties are fundamental characteristics of a participating medium such as water. Since measuring these two properties of real water or other media is difficult, In this project, we use camera simulations and virtual 3D environment techniques to test these parameters in order to create accurate models that faithfully model light interactions with such a medium, understand how they impact a medium, and hence simulate the appearance of an image captured by an underwater camera. Using the captured image as the ground truth, we then model the image formation process mathematically and estimate the properties of the medium by solving a convex optimization problem.

Background

Absorption

The absorption of light is the process by which water (or other medium) molecules absorb certain wavelengths of electromagnetic radiation, particularly light. This process takes place due to certain energy levels of light interacting with specific molecular structures of water molecules. The absorption of light by water is wavelength-dependent, and different light wavelengths can be absorbed differently. The absorption property of the medium is crucial to understanding underwater imaging and optics because it directly affects how light behaves through a media and the appearance of images captured by cameras.

Fig. 1: Visible light absorption spectrum of pure water (absorption coefficient vs. wavelength).

Scattering

The scattering of light is the process where light interacts with particles, molecules, or impurities in water (or other media), which causes it to deviate from its original path. When lights are scattered, the phenomenon affects the optical properties of water, light transmission underwater, and the appearance of images captured by cameras.

Fig. 2: Visible light scattering example in water.

Methods

Experimental Setup

The experimental setup in our simulation of underwater images is shown below in Fig. 3. We have a camera that is facing the direction of the image. There are some mediums between them, such as water, fog, or plankton. When lights transfer through the medium and are captured by the camera, the light absorption and scattering of the medium take place, which causes the image captured by the camera to be distorted. In the figure, green lights (green arrows) indicate the water scattering scenario, and the black light (black arrow) indicates the water absorption scenario.

Fig. 3: The simulation experimental setup between camera, medium, and image.

MATLAB Simulation Tools

In this project, we have used three MATLAB simulation tools:

• iset3d toolbox We used iset3d toolbox in MATLAB to create simple 3D scenes.

• PBRT We Ran PBRT in MATLAB to produce physically accurate images by physically based and ray tracing through scenes.

• isetcam toolbox We explored isetcam to provide accurate camera simulation and modeling.

We ran these simulations through remote rendering on Stanford mux through Dr. David's computer.

Fig. 4: The output scene from MATLAB toolboxes. Example 1.

Fig. 5: The output scene from MATLAB toolboxes. Example 2.

MATLAB Simulation Script

Our MATLAB Simulation script t_mediumExample221.m is attached in Appendix I. This script does the following:

• First: Setup scene with Macbeth chart and process the scene
  • Create an optical image
  • Create a camera sensor
    • Set the sensor integration time to be a constant to ensure we get a constant g value
  • Sample RGB data from the sensor for each Macbeth patch
  • Build a linear image formation model
    • Obtain constant g value

• Second: Setup medium (Water & Fog)
  • Absorption
  • Scattering

• Third: Submerge scene in our created medium and process the submerged scene
  • Repeat sub-steps in the First step

• Fourth: Output Result

Image Formation Model

After simulating, we can use the captured image to get the properties of the medium. First, we need to generate an image formation model. In general, the image formation in air and across short distances can be described with a simple linear model that integrate over wavelength ${\textstyle \lambda }$:

$$m=g\int _{\lambda }I(\lambda )r(\lambda )c(\lambda )d\lambda$$

where ${\textstyle m}$ refers to the pixel value of the estimated image, ${\textstyle I_{0}}$ is illuminant, ${\textstyle r}$ is the scene reflectance, ${\textstyle c}$ is the spectral response of camera, ${\textstyle g}$ is the gain related to camera settings. However, we need to modify the equation as our model would be affected by the medium.

The light rays received by the camera consists two type of rays. The direct rays is the one originated from the light source, go through the media, then after getting reflected from the target image travel through the medium back and reaches the camera. The scattered rays get reflected towards the camera directly by the medium and do not interact with the target image.

For direct rays, light decays exponentially in the medium according to Beer-Lambert’s law. After the rays traveling distance ${\textstyle d}$ in the medium with spectral absorption coefficient ${\textstyle \alpha }$, the light intensity is:

$$I_{d,i}=I_{0,i}e^{-d\alpha _{i}}$$

where ${\textstyle I_{0,i}}$ is the original light intensity, subscript i refers to the ith waveband. From the equation, we extract the transition form T when light travel through the medium:

$$T=e^{-d\alpha }$$

Therefore, the image formation model for direct rays is:

$$m_{direct}=g\int _{\lambda }I_{0}(\lambda )T(\lambda )r(\lambda )T(\lambda )c(\lambda )d\lambda$$

For scattered rays, it's very complex because it involves complicate interaction with the medium. Therefore, we simplify it, combining all the effect and representing it as a single variable ${\textstyle P_{scatter}}$. Then the model for scattered rays is:

$$m_{scatter}=g\int _{\lambda }I_{0}(\lambda )P_{scatter}(\lambda )c(\lambda )d\lambda$$

We add the effect of two types of rays to generate the whole image formation model:

$$m=g(\int _{\lambda }I_{0}(\lambda )T(\lambda )r(\lambda )T(\lambda )c(\lambda )d\lambda +\int _{\lambda }I_{0}(\lambda )P_{scatter}(\lambda )c(\lambda )d\lambda )$$

For computational convenience, we discretize the continuous integration into N wavebands with wavelength discretization step ${\textstyle \Delta \lambda }$:

$$m\approx g\Delta \lambda (\sum _{i=1}^{N}I_{0,i}e^{-2d\alpha _{i}}r_{i}c_{i}+\sum _{i=1}^{N}I_{0,i}p_{i}c_{i})$$

and use the vectorization version in coding:

$$m=g\Delta \lambda \cdot r^{T}\cdot Diag(I_{0})\cdot Diag(e^{-2d\alpha })\cdot c+I_{0}^{T}\cdot Diag(p)\cdot c$$

If the value of ${\textstyle I_{0},c,r,g}$ is given, this equation becomes an affine function of ${\textstyle P_{scatter}}$ and a convex, non-increasing function of ${\textstyle \alpha }$. This character can help us solve the problem.

Algorithm

Our task is to estimate the properties of the medium, absorption ${\textstyle \alpha }$ and scatter ${\textstyle P_{scatter}}$, given known parameter ${\textstyle I_{0},c,r,g}$ and using the pixel values of the image captured by the camera in the simulation process as the ground truth. Firstly, we use the model above to estimate the pixel value with variables ${\textstyle \alpha }$ and ${\textstyle P_{scatter}}$. The estimated result should be as close to the captured value as possible. Therefore, we can extract the variable by solving the optimization problem:

$$Minimize_{\alpha ,p}|{\hat {m}}-m|^{\gamma }$$

$${\text{subject to: }}\alpha \geq 0$$

$$P_{scatter}\geq 0$$

where ${\textstyle m}$ refers to the pixel values estimated by the model, ${\textstyle {\hat {m}}}$ refers to the pixel values captured by the camera in simulation. The object is the norm of the difference, and the constraints are non-negativity of the spectral absorption coefficients and scatter.

Because the function m is a smooth, convex, non-increasing function of ${\textstyle \alpha }$ and ${\textstyle P_{scatter}}$ as mentioned before, the objective is a quasiconvex function when ${\textstyle \gamma \geq 1}$ ( we set ${\textstyle \gamma =1}$ in the experiment). Therefore, this problem is a convex optimization problem, which means it can be solved utilizing the package CVXPY.

The variable ${\textstyle \alpha }$ and ${\textstyle P_{scatter}}$ can be found using iterative method, and the process is shown below:

1. Initialize ${\textstyle \alpha _{0}}$;

2. Represent ${\textstyle m}$ as its first order Taylor expansion with approximation center ${\textstyle \alpha _{0}}$:

$$m=m(\alpha _{0})+(\alpha -\alpha _{0}){\frac {\partial {m}}{\partial {\alpha }}}|_{\alpha _{0}}$$

In this way, the function m is an affine function, which helps to make the computation easier.

3. Apply m into the minimization problem and solve it;

4. The minimizer becomes the new approximation center;

5. Repeat step(2) - (4) until it converges.

Results

Accuracy

The original scene is generated from a Macbeth chart and is shown below.

Fig. 6: Original scene generated from a Macbeth chart.

We generate 5 sets of data. In each set, the original scene is submerged in a medium with different absorption spectrums and scattering spectrums. The specific medium settings are listed below

• Default absorption spectrum
• Constant absorption spectrum
• Double default absorption spectrum
• Default absorption spectrum and water scattering (inputs.waterSct = 1)
• Default absorption spectrum and water scattering (inputs.cLarge = 1)

Set 1

Fig. 7: Underwater images in a medium with default seawater absorption spectrum and ignoring scattering spectrum.

Fig. 8: Comparison between the actual absorption spectrum and calculated absorption spectrum from our algorithm.

In this set of images, we submerge the original scene in a medium with the default absorption spectrum of seawater. First, the simulated image is generated from PBRT rendering. Then, our algorithm selectively uses results(excluding information about the medium) from the original PBRT rendering, and estimates the medium properties from our algorithm. Finally, we feed in the estimated medium properties to PBRT rendering, and generate modeled images. As we expect, both underwater images appear bluish than the original scene. There exists a bigger discrepancy between the actual absorption spectrum and the calculated absorption spectrum derived from our algorithm towards larger wavebands. Notice that we used 31 wavebands: ranging from 400nm to 700nm with 10nm intervals. Although the error of the calculated absorption spectrum is almost 28%, we only notice a minor difference between the simulated and modelled underwater images.

Set 2

Fig. 9: Images in a medium with constant absorption spectrum and ignoring scattering spectrum.

Fig. 10: Comparison between the actual absorption spectrum and calculated absorption spectrum from our algorithm.

In this set of images, we submerge the original scene in a medium with a constant absorption spectrum, which mimics a fog. As we expect, the images overall appear darker than the original scene. There is a minor discrepancy between the actual absorption spectrum and the calculated absorption spectrum derived from our algorithm. We almost cannot tell a difference between the simulated and modelled images.

Set 3

Fig. 11: Underwater images in a medium with double the default seawater absorption spectrum and ignoring scattering spectrum.

Fig. 12: Comparison between the actual absorption spectrum and calculated absorption spectrum from our algorithm.

In this set of images, we submerge the original scene in a medium with a doubled default absorption spectrum of seawater. As we expect, the underwater images appear even more bluish than the original scene, since the scene is now submerged in more concentrated seawater. There still exists a bigger discrepancy between the actual absorption spectrum and the calculated absorption spectrum derived from our algorithm towards larger wavebands. Although the error of the calculated absorption spectrum is high, we only notice a minor difference between the simulated and modelled underwater images.

Set 4

Fig. 13: Underwater images in a medium with default seawater absorption spectrum and scattering from setting inputs.waterSct = 1.

Fig. 14: Comparison between the actual absorption spectrum and calculated absorption spectrum from our algorithm.

In this set of images, we submerge the original scene in a medium with the default absorption spectrum of seawater. In addition, we turned on inputs.waterSct, which controls the overall scattering coefficient of the water, affecting the scattering of light in the medium. The underwater images appear bluish than the original scene. There exists a bigger discrepancy between the actual and calculated absorption spectrums towards larger wavebands. In addition, there exists a bigger discrepancy between the actual and calculated scattering spectrums towards smaller wavebands. Although the error of the calculated absorption spectrum and scattering spectrum is big, we only notice a minor difference between the simulated and modelled underwater images.

Set 5

Fig. 15: Underwater images in a medium with default seawater absorption spectrum and scattering from setting inputs.cLarge = 1.

Fig. 16: Comparison between the actual absorption spectrum and calculated absorption spectrum from our algorithm.

In this set of images, we submerge the original scene in a medium with the default absorption spectrum of seawater. In addition, we turned on inputs.cLarge, which controls the concentrations of large particles influencing the scattering properties of the medium. The underwater images appear bluish than the original scene. There exists a bigger discrepancy between the actual and calculated absorption spectrums towards larger wavebands. In addition, there exists a bigger discrepancy between the actual and calculated scattering spectrums towards smaller wavebands. The error of the calculated absorption spectrum and scattering spectrum is huge, yet we only notice a minor difference between the simulated and modelled underwater images.

Inadequacy

Although the result images looks like the same, we still find some problems in the result by comparing the plots of two characters. In absorption plots, it fits well when wavelength is short, but has large error at long wavelength. This is because the image formation model can be seen as the product of some matrix and ${\textstyle c}$, which is the projection on ${\textstyle c}$. The variable ${\textstyle c}$ is related to camera's spectral sensitivity curve. The camera we used has RGB channel, so it has very small values at long wavelength on the RGB spectral sensitivity curve. Therefore, when projection on ${\textstyle c}$, big difference in absorption at long wavelength will not causes big change in result. In conclusion, the resulting image is not sensitive to the absorption value at long wavelength.

In addition, the estimated scatter value has large error at short wavelength. The reason is similar, which is that the model is the product of some value and ${\textstyle P_{scatter}}$, and this value is very small at short wavelength, about one to ten percent of others. Therefore, the resulting image is not very sensitive to the scatter value at short wavelength. Because the estimated ${\textstyle P_{scatter}}$ tends to be large at short wavelength, the modeled image looks bluish.

Conclusions

Our project focuses on characterizing a participating medium through camera captures. We employ camera simulations and virtual 3D environments to predict and test the absorption and scattering properties of the participating medium. In the end, we create precise models that faithfully represent light interactions in the medium and simulate the appearance of underwater camera images.

Our experimental designs for characterizing a participating medium, specifically water, using camera captures have provided valuable insights into the challenges and discrepancies in modeling absorption and scattering properties. The larger deviation in absorption towards larger wavebands and in scattering towards smaller wavebands underscores the complexity of accurately modeling light interactions in a medium like water. Surprisingly, despite significant errors in these fundamental properties, the simulated underwater images exhibit only minor deviations from the modeled ones.

To address these discrepancies, future work should focus on enhancing the camera simulation aspect. Introducing cameras with different color filters to compensate for small original RGB responses, and exploring algorithms that integrate outputs from multiple cameras offer promising avenues for refinement. Additionally, adjusting medium settings, such as colored dissolved organic matter (CDOM), non-algal particles (NAP), and phytoplankton parameters, will contribute to aligning the simulation with real-world conditions.

In summary, refining the simulation model through improved camera characteristics and fine-tuning medium settings will advance our understanding of light interactions in participating media. This work is crucial for applications in computer graphics, remote sensing, and underwater imaging, where accurate simulations are paramount for realistic and reliable results.

References

• Blasinski, Henryk, et al. “A model for estimating spectral properties of water from RGB images.” 2014 IEEE International Conference on Image Processing (ICIP), 2014.

• Kaushal, Hemani, and Georges Kaddoum. “Underwater Optical Wireless Communication.” IEEE Access, vol. 4, 2016, pp. 1518–1547.

• T. Doster and A. T. Watnik, "Machine learning approach to OAM beam demultiplexing." Appl. Opt., vol. 56, pp. 3386-3396, 2017.

• The iset3d GitHub Wiki's tutorial

• The isetcam GitHub Wiki's tutorial

Appendix I

Our Matlab simulation script can be found in t_mediumExample221.m. This script creates a simple scene, a medium, and renders the scene submerged in that medium.

Change the medium settings in piWaterMediumCreate.m, where one can adjust the water absorption and scattering spectrum.

Be sure to clone the entire iset3d to the local desktop, and set up the remote rendering environment appropriately to run the MATLAB simulation script smoothly.

Our algorithm can be found in algorithm.py. This file implement the algorithm to estimate the properties of the medium.

Appendix II

Tianyun Zhao: Scheduled meetings with Henryk, communicated with the mentor and instructors, worked in person on the lab computer, worked on the MATLAB simulation parts, collected pixel data with simulation outputs, tested if our algorithm data was valid, and re-collected pixel data with algorithm outputs.

Cecilia Xie: Scheduled meetings with David, worked remotely on the lab computer, worked on MATLAB simulations, collected pixel data with simulation output figures, tested if our algorithm data was valid, and re-collected pixel data with algorithm output, compared the captured and modeled output figures and evaluated results.

Shiqi Xia: Communicated with the mentor and instructors, understood and generated the model, built the algorithm, coded based on the model and algorithm, ran and got the estimated character of medium using the simulation outputs, evaluated the results and gave possible solutions for future work.