ChhangMak

Introduction

Motivation

As we all know, oral cancer is in increasingly large industry, and there is a driving need to catch these illnesses earlier. Currently, the way the screening is done is by shining a short wavelength light (typically purple in appearance), into the mouth. When certain wavelengths of light are emitted into the mouth, cancerous and pre-cancerous tissues will have different fluorescent properties compared to healthy tissue. As such, we can use this property to identify cancerous and pre-cancerous tissue in the mouth. Normal tissue is less fluorescent than the cancerous tissue; however, in the mouth, emitted fluorescent light is often much weaker in magnitude than the reflected ambient light or even from our original light source we put into the mouth in the first place. This method would look something like this:

Thus, we want to be able to create a light that excites the fluorescent areas of the mouth. We want to be able to detect this fluorescence, however weak the fluorescence may be relative to the reflected light, so that we can identify cancerous and pre-cancerous tissue in the mouth.

Here is an example of what the initial contrast looks like:File:OriginalContrast.png

Describing a Physical System

That the fluorescence is much weaker than the reflected light presents a challenge to us. We create a system where we pass a narrow bandwidth light (within the excitation spectrum of the fluorophore, which is our fluorescent object, we want to measure) through a shortpass filter. The resultant light would ideally be one that does not have any light with wavelength outside of the excitation spectrum, so that we see as little of the reflected light as possible. This light hits the fluorophore, which absorbs some of the energy, and returns to us via the camera. We pass the light heading into the camera through a longpass filter, again to allow fluorescent light through and block any reflected light (ambient).

This will allow us to detect the fluorescence in a standard camera with Bayer (red-green-green-blue) sensors. We would ideally want to present the camera's output to a doctor who can examine the fluorescence to screen for oral cancer. Thus, we want to optimize our filters and light such that the camera's output demonstrates a great amount of contrast between fluorescing and non-fluorescing parts of the image.

Simulating the Physical System

In this project, we use the ISETCam and ISETCamFluorescence repositories to simulate our camera, longpass filter, shortpass filter, light spectrum, and fluorophore. Using many simulations, we can find optimal cutoff wavelengths for our filters, optimal wavelengths for light emitted for each fluorophore tested.

A representation of our physical system that we simulate is produced below in the figure.

Background

Two important bits of background are necessary as a basis for this project. We must know which color filter arrays we are using, as well as the mechanics of fluorescence to be able to model the system successfully.

Fluorescence Properties

The properties of fluorescence are known to us for various fluorophores, including porphyrins, NADH, FAD, etc., a few fluorophores one might expect to see in the mouth. We use their emission and excitation spectra in our simulations.

Filtering to find fluorescence is nothing new, as Edmund Optics has a line of filters called "Fluorescence Filters" that consumers can purchase. This line of filters, as well as many others, provide us with known transmittances that we can incorporate into our simulation.

Color Filter Arrays

We know that the sensor will interact with the fluorescent and reflected light through its Bayer filter array. That the three filters are sensitive to different wavelengths will play a part in determining what image we see at the output.

We use the above information to guide us in our quest to model physical objects. We build on the code bases of ISETCam and ISETFluorescence, which already contain a lot of the models we wish to use in our simulations.

Methods

To approximate the physical system in simulation, we must have suitable models that are accurate enough to produce realistic-enough outputs on a real camera. We then input these models into ISETCam so that we can acquire simulated images.

Modeling the Light

In the physical system, we would ideally choose a light with a narrow bandwidth. After all, a blue LED in real life has a very narrow spectrum. We choose a simple model to model our light: a gaussian with a standard deviation of 35 nanometers. We can vary the mean of this gaussian to simulate LEDs of different wavelengths and spectral distributions. Here is an example of a light spectrum we may use, with wavelength 400 nanometers and standard deviation of 35 nanometers.

We have also elected to select a few lights that come stock with ISETCam as well, in our many iterations of simulations.

Modeling the Longpass and the Shortpass Filters

Real-life shortpass and longpass filters look like so:

We can approximate that the nonlinear, sinusoidal, and nonzero parts of the transmission spectrum are constant. We can model the transition from the nonzero part to the zero part as a Gaussian distribution. We can model the zero part as just zero. Thus, we model our filters using a piecewise function: before the cutoff frequency, the transmission spectrum is equal to the maximum of the Gaussian distribution; after the cutoff frequency, the transmission spectrum is a one-sided Gaussian.

Modeling the Fluorophore

In our camera image, we want to be able to maximize the contrast between non-fluorescing parts and fluorescing parts of the image. Therefore, we create a two-dimensional scene in ISETCam which contains a square tile. Half of it contains the fluorescent material of our choice. Half of it does not contain the fluorescent material. We use stock fluorescent materials from ISETCamFluorescence (which have preset excitation and emission spectra) for our simulations. Below is an image representing which parts of the tile are fluorescent and which are not, as well as an example camera output that very clearly demonstrates which part is fluorescent.

Modeling the Camera

We model a generic RGB camera in ISETCam. For reference, we use color filter arrays from the Nikon D100 camera. The color filter array sensitivities for this camera is contained in the background section of this project writeup. Here are some of the specifications that we arbitrated for this camera.

• Exposure Time: 1 s
  • This is significantly longer than exposure times for normal cameras. This is because the fluorescence luminance is so weak and so the exposure time must be much longer to capture enough photons from the fluorescence. We know that this may be suboptimal since the mouth may move within a second, but we think 1 second is a reasonable duration to ask a screened patient to keep the mouth still.
• Pixel Size: 2.2 μm squares
• Fill Factor: 90%
• Voltage Swing: 1.15 V
• Electron Well Capacity: 9000

Figure of Merit

To effectively quantify how well a configuration of highpass cutoff, lowpass cutoff, light spectrum, etc. does, we must have a quantitative method for finding how well a configuration produces contrast. To this end, we have elected for a simple approach. Since we know where the fluorophore is in the scene (because we put it there), we also know which parts of the image belong to the fluorescent part and which parts of the image belong to the non-fluorescent part. We can take a section of the fluorescent part of the image and take the average of the intensities of its pixels, and we can do the same for the non-fluorescent part. The difference of these two averages is a simple but effective way of quantifying contrast, and we have elected to use this method as a figure of merit for different cutoff and light configurations. Below we display two partitions we might choose as sections we would want to average over and take their differences to find a figure of merit.

Results

The simulation, when run, produces a simulated camera image on the screen. Here is an example of such output. This particular output is acquired from an exposure time of 1 s, Porphyrins fluorophore model, a lowpass cutoff of 430 nm, a highpass cutoff of 510 nm (both with Gaussian standard deviation of 15 nm), and a narrow bandwidth light that came stock with ISETCam.

Finding Optimal Cutoff Frequencies

Using our method to calculate figure of merit, we sweep across many different lowpass cutoff frequencies and highpass cutoff frequencies for a given light spectrum and fluorophore. We compile the data together to produce a heatmap which is helpful for visualizing which cutoff frequencies are optimal for producing the most contrast. The below heatmap was generated using auto-exposure in the ISETCam settings, the Porphyrins fluorophore, and stock ISETCam light no. 2. We also filtered out noise in the heatmap by setting all negative values to zero, since we don't expect negative values and thus such values are produced by noise.

We see in the contrast map that there is an optimal configuration for a highpass cutoff frequency around 510 nm and a lowpass cutoff frequency around 430 nm. These cutoff frequencies can be used to produce the first image shown in this results section using an exposure time of 1 second. This image is higher contrast than an image produced with no filters:

We can run this simulation again on other fluorophores, like NADH:

Finding Optimal Light Spectrums

We can also run our simulation to plot our figure of merit for different wavelengths of light. Sweeping across the different wavelengths of light yields the following plot. This plot in particular was made with a very small Gaussian distribution of standard deviation 5 nm, and using the fluorophore Porphyrins.

For lights modeled with Gaussian distributions of higher standard deviations, we see that contrast drops off much quicker as we sweep across the wavelengths. We also see that the contrast is much lower as we get more reflected light mixed into the image.

We see that it is thus optimal to use a light with a very narrow spectrum.

Appendix

You can write math equations as follows: $y=x+5$