Digital Twin for Imaging Skin
Introduction
Blood oxygen saturation is an important health indicator. When blood oxygen levels drop below a certain threshold, extremely dangerous, even life-threatening, conditions can occur due to the lack of oxygen reaching vital organs. Platforms to monitor blood oxygen saturation have therefore been a sought-after solution, as seen in the development of pulse oximeter finger monitors and the addition of monitors onto smartwatches, such as the Apple Watch. Both of these types of monitors use reflected light to perform measurements since oxygen levels affect the reflectance characteristics of blood within the skin.
Another skin chromophore that affects reflectance characteristics of skin is melanin content. The distribution of melanin throughout the skin is an important indicator for skin cancers and skin health, and can affect blood oxygen measurements, making it imperative to consider.
For our project, we explore how imaging sensors can capture differences in blood oxygen levels and skin melanin content. Using imaging sensors for this task is attractive to naturally provide spatial information on the levels of the two chromophores of interest. In our project, we first explore methods to model the skin using MCMatlab [1], and simulate changes in the reflectance spectra given changes in blood oxygen levels and/or skin melanin content. Then we model imaging skin with these characteristics using iSetCam [2], to predict how differences will be captured.
Background
Blood Oxygen Saturation Blood oxygen saturation levels can be measured using concentrations of oxygenated and deoxygenated hemoglobin–a protein that binds to oxygen molecules in the blood. By measuring the ratio between the concentration of oxygenated hemoglobin, cHbO2, to the total concentration of hemoglobin in the blood, cHbO2+cHHb, a blood-oxygen saturation level, sO2(a), can be determined and used for medical diagnosis [3].
sO2(a)=cHbO2cHbO2+cHHb100% (1)
The optical absorption characteristics for oxygenated hemoglobin and deoxygenated hemoglobin are different at different wavelengths, as seen in the absorption spectra in Figure 1. Because of the distinct differences in their absorption spectra, we recognize that we can use reflected light as a measurement tool to differentiate between oxygenated and deoxygenated hemoglobin [2].
Methods
MCMatlab for Modeling Skin Reflectance
iSetCam for Modeling Skin Imaging
Once reflection data has been simulated in MCMatlab, we can use iSetCam to model imaging skin with the specified reflectance data. To start we first combine reflection data for various skin types into a single scene; this is useful for easy comparison between skin properties.
With the scene created from the reflectance data, we can use iSetCam to adjust the illuminant on the scene. Using illuminants with different spectral densities, different portions of the spectrum can be probed more or less heavily. The simulated reflectance of skin may show more or less sensitivity to blood oxygen levels and skin melanin content around specific wavelengths. Therefore tuning the spectral density to probe these regions of interest explores how illuminant conditions affect the distinguishability of skin characteristics. In our study, we include both broadband white light sources and LED-like sources with high spectral density around a single wavelength.
The scene and illuminant combination represents the light that would be expected to be reflected back from the scene given the illuminant’s spectral density. This light can then be processed at a sensor to model how different sensors respond.
Using iSetCam the sensor properties can be adjusted; many characteristics can be explored including, pixel size, noise properties, filter arrays and many more. In this study, we are largely interested in the spectral response of the light, and how to tune the illuminant spectrum and the sensor filter spectrum to best capture various skin features. In this case we kept noise properties to include both electronic and shot noise in all cases, as well as imaging conditions like field-of-view and imaging distance constant. We considered real sensors based on data from physical and purchasable items, as well as idealized sensors. For our study we investigated both monochrome or intensity sensors, as well as color sensors.
Once a filter with specific properties has been set, the light can be processed at that sensor. This leads to expected pixel values that can be considered in either voltages or digital values. To predict distinguishability we pull these pixel values and look at the average value for each set of skin characteristics. We can evaluate these pixels by looking at which filters provide the most and least sensitivity to skin characteristics to lay the groundwork for data processing algorithms to accurately describe skin characteristics. Considering low sensitivity sensor responses is useful to calibrate the data, and then use high sensitivity portions to accurately read the value after calibration.
Results
Melanin and Blood Oxygen Level Sensing
Using MCMatlab with the absorption and scattering coefficients discussed in the background and methods section, the output spectrums are obtained for varying skin melanin content and blood oxygen levels. Higher melanin counts show reduced sensitivity to blood oxygen level changes. For this reason, shown are the two melanin content extremes, with type I skin containing a low percentage of melanin, and type VI containing a high percentage of melanin.
With this simulated reflectance data, iSetCam is then used to predict the light reflected from skin with certain blood oxygen and melanin levels under a specified illuminant. Fig. X+1 shows the expected reflected light under four illuminants: equal energy white light, D65 white light, 620 nm red LED, 520 nm green LED. Additionally, eight skin types are shown under each illuminant, type I epidermis with blood oxygen levels of 85%, 90%, 95% and 100%, and type VI epidermis with blood oxygen levels of 85%, 90%, 95% and 100%.
With the light charts shown in Fig. 10, we can now process these different spectral reflections at various sensors. Considering the output of the monochrome sensor, with unitary quantum efficiency at all wavelengths a processed image is given in Fig 11.
The processed images give a prediction of what the output image would look like using standard post processing of the pixel values. However, since signatures appear rather faint, looking at the raw pixel data, and doing specific post processing may enable better skin characteristic detection. Figure 12, shows the average pixel value for a given skin characteristic combination.
The curves in Fig. 12, depict the average pixel values for a given skin type under the specified illuminant. These curves can be thought of as taking horizontal slices of pixel values in an image like Fig. 10 and Fig. 11 and averaging them for an epidermis type. The white illuminants, equal energy and D65, wash out the type I skin to the point signals held about the blood oxygen level are indistinguishable. In type VI epidermis skin the white lights see a slight upward trend in value, coming from the increased reflectance at higher blood oxygen levels. The red LED illuminant shows the greatest contrast between blood oxygen levels. The increased contrast can be observed in the clearly defined steps at each transition in skin oxygen level in the red LED illuminant column. The green LED is fairly flat, or non-responsive to blood oxygen levels. A monochrome sensor may be attractive to take multiple images of skin using various LED illuminants, where a green or blue LED may be used to calibrate the image taken with the red LED. Since the blue or green LED can be insensitive to blood oxygen levels and sensitive to other factors like melanin content or blood volume, images under green or blue LEDs could be used to map melanin and blood volume. Then using the blue-green calibration map, the red LED image can be processed to pull out blood oxygen content.
The AR0132AT imaging sensor, with a Red-Clear-Clear-Clear (RCCC) filter arrays, is a more realistic sensor than the idealized monochrome sensor discussed above. It consists of a repeating four-block array, where one pixel has a red filter and the other three have clear filters. Unlike the idealized monochrome sensor the clear filters do not exhibit perfect unitary quantum efficiency. It has a given quantum efficiency in Fig. 13.
Processing the light at the sensor and filter, the pixel data gives insight into the sensor's ability to capture skin characteristics. The pixel data is processed the same way as before, by average horizontal slices of the pixel values for each epidermis type.
Figure 14, shows in both type I and type VI epidermis the filter is able to resolve clear steps from blood oxygen levels in the red pixel data when the light contains some fraction of red light. Additionally, the clear pixels also pick up this sensitivity; this can partially be attributed to the slight favoring of quantum efficiency the clear pixels have towards red wavelengths. However, in bright conditions, like white light with type I skin, the steps get washed out in the clear pixel but are clearly obtained in the red pixel values. The clear filter slightly favors red light and can be useful for imaging brightness, as well as blood oxygen level. The values of the red pixel and clear pixel could be used in conjunction. Clear pixels could be used to calibrate lightness which can relate to skin melanin count and red pixels could be used to determine blood oxygen levels based on the calibration of brightness. In the calibration and analysis process, taking into account the spectral quantum efficiency of each pixel filter is imperative to properly extract data. This imaging system is similar to the monochrome, where it is largely dominated by intensity, but has selective sensitivity to red wavelengths where maximum blood oxygen contrast resides.
The monochrome and RCCC filtered sensors show distinguishable sensitivity to blood oxygen levels under red light conditions but may require additional images with specific illuminants to perform accurate calibration. Due to the desire to probe certain wavelengths for certain information about the skin, it is interesting to consider a colored image, and how one could pull pixel data for broad spectral information. The first color imaging sensor considered in this study will be the Sony IMX363. The IMX363 sensor has the spectral quantum efficiency given in Figure 15a, with corresponding pixel layout shown in Figure 15b.
Considering the same four illuminants as before, and using the Sony IMX363 sensor with this spectral quantum efficiency to process an image of the skin, the following pixel values are obtained.
Both the type I and type VI epidermis show the upward trend with blood oxygen level in the raw green and red pixel values. The green pixel has minor quantum efficiency in the 600-700 nm region, where blood oxygen levels most heavily influence the reflectance spectrum. These upward trends in both red and green pixels are therefore expected, as both pixels exhibit spectral sensitivity to the increased reflectance with increased blood oxygen levels. The blue pixel has almost no quantum efficiency in this region and obtains very flat bands with respect to blood oxygen levels. Using post processing on this data, the blue pixels can be important to calibrate the sensor’s melanin values. The green and red pixels can be used in conjunction with the calibration from blue pixel values to resolve blood oxygen levels. The relative intensities of each of the pixels can be used to determine melanin content and blood oxygen content in a post processing algorithm that takes into account the quantum efficiency of each color filter.
The type VI epidermis skin exhibits reduced pixel value contrast between blood oxygen levels, and noise effects become apparent increasing the difficulty of distinguishing different blood oxygen contents. The filter array used for the IMX363 sensor shows the greatest sensitivity to green wavelengths, and filters out much light in the region of interest between 600 and 700 nm. The low sensitivity to red light reduces the number of photons processed at the sensor containing important blood oxygen level information when lower light conditions are experienced. Therefore, a filter array that uses multiple color filters to probe specific portions of the spectrum, but shows high sensitivity to red wavelengths is attractive to explore.
A Cyan-Yellow-Yellow-Magenta (CYYM) may be the answer. The CYYM filter with the quantum efficiencies and filter layout shown in figure 17 has increased sensitivity to red wavelengths over the Bayer-RGB filter used for the Sony IMX363.
In this filter array both the yellow and magenta pixels show significant sensitivity to red wavelengths, enabling high capture rate of important red photons. The cyan pixel shows a dip in quantum efficiency in the region, it is expected to show low sensitivity to blood oxygen levels. Processing the skin image through a sensor with this CYYM filter array gives the average pixel values shown in Fig 18.
In both the yellow and magenta pixels, and type I and type VI epidermis, clear steps are visible due to blood oxygen levels. This response is expected as these pixels exhibit significant quantum efficiency in the red wavelengths that are sensitive to blood oxygen levels. The cyan pixel has a dip around this region, leading to flat bands with respect to blood oxygen content. This filter array shows high sensitivity to blood oxygen content, and has cyan pixel information to calibrate melanin content. With post processing algorithms, it is expected a reliable analysis of the images can be made using the three pixel values to calibrate melanin and blood oxygen levels. We have highlighted how it is necessary to have high sensitivity to red wavelengths to successfully obtain signatures from blood oxygen content, while it is also useful to extract calibration data for melanin content that is not sensitive to blood oxygen levels to prevent melanin content from affecting blood oxygen level predictions.
Pulse Sensing
Up to this point blood volume has been considered constant. However blood volume in a given area changes with time due arterial pulses. In MCMatlab blood volume dependent spectra can be simulated by adjusting the volume of blood taken up by the blood layer. Figure 19 shows the reflection spectra of skin when there is a small volume of blood and a large volume of blood in type I epidermis skin.
There is a clear bump in the reflection spectrum for small blood volume slightly past 500 nm coming from the absorption increase of blood around this spectral location. In contrast the large blood volume sees a decrease in this feature from increased absorption. This spectral feature can then be probed with green light around this wavelength. This difference can be considered using the IMX363 sensor again, and pulling green pixel information.
There is a clear difference between the two blood volumes visible in the green pixel data. However, pulling the raw pixel data exhibits the importance of considering other pixel values. The green pixel in the imx363 sensor has quantum efficiency overlapping with the 600-700 nm region. Therefore the green pixel values show sensitivity to blood oxygen level that could blur proper blood volume calibration. Post processing the pixels considering the red pixel response can flatten out the green value response, removing sensitivity to red wavelengths that could blur proper blood volume calibration.
Figure 21, shows an example of post processing pixel data in the default way to render an image in iSetCam, eliminating some negative effects from quantum efficiency overlap. In this method the green values are calibrated against other pixel data flattening out the response, making these values more useful for blood volume analysis than raw green pixel values that exhibit blood oxygen level dependency.
Conclusions
MCMatlab has positioned itself as a useful tool for simulating skin reflectance data. MCMatlab has the freedom to change multiple skin parameters, enabling many datasets to be generated. In experimental settings similar data may be hard to gather, further highlighting the benefits of developing an easy to use, and accurate simulation software. We first discuss improvements made to multi-layer skin modeling to better match output reflection spectra to experimentally measured skin reflectance values. We show important features in the reflection spectra dependent on blood levels, blood oxygen levels and melanin levels are all visible in the simulated data.
We then discuss how imaging of these generated reflection spectra can be modeled in iSetCam, exploring sensor and illuminant properties that enable optimal skin characteristic imaging. We note the importance of having high sensitivity to red wavelengths corresponding to the portion of the skin reflectance spectra with the greatest variation due to blood oxygen content. Additional data in the form of multiple pictures under multiple illuminants, or processing of color-filtered pixels is imperative to properly calibrate the data to melanin and blood volume. In monochrome or RCCC filters, the method of multiple images with calibration between these images may be useful to create accurate detection algorithms. In color pixels, comparative processing of pixel data can probe specific portions of skin reflection spectra that hold information on melanin content, blood volume and blood oxygen levels when considered against one another. A blend of high sensitivity to red wavelengths to pick up small changes caused by blood oxygen content, and non-negibile sensitivity to outside wavelengths could be a powerful platform to accurately image the spatial distribution of melanin and blood in the skin.
References
[1] Marti, Dominik, et al. “MCmatlab: an Open-Source, User-Friendly, MATLAB-Integrated Three-Dimensional Monte Carlo Light Transport Solver with Heat Diffusion and Tissue Damage.” Journal of Biomedical Optics, vol. 23, no. 12, SPIE-Intl Soc Optical Eng, Dec. 2018, p. 1, doi:10.1117/1.jbo.23.12.121622. [2] Brian Wandell (2024). “isetcam” (https://github.com/ISET/isetcam), GitHub. [3]Chris Higgins, “Oxygen saturation: Better measured than calculated” https://acutecaretesting.org/en/articles/oxygen-saturation-better-measured-than-calculated [4] George Zonios and Aikaterini Dimou, "Light scattering spectroscopy of human skin in vivo," Opt. Express 17, 1256-1267 (2009). [5] Steven L. Jacques, “Skin Optics”, Oregon Medical Laser Center News, (1998), https://omlc.org/news/jan98/skinoptics.html [6] Ritvik Sharma, Pranil Joshi, Arjun Deopujari "Blood oxygen modelling from skin reflectance" (2022) https://vista.su.domains/psych221confluence/2022/BloodOxygenModellingFromSkinReflectance.pdf [7] Pranil Joshi (2022) "skin_reflectance_matlab" (https://github.com/PranilJ/skin_reflectance_mcmatlab), GitHub. [8] Empirical Health, "Oxygen saturation", https://www.empirical.health/metrics/oxygen.