JaroensriTanDejesus

Introduction

Melanin and hemoglobin are the dominant pigments that determine the color of human skin. Tsumura’s work in image-based skin color and texture analysis proved it is possible to extract melanin and hemoglobin quantities from an RGB image. By changing the quantities of melanin and hemoglobin, it is possible to morph one person’s skin color into another. However, it should also be possible to do transformations on the spectral reflectance of skin to produce skin tones that are more realistic than color shifts in RGB space. We aim to reproduce this effect of transforming one person’s skin color into another using hyperspectral data either by shifting the quantities of melanin and hemoglobin or by transforming the reflectance values to achieve a match.

ICA Approach

This approach was motivated by Tsumura’s work. We attempted to separate our facial images, either in RGB space, or in hyperspectral space, into several meaningful channels such as melanin, and hemoglobin. The analysis was repeated on many facial images, all with different channel profile. We tried morphing between different channels of different people to alter the skin profile on one into another. We started by working in RGB space to test our algorithm, before actually using hyperspectral information.

Theory and Previous Work

According to Tsumura, the skin obeys Beer-Lambert law of absorption. This means that the intensity of light reflecting of the skin at each pixel is proportional to the exponential of skin absorbance and pigment density, coupled with shading function and the illuminant, as shown in equation 1.

$C(x,y,\lambda )=e^{\sigma _{m}(\lambda )\rho _{m}(x,y)+\sigma _{h}(\lambda )\rho _{h}(x,y)}p(x,y)E(\lambda )-(1)$

, where c represents signal at wavelength $\lambda$ on pixel (x,y), $\sigma _{m},\rho _{m},\sigma _{h},\rho _{h}$ represents absorbance and density of melanin and hemoglobin respectively, $p(x,y)$ represents shading strength, and $E(\lambda )$ represents illuminant at that wave length. When we take a logarithm, all these dependencies become linear, and we could use independent component analysis to separate different absorbance channels, thus allowing us to decompose one’s facial image in a meaningful way [1].

$C^{log}(x,y,\lambda )=\sigma _{m}(\lambda )\rho _{m}(x,y)+\sigma _{h}(\lambda )\rho _{h}(x,y)+p^{log}(x,y)+E^{log}(\lambda )-(2)$

For different $\lambda$, we can write a set of linear equations, and use many ideas from linear algebra to solve for $\sigma _{m},\sigma _{h}$. ${\begin{bmatrix}C^{log}(x,y,\lambda _{1})\\C^{log}(x,y,\lambda _{2})\\C^{log}(x,y,\lambda _{3})\\C^{log}(x,y,\lambda _{4})\end{bmatrix}}={\begin{bmatrix}\sigma _{m}(\lambda _{1})\\\sigma _{m}(\lambda _{2})\\\sigma _{m}(\lambda _{3})\\\sigma _{m}(\lambda _{4})\end{bmatrix}}\rho _{m}(x,y)+{\begin{bmatrix}\sigma _{h}(\lambda _{1})\\\sigma _{h}(\lambda _{2})\\\sigma _{h}(\lambda _{3})\\\sigma _{h}(\lambda _{4})\end{bmatrix}}\rho _{h}(x,y)+{\begin{bmatrix}1\\1\\1\\1\end{bmatrix}}p^{log}(x,y)+{\begin{bmatrix}E(\lambda _{1})\\E(\lambda _{2})\\E(\lambda _{3})\\E(\lambda _{4})\end{bmatrix}}-(3)$

This method of decomposing does not limit to just hyperspectral data. Tsumura showed that certain approximation can be made to make this analysis possible with RGB camera. He performed the test according to his assumption about the skin property, and he was able to separate facial images into melanin, hemoglobin, and shading channels [1]. Tsumura’s algorithm started by assuming that the shading direction was known (which followed from his theory--the vector of 1's), then compressed the image down to two dimensions using principal component analysis, before he actually performed ICA [1]. This led to many iterations, all of which needed to be judged manually for successful separation, and thus would take considerable amount of time.

Our Method

We simplified Tsumura’s algorithm by performing ICA on the (RGB) image and separate them into three independent components. In reflectance space we would be able to extract more channels. To reduce computation time and avoid including non-skin part of the image into the analysis, we select the area around the cheek of the subject to find separation matrix, and use that to separate the whole image. The result in RGB seemed to reveal some underlying property of the skin, whereas the same analysis on hyperspectral data yielded many unrecognizable channels. (See image below) So, most of our work with this approach was mainly in RGB space.

Separation in RGB space. Top left: original image, Top right: first independent component which seemed to be "shading" component, Lower left: "melanin" channel, and Lower right: hemoglobin channel. We can see in the melanin channel the melanin spots which appeared on his face, but some shading also show around his nose.

Separation in reflectance space. Top leftmost: original image (zoomed in). Only the first and the second separated channel (the two to the right of the original) were conceivable as hemoglobin and shading respectively.

Result

In RGB space, we seem to get independent channels that were uniformly recognizable across different people, especially the hemoglobin channel always seemed to be distinct and shading-free. However, morphing different people by substituting their channel profiles, i.e. separation matrix/mixing matrix (see [1]), does not seem to work well. The channel that seemed to be melanin do not truly represent the melanin color, because the “shading” component was not completely black-white, so it also played a role in rendering the correct skin color.

We tried selecting different channels for morphing. We also tried adjusting and matching the channel strength. These techniques worked in some cases (see image below), but it fails most of the time, especially when we tried to morph hemoglobin channel. The morphed image always turned green due to imbalance of red/yellow in the skin. In many cases, the extracted melanin profiles for both dark skin and light skin were quite similar, so when we morph the melanin channel, almost no effect is seen. On the contrary, some similar skins have different profile, so when we morph between the two, an unrealistic effect was created (morphing white skin to yellow skin, but the result becomes a dark skin)

Morphing of light skin (middle) to dark skin (left). The result is shown on right. Note that the skin was darkened, but the color still don't match

Morphing of light skin (middle) to dark skin (left). The result is shown on right. Note that the skin was completely darkened, and it looks plausible that the morphed face has dark skin. The hair was also affected, because we did not mask the hair out

Reverse Morphing of the image above. Again, we morph middle skin into left one, result is shown on the right. The independent component of these two images are quite similar, which is why the morphed images do not change much

Morphing of middle to left. The result is shown on right. Even though the original skins looked similar, the morphed was totally changed.

Morphing of middle to left. The result is shown on right. Same morph as above right, but with adjusted signal power. Note that the skin tone looked more similar, but still noticeably different

Morphing of middle to left. The result is shown on right. Morphing hemoglobin often result in green-ing image, the cause seems to be the subtle different in the green channel of hemoglobin color for each person

Reflectance Domain

Since we were unsuccessful in separating out melanin and hemoglobin channels from the hyperspectral data, we tried to achieve the skin morphing effect by direct operations on the reflectances of skin. A person’s skin color can be characterized by the mean reflectance (mean spectral reflectance vector) of his or her skin. Each element of the mean reflectance is the mean across many skin samples for a single wavelength. In class, we learned that by multiplying spectral reflectance by the spectral power distribution of the illuminant produces the color signal that would be incident on our eyes. The color signal is converted to an RGB quantity by multiplying by the color matching functions. Therefore, transforms in the reflectance domain should affect the color of skin in the RGB domain.

Skin Mask

The skin mask algorithm labels pixels in a scene as skin or not skin. The purpose of the mask is to limit the transform to regions of skin and not to other objects in the scene. We assigned a value of 1 if the pixel was skin, a value of 0 if it was not skin, and a value between 0 and 1 for gray area. A value between 0 and 1 would limit the effect where the strength of the transform approaches full strength as the value approaches 1.

Method

Hyperspectral data gives us more information about an image than an RGB image. Our assumption was that this extra data could be used to identify skins samples in the scene. We model skin reflectances as a sphere that surrounds the mean reflectance in N-dimensional space. If a reflectance vector in the scene falls within this sphere, it is labeled as skin, otherwise it is labeled as not skin. A rectangular region of a person’s cheek is used as a set of skin samples. We compute the mean reflectance vector and the largest distance from the mean vector within the set. The largest distance is a threshold used to determine the radius of the sphere. Through a series of tests, we decide that vectors within a sphere with radius of 0.9*threshold is absolutely skin. Vectors outside this region but inside a sphere with radius of 1.3*threshold is considered gray area, with the value being a linear interpolation from 0 to 1 as the vector approaches this outer sphere. All vectors outside second sphere is considered absolutely not skin.

Results

The algorithm correctly identifies human skin. However, there are a number of false positives detected by the algorithm including background, hair, eyes, teeth, and clothes. In most cases, specular reflectance is labelled as not skin. The limitation of the algorithm is that the reflectance values in each wavelength are weighted equally. Hyperspectral data does not provide a way to uniquely identify skin from other objects in this way. A different approach that would analyze each wavelength separately would be necessary to produce an accurate mask. But for the purposes of this project, this mask algorithm was sufficient

Transform

At first we believed that the values of the mean spectral reflectance vector would be a good indication of the skin color of a person. This holds true in RGB space, but after a series of simple experiments with shifting the mean, we concluded that this does not hold true in reflectance space. Instead, it is the shape of the mean reflectance vector that determines the color of the pixel. Thus we devised a method to imitate the the shape of the reflectance in order to morph one person’s skin color into another person’s skin color.

Method

The algorithm begins by computing the mean reflectance of person 1 and person 2. Each skin reflectance sample from person 1 is divided by person 1’s mean reflectance elementwise. The resulting signal has the property of being equalized across different wavelengths. Next, the signal is multiplied by person 2’s mean reflectance elementwise. This produced spectral reflectance is then stored back into the scene.

Skin Morphing Results

Figure 1a: Transformation of Caucasian skin color into African American skin color

Figure 1b: Spectral reflectance of Caucasian skin color (red line), African American skin color (green line), morphed skin color (blue line)

Figure 1a shows the transformation of a light skin color to a dark skin color. In the RGB space, the morphed skin color looks close to the target dark skin color. We chose to plot the reflectance values at the center of a person’s right cheek to compare the spectral reflectances between the source person, the target person, and the morphed person. In the reflectance space,the spectral reflectance of the morphed skin sample does not matches the shape of the target (person 2) reflectance exactly, but it follows the general trend (Figure 1b). Shading seems to be preserved in the transformation.

Figure 2a: Transformation of African American skin color into Asian skin color

Figure 2b: Spectral reflectance of African American skin color (red line), Asian skin color (green line), morphed skin color (blue line)

Figure 2a shows the transformation of a dark skin color into a light skin color. Visually, we can the skin color is close to asian colored skin, but it comes out lighter. However in our comparison plot of reflectance values, the spectral reflectance of the morphed image is very similar to the spectral reflectance of the the target light skin (Figure 2b). We suspect that reflectance variance is boosted when morphing dark skin into light skin. Specular reflectance can be observed at the forehead and bottom of the nose which makes the morphed image look unrealistic.

Figure 3a: Transformation of Joyce skin color into African American skin color

Figure 3b: Spectral reflectance of Joyce's skin color (red line), African American skin color (green line), morphed skin color (blue line)

Figure 3a shows the transformation of a light skin color into a dark skin color. The morphed skin color is not an accurate match to the target skin color. We believe this is caused by incorrect estimates of the spectral reflectances in Joyce’s scene. Joyce’s light skin color spectral reflectance has lower values compared to other light skin samples we tested on, hence making our computation results inaccurate. Specular reflectance can be observed at the forehead, edges of the face, and nose which causes regions to not be transformed.

Figure 4a: Transformation of Caucasian skin color into East Indian skin color

Figure 4b: Spectral reflectance of Caucasian skin color (red line), East Indian skin color (green line), morphed skin color (blue line)

Figure 4a shows the transformation of a light skin into a slightly darker skin color. There isn’t much change in the morphed skin color. We believe this due to the fact that the mean reflectance vectors are very similar in shape. When looking at the reflectance comparison plot, all three reflectances have a very similar shape even though the values are different. This shows that our morphing is only obvious when the spectral reflectance is relatively different in shapes - dark skin and light skin.

Changing Illuminance: We tried changing the illuminace to see how it affects the morphed skin color. We used daylight and fluorescent illuminance and Figure 5a,5b shows our results. Based on the images rendered, the morphed skin color does not look any more realistic or unrealistic, but the group reached a common opinion that the fluorescent illuminance seems to give us a more pleasant image.

Figure 5a: Morphed image using daylight illuminance

Figure 5b: Morphed image using fluorescent illuminance

Conclusion

In this project, we have tried to morph skin colors using hyperspectral data by using ICA to shift the quantities of melanin and hemoglobin and by transforming the reflectance values to achieve the target skin color.

ICA Approach

The ICA approach was not robust enough as color channels were not accurate representation of melanin and hemoglobin as done in Tsumura’s paper. We have tried altering some of the channels using information from another, and we tried adjusting/matching channel power (rms of signal), but we were not able to consistently morph one person’s skin into another. Due to limited time, we were not able to find a way to successfully morph the skin with consistence. It is also possible, however, to parameterize the morph the same color channel between two people.This would be a very interesting extension in the future, as we might be able to write a program to find the parameters which minimize the error in skin color.

Reflectance Domain Approach

The skin color morphing is successful in cases of where the skin color is distinctly different between the two faces. Specifically, this occurs between African American colored skin and light colored skin (caucasian and asian). We conclude that these results occur because the shape of the spectral reflectance is relatively different shapes for dark skin and light skin. Skin color morphing fails in cases where the reflectance estimates in a scene appear to be incorrect. Thus the method is very sensitive to the values of the reflectance estimates in the data. We also tried changing the illuminant to determine the effect of lighting on the transformed skin. In our opinion, changing the illuminant does not make the morphed skin look more realistic, but the fluorescent lighting produced the most pleasant image. Overall, there are several limitations that prevent our morphs from look realistic: specular reflectance which makes us unable to detect it as skin, inaccurate mask which cannot completely separate the background from skin, and the lack of characterization of values of the reflectance to morph between two similar skin colors .

References

[1] Image-based skin color and texture analysis/synthesis by extracting hemoglobin and melanin information in the skin
[2] fastica ICA software package

Appendix I : Source Code

Reflectance Code
ICA code

Appendix II : Group work load distribution

Ronnachai worked on using ICA to separate the melanin and hemoglobin channels using high resolution hyperspectral face data. Michael and Mei worked on analyzing spectral reflectances in low resolution hyperspectral face data and coming up with different ideas for morphing skin color. Once the reflectance method was finalized, all three of us extensively tested the method across different pairs of samples and analyzed the results.