DXMPsych2012Project

Back to Psych 221 Projects 2012

Introduction

With the proliferation of digital cameras and advancement of processors, software image processing and image perception applications have become increasingly important and widely-used. In applications such as face localization, human detection and pose classification, skin detection is usually the first step for segmentation. Skin detection using color information has been long studied. Color based skin detection is fast and simple to implement, however, it usually suffers from high false detection error due to metamerism, i.e. nonskin surface having the same color values as skin. This project will evaluate performance of skin classifiers in different color spaces, and answer the question - is there a color space that is the best for skin detection? In this project, I will build skin classifiers in RGB, YCbCr, HSV, xyY and CIELAB, and evaluate their performance based on the false detection and false dismissal rate. In addition, we will construct hybrid skin classifiers that combine information of multiple color spaces.

Methods

Color Space

We use a pixel based histogram approach to construct skin detectors. For each pixel, we extract the chromatic values in all 5 color spaces, we call these values chromatic features. These chromatic features are used to construct histogram-based skin detectors. There are 10 chromatic features in a pixel, $[r\;g\;Cb\;Cr\;H\;S\;x\;y\;a\;b]$. We ignore brightness and lightness values in order to achieve illuminant invariance for the skin detectors. In the next section, we will explain how the chromatic features are defined.

RGB

$[r\;g]$ are the chromatic features in RGB color space. They are normalized red and green values that are defined as

$[r\;g]={\frac {[R\;G]}{R+G+B}}$

YCbCr

$[Cb\;Cr]$ are the blue-difference and red-difference values in YCbCr color space. They are normalized to range between 0 and 1 for this project.

HSV

$[H\;S]$ are the hue and saturation values in HSV color space.

xyY

xyY color space is a normalized presentation of XYZ color space. $[x\;y]$ are defined as,

$[x\;y]={\frac {[X\;Y]}{X+Y+Z}}$

CIELAB

$[a\;b]$ are normalized $a^{*}$ and $b^{*}$ values in CIELAB color space. $a$ and $b$ range between 0 and 1. $a=0$ indicates green and $a=1$ indicates red, similarly, $b=0$ indicates blue and $b=1$ indicates yellow.

Classification

Histogram

We used MCG-Skin dataset to train, validate and test our skin detectors. MCG-Skin consists of 1000 images of people collected from the Internet, the dataset also contains the ground-truth of human skin for the 1000 images. From the MCG-Skin dataset, we chose 100 images for training, 100 images from validation, and 100 images for testing.

With all the chromatic features define, we constructed a two-dimensional histogram for each color space from the training set. Let $H_{rg}$ be the histogram of RGB color space, and $H_{rg}(i,j)$ is the value of bin $[i,j]$ of the histogram. Let $N(k)$ be the number of pixels in training image $k$, and $I_{(k,l)}$ be the $l^{th}$ pixel of $k^{th}$ image. Define $r_{(k,l)}$ and $g_{(k,l)}$ to be the $r$ and $g$ features of $I_{(k,l)}$. In addition, we denote the bin size of the histogram as $\tau$. The histogram can be expressed as

$H_{rg}[i,j]=\sum \limits _{k=1}^{100}\sum \limits _{l=1}^{N(k)}\mathbf {1} \{i\tau \leq r_{k,l}<(i+1)\tau \}\mathbf {1} \{j\tau \leq g_{k,l}<(j+1)\tau \}(\mathbf {1} \{isskin(I_{k,l})\}-\mathbf {1} \{\lnot isskin(I_{k,l})\})$

In other word, $H_{rg}(i,j)$ is the number of skin pixels minus the number of non-skin pixels in the range of bin $(i,j)$.

We then normalize the histogram to $[0,1]$ by

$H_{rg}[i,j]={\frac {max(H_{rg}(i,j),0)}{max(H_{rg})}}$

The histogram is further smoother by a 2D Gaussian filter $G_{\sigma }$ whose variance is $\sigma ^{2}$.

$H_{rg}[i,j]=(H_{rg}\star G_{\sigma })[i,j]$

Histograms for other color spaces $H_{CbCr}$, $H_{HS}$, $H_{xy}$ and $H_{ab}$ are similarly constructed.

Figure 1: Mesh plot of the histogram in RGB color space

Figure 1 shows a mesh plot of the histogram in RGB color space. Figure 2 shows the same histogram in heat map representation. The x axis represents $r$ value and the y axis represents $g$ value.

Figure 2: Heat map of the histogram in RGB color space

Classifier

The skin classifier for a color space is a binary matrix that has the same size as the corresponding histogram. Let $C_{rg}$ be the classifier of RGB color space, and let $T_{rg}$ be the threshold for $H_{rg}$. Then

$C_{rg}[i,j]=\mathbf {1} \{H_{rg}[i,j]>T_{rg}\}$

Classifiers for other color spaces $C_{CbCr}$, $C_{HS}$, $C_{xy}$ and $C_{ab}$ are similarly constructed. Figure 3 shows the trained classifiers of the 5 color spaces.

We define 1 to indicate skin, and 0 to indicate non-skin. A pixel $I$ with chromatic features $[r\;g\;Cb\;Cr\;H\;S\;x\;y\;a\;b]$ is labeled as either 1 or 0 with following equations. We only present classification using RGB classifier $C_{rg}$, since classification using classifiers in other color spaces are applied in a similar way.

$\quad I_{label}=C_{rg}[i,j]$

$i=floor({\frac {r}{\tau }})\quad j=ceiling({\frac {g}{\tau }})$

Note that $\tau$ is a hyperparameter that is specific to the classifier. $\tau$ of a diffident classifier will take on a different value.

Figure 3: Classifiers of the 5 Color Spaces

Evaluation

We evaluate the performance of a classifier by computing the false dismissal rate $\epsilon _{FN}$ and the false detection rate $\epsilon _{FP}$ of the classifier. The false dismissal rate $\epsilon _{FN}$ of a classifier is the number of skin pixels that are not classified as skin divided by the total number of skin pixels. The false detection rate $\epsilon _{FP}$ of a classifier is the number of non-skin pixels that are classified as skin divided by the total number of non-skin pixels.

Hyperparameters

In constructing classifiers, we need to find the value of hyperparameters $\tau$, $\sigma$, and $T$ for each color space. The hyperparameters of a classifier are calculated by minimizing the false detection and false dismissal error rate of the classifier.

$[\tau \;\sigma \;T]=\operatorname {arg\,min} _{\tau \;\sigma \;T}(\epsilon _{FP}+\epsilon _{FN})$

Hybrid Method

We also constructed hybrid classifiers that combines the results from multiple single-color-space classifiers. The hybrid classifiers used a majority vote approach. For example, if we use $C_{rg}$, $C_{CbCr}$, $C_{xy}$, and $C_{ab}$ in our hybrid classifier, then a pixel is labeled 1 if two or more single-color-space classifiers labeled the pixel to be 1. We evaluated the performance of hybrid classifiers with different combination of single-color-space classifiers.

Results

Results of single-color-space classifiers

Figure 4 presents the error plot of skin classifiers of the 5 color spaces. The y axis is the error rate. An ideal classifier would have very small values for both false dismissal and false detection.

Figure 4: Error Plot of Single-Color-Space Classifiers

First, we observed that false detection rate is higher than false dismissal rate for all classifiers. This is expected because of metamerism. We also noticed that the false dismissal rate and false detection rate of $C_{CbCr}$ and $C_{ab}$ are lower than the error rates of $C_{sv}$. This result indicates that skin detection performed in YCbCr and CIELAB color spaces is more accurate than skin detection in HSV color space.

Results with combined method

Figure 5 presents the error plot of hybrid classifiers. The first five hybrid classifiers combined results from 4 single-color-space classifiers. The name of the classifiers indicates which single-color-space classifier it did not use. For example, hybrid classifier "-RGB" combined results from $C_{CbCr}$, $C_{hs}$, $C_{xy}$, and $C_{ab}$, but it did not use the results from $C_{ab}$. The last hybrid classifier used results from all five single-color-space classifiers.

Figure 5: Error Plot of Hybrid Classifiers

Again, the false detection rate is high across the hybrid classifiers. The false detection rate, however, is lower than any of the single-color-space classifiers. "-HSV" hybrid classifier achieved a false dismissal rate of 0.09. Figure 6 shows an example of a test image classified by "-HSV" hybrid classifier. The classified image is filtered with morphological filters to remove some of the false positives.

Figure 6: Example of Classified Image Using "-HSV" Hybrid Classifier

Conclusions

From the results of this project, we conclude that human skin classifiers in YCbCr and CIELAB color spaces perform better than skin classifier in HSV color space. However, the difference in performance between classifiers in different color spaces is not very significant. As a result, when designing a skin-detection software, performance different in different color spaces should not be the biggest factor in deciding which color space to use. The engineer who designs the application should consider to work in device output format to safe computation cost. We observed that hybrid classifiers have lower false dismissal rate than single-color-space classifiers. In addition, we confirmed the fact that false detection rate is higher for all classifiers, this error is a result of metamorphism. Because of high false detection rate, we have to do some post processing, such as applying morphological filters and connected component noise removal. In this project, we implemented pixel based human skin classifiers. The results show that the performance of this kind of classifiers is far from ideal. The next step is to implement super-pixel based classifiers in order to improve the classification performance.

References

[1] Cahi, D. and Ngan, K. N., “Face Segmentation Using Skin-Color Map in Videophone Applications”, IEEE Transaction on Circuit and Systems for Video Technology, Vol. 9, pp. 551-564 (1999).

[2] Crowley, J. L. and Coutaz, J., “vision for Man Machine Interaction”, Robotics and Autonomous Systems, Vol. 19, pp. 347-358 (1997).

[3] Kjeldsen, R. and Kender., J., “Finding Skin in Color Images”, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition, pp 312-217 (1996).

[4] Singh, S. K., Chauhan, D. S., Vatsa, M. And Singh, R., “A Robust Skin Color Based Face Detection Algorithm”, Tamkang Journal of Science and Engineering, Vol. 6, No. 4, pp. 227-234 (2003).

Appendix I - Code and Data

Code

CodeFiles

Data

zip file with my data