LeeKosarajuSankaranarayanan

Introduction

The human visual system responds to only wavelengths of light in the range 400-700nm. Although not perceived by the human eye, light in the wavelengths from about 800-2500nm, which represents the near-infrared region of the electromagnetic spectrum, is rich in information about the scene that can be utilized for a variety of applications. Remote sensing community has a long history of doing this. Since silicon is sensitive to these wavelengths, near-infrared capture could be enabled in digital cameras and the information could be used to enhance photographic images. A number of such applications have been proposed by researchers [place holder for references]. In our work, we are concerned with evaluating some of these algorithms used for contrast enhancement, haze removal and skin smoothing.

As these applications work independent of each other, methods and results are discussed separately for each of these.

Contrast Enhancement

Motivation

Images taken under normal exposures suffer from lack of contrast under low light conditions. Long exposure shots are not suitable to fix the problem if there are moving objects in the scene. On the other hand, NIR images tend to have a better contrast than the visible images. Zhang et al suggest using near-infrared images to enhance the contrast by a suitable transfer to the visible image <ref no>. The reason we think NIR images have better contrast is because silicon has a better quantum efficiency in the near-infrared than in the visible regions. From Figure <#> one could see that the quantum efficiency in the NIR bands is 50-60% greater than that in the visible portions.

Response of a silicon photodetector

Skin Smoothing

Motivation

As is the motivation of this project in general, the skin smoothing portion is designed to be an automation of a commonly performed post processing technique. In many cases, photographs (specifically portraits) are edited to remove small unwanted characteristics in faces. This can be seen in images of models in ads, actors in movie posters, and professional grade portraits. It is common to "touch up" a photograph by removing small wrinkles, blemishes, and freckles from the skin; often this is done in photoshop by a professional editor or with relatively complex algorithms to attempt to automate the process. It is proposed by Susstrunk and Fredembach that near infrared data can be utilized to more accurately automate this process.

The theory for using infrared is described best by the reflective properties of melanin and hemoglobin -- longer wavelength light reflects less. This means that near infrared light will penetrate further into the skin than that in the visible spectrum. With that in mind, it can be shown that infrared images effectively capture the deep, important structural information of a portrait, while not capturing the shallow, unwanted information. For example, freckles and small wrinkles do not show up in an infrared image, but the edges that define the eyes, mouth, and nose do. Using this as a basis for the theorem, this portion of the project explored possible algorithms to generate smoothed images that maintain all of the structural information intended to be kept in the final portrait.

Methods

In exploring algorithms for skin smoothing, several papers were referenced and read, but the basis of the final algorithms originated from two sources. Each of the algorithms have the same general structure, but have a few distinct differences. Each will be discussed independently, and the results will be shown separately in the next section.

The first algorithm stems from the work of Fredembach and Susstrunk [REFERENCE], in their work on skin smoothing using infrared. The first step in this algorithm is to convert the RGB image to the YCbCr color space. A bilateral filter is then applied to both the luminance channel of the visible spectrum (the Y), and the infrared image. In doing this, the luminance channel blurs the color image, making all edges more gradual and thus all details a bit more smooth. This effectively makes the image a smoothed version of the original. After applying a bilateral filter to the Near Infrared image, the detail layer must be obtained. In this algorithm, the method for defining the detail layer is undefined, but the idea is to keep only the important edges. By calculating the residual image (filtered image minus the original image) only the edge details remain as non-neutral values. It is then possible to threshold for large values of the residual image (in both positive and negative directions), so that when an edge is known the luminance of the combined image comes from the residual, and when there is no edge the luminance is the blurred version calculated from the filtered color image. Now, after the final luminance is calculated, the chrominance information from the color image (the CbCr) is added back into the image, and all is converted back into the RGB color space.

The second algorithm is based on the work of Edelmann and Durand [REFERENCE]. The inspiration for this algorithm came from exploring uses for bilateral filters. The authors were attempting to use two images -- one taken with a flash and one taken without -- to create a combined image that has the structure of a well lit scene with the smooth and warm lighting of a scene lit with ambient light. The idea is that the structure can be taken from the information of a flash image and the color taken from the non-flash image. Similar to the first algorithm, the first step is to convert to YCbCr. A bilateral filter is still applied to the luminance channel of the RGB image, but now two bilateral filters of differing widths are applied to the IR image. A smaller width image will smooth less, and will preserve the details of the edges more precisely, while a wider filter will smooth more broadly. The two filtered results are then divided to provide a ratio, which creates a smoothed luminance channel from the original IR image. Now the chrominance information is added back in, and the entire image is converted back into the RGB color space.

Results

In examining the outputs from the different algorithms, it can be concluded that using near infrared as a method of automating skin smoothing does work quite well. In the outputs of the first algorithm, the blemishes and wrinkles exhibited in the original images are smoothed out of the final image. In the zoomed in image, it can be seen that the details fundamental to the structure of the face are still in tact (e.g. the edges around the eyes). What is interesting to point out is that the first algorithm allows for easy manipulation of the amount of smoothing in the final image. By allowing the user access to the width of the bilateral filters as a variable parameter, this can become an easy automatic version of skin smoothing.

The second algorithm, on the other hand, allows for more precise manipulation of the amount of smoothing allowed. Since three bilateral filters are used, each parameter can change the final output quite significantly, and thus would not be as easily used by an average user to automate skin smoothing. This method has the potential to yield the same or better results, but requires more precise manipulation of several variables to get a desired output.

Overall, the skin smoothing algorithms that utilize near infrared are comparable to those that use just RGB data, and are generally more simple for an average user. They do not require an experienced photoshop user or any significant user input to generate images that remove unwanted features while maintaining all necessary structural features of a face. In the end, there is improvement that can be done on the algorithmic side, but this is proof that the theory is sound.

Conclusion and Future Directions

Future Directions

Shadow Elimination

Illuminant Detection

Conclusion

The algorithms and theories explored in this project are proof that near infrared data is useful for post-processing techniques and may allow for automation of computational photography methods. They improve upon the methods currently used, and show that exploration of further uses is worthwhile. It may not, however, be yet the time to rework the imaging pipeline to allow for near infrared data to be captured by digital cameras.

Acknowledgements

We would like to thank Dr.Joyce Farrell and Dr.Torbjorn Skauli for providing us with copious hyperspectral data and also for their help throughout the project. We also appreciate the feedback from Prof.Brian Wandell and Henryk Blasinski that helped us refine our project thought process. I'm missing some people here, Dr.Hagit, yes? please add people who helped you Serene and Evan