Irtorgb

Introduction

Near-Infrared (NIR) images have broad application in remote sensing and surveillance for its capacity to segment images according to object’s material. Although NIR images made object detection an easier task, its monochrome nature is conflicted with human visual perception and thus might not be user friendly. Lack of color discrimination or wrong colors on NIR images would limit people’s understanding and even lead to wrong judgement. So colorizing the grayscale NIR images would be desired.

Colorization of NIR images is a difficult and challenging task since a single channel is mapped into a three dimensional space with unknown interchannel correlation, which greatly reduces the effectiveness of using traditional color correction/transfer method to solve this problem. Moreover, since surface reflection in the NIR spectrum band is material dependent, some objects might be missing from the NIR scenes due to their transparency to NIR. Therefore, different from grayscale image colorization which only estimates chrominance, IR colorization requires estimating not only the chrominance, but also the illuminance, which add a lot complexity to the problem.

In this project, we proposed several machine learning solutions like L3 (Local, Linear and Learned) and a neural network based model to find the appropriate mapping from NIR to RGB visible spectrum representation which human eyes are more sensitive to. The results are evaluated by CIELAB ∆E and MSE.

Method

L3 Method

Dataset

Our input dataset are the original scene NIR images with 6 fields (mcCOEF, basis, comment, illuminant, fov, dist). We started with 26 such images. With some adaptation from the script in L3(https://github.com/ISET/isetL3), we first created a RGB sensor, which generates the corresponding IR sensor with a irPassFilter. Then these two sensors read in the NIR images, along with the padded spectral radiance optical image computed from those spectral irradiance NIR images (to allow for light spreading from the edge of the scene), and output from sensorCompute(irSensor/rgbSensor, oi) the sensor volts(electrons) at each pixel from the optical image. Finally we used an ipCompute on these sensor volts images for the final sensor data images after demosaicing, sensor color conversion, and illuminant correction.

Note that our output images shows a reddish effect on the RGB images. We couldn't find the right way to get the normal colored RGB images and IR images with the same size - saving the images directly from the optical images gives the right hue on a complete size image, however the corresponding IR images cannot be saved this way. So for the purpose of the L3 training we decided to keep our original output from the image processing pipeline(ipCompute), which includes IR and RGB images of size 198*242, with the RGB images showing a reddish effect.

CNN Method

Due to the great performance of CNN models in image classification tasks [1], we propose an integrated approach based on deep-learning techniques to perform a spectral transfer of NIR to RGB images. As inspired by Matthias et al. [2], a Convolutional Neural Network (CNN) is applied to directly estimate the RGB representation from a normalized NIR image. Then the colorized raw output of CNN model would go through an edge enhancement filter to transfer details from the high resolution IR input image and obtain better image quality.

Dataset

A deep CNN model might have thousands or even millions of trainable variables, so a large amount of data is required to prevent overfitting, where the model perfectly fits the training data set but loses the ability to inference the spectrum transfer on a new IR image. We use the RGB-NIR Scene Dataset [3] which consists of 477 images in 9 categories captured in RGB and Near-infrared (NIR).

Example NIR image of Urban Building

Example RGB image of Urban Building

Number of 64 * 64 Patches used in CNN Model
Data Type/ Data Use	Train	Test
Input IR (64 * 64)	7572	7572
Output RGB (64 * 64 * 3)	1044	1044

Based on promising result of single category training in L3, we used one category data that consists of 102 high resolution Urban Building images in our CNN model. All the images are split into 4/5 training data and 1/5 test data and then cropped into a 64 * 64 patches to feed into the CNN model.

CNN Architecture

CNN Model Architecture

Illustration of our CNN network architecture is given below. We use many convolution/deconvolution layers and relatively few pooling layers to increases the total amount of non linearities in the network. The-non linearities would help us learn complex mapping from IR to RGB. The activation function of each convolution layers is the Relu function: $Relu(x)=max(0,x)$, batch normalization is then followed to avoid overfitting.

In the training process, we use stochastic gradient descent to minimize the mean squared error (MSE) between the pixel estimates of normalized output (oRGB) and normalized ground truth RGB images(GT). MSE is defined as follows:

$MSE=\frac{1}{n}\sum _{i=1}^n(oRG{B}_i-\widehat{G{T}_i}{)}^2.$

Post Processing

The raw output of the CNN is blurry and has visible noise, which might be caused by inaccurate pixel-wise estimations. The subsampling property of the pooling layers combined with the correlation property of the convolution layers amplify this effect.[[4]] So post-processing is necessary to recover the lost details.

In the postprocessing step, the raw CNN output would go through guided filter[5], guided by high frequency details from the input IR image to produce the final colorized image. In the result section, you could see that compared to the raw output Eμ, object contours and edges are clearly visible.

Results

L3

L3 model result

Main result for our L3 model (22 images for training set, 4 images for testing set):

Category	Data Amount	DeltaE
Training+Testing	22+4	7.1602
Test: Scenery	4	15.7399
Test: Male	10	5.9912
Test: Female	4	5.1804
Test: Fruit	8	7.1626

CNN

CNN model result

To compare our result with the ground truth RGB images, we used CIELAB ΔE metric to measure difference in human perception level and MSE metric to quantitatively measure pixels differences across image. Moreover, we also plot the distribution of true R and predict R to measure how close we are to the ground truth image. The training process lasted 1000 epoches in Nvdia GPU, using CUDA Toolkit 9.0 [6]. The following images gives the results for both train and test set colorization.

(1) Train set results

     MSE = 0.119

     CIELAB ΔE = 2.3265

blue: Output RGB values v.s. Target RGB of Example Train image; orange: true line

(2) Test set result

   MSE = 0.0134

   CIELAB ΔE = 4.8342

blue: Output RGB values v.s. Target RGB of Example Train image; orange: true line

We could see that training set with smaller MSE has best performance compared to the test set in the following three aspects:

a) the predicted RGB values locates more concentrated around the true line b) the smaller CIELAB ΔE, which indicates smaller visual difference c) visually more colorful

Also, in the test set, the first two images has better performance than the third image with contain artificial colorful objects. Since the training mainly contains neural colorful images. That might be result from the different color distribution of test set, so the trained model lose the inference ability when it encounter a brand new IR image.

Guided Filter result

One more thing to notice is that the raw output image of CNN is burry, it seems that we obtain color at the expense of resolution. So we put the results into an edge enhance filter. Guided filter main two parameters, a) local window size r and b) epsilon. Local window determined the smooth region of a local filter and epsilon indicates the denoising ability of the filter. In the following experiment, we try 3 three common window size and tuned the epsilon values. The experiment results is given below, we found that when r = 5, and epsilon = 1e-5, the edge are sharper and more details are saved.

Guide Filter

CNN Final result

In this section, we give the final results of the integrated approach.

(1) Train set

(2) Test set

Conclusions

After 1000 epoch iteration, MSE keep getting smaller, the model has not face any overfitting problem.

Reference

[1] https://github.com/ISET/isetL3

Appendix I

Github Page for this project: [7]

Appendix II

You can write math equations as follows: $y=x+5$

You can include images as follows (you will need to upload the image first using the toolbox on the left bar, using the "Upload file" link).

Example NIR image of Urban Building ]

Example NIR image of Urban Building

2016 Summer Olympics bidding results<ref>Template:Cite web</ref> File:Bids for the 2016 Summer Olympics.svg
City	NOC	Round 1	Round 2	Round 3
Rio de Janeiro	Template:BRA	26	46	66
Madrid	Template:ESP	28	29	32
Tokyo	Template:JPN	22	20	—
Chicago	Template:USA	18	—	—