Irtorgb
Introduction
Near-Infrared (NIR) images have broad application in remote sensing and surveillance for its capacity to segment images according to object’s material. Although NIR images made object detection an easier task, its monochrome nature is conflicted with human visual perception and thus might not be user friendly. Lack of color discrimination or wrong colors on NIR images would limit people’s understanding and even lead to wrong judgement. So colorizing the grayscale NIR images would be desired.
Colorization of NIR images is a difficult and challenging task since a single channel is mapped into a three dimensional space with unknown interchannel correlation, which greatly reduces the effectiveness of using traditional color correction/transfer method to solve this problem. Moreover, since surface reflection in the NIR spectrum band is material dependent, some objects might be missing from the NIR scenes due to their transparency to NIR. Therefore, different from grayscale image colorization which only estimates chrominance, IR colorization requires estimating not only the chrominance, but also the illuminance, which add a lot complexity to the problem.
Traditional colorization method extracts color distribution from input image and then fit it into the color distribution of the output image. Typical method[1-2] involves segmenting image into smaller parts that receives the same color, then retrieve each color palette to estimate responding chromince. Recent approaches[3-5] leverage the use of deep neural networks to enable colorization automatically. Some[3] train from scratch, using the neural network to estimate the chrominance values from monochrome images, other methods[4,5] include using a pre-trained model as a starting point, and then apply transfer learning to adapt the current model for their own colorization tasks. All these work shows that deep learning techniques provide promising solutions for automatic colorization.
In this project, we proposed several machine learning solutions like L3 (Local, Linear and Learned) and a neural network based model to find the appropriate mapping from NIR to RGB visible spectrum representation which human eyes are more sensitive to. The results are evaluated by CIELAB ∆E and MSE.
Method
L3 Method
The L3 (Local Linear Learned) method[6] combines machine learning and image systems simulation that automates the pipeline design. It comprises two main steps: rendering and learning. The rendering step adaptively selects from a stored table of affine transformations to convert the raw IR camera sensor data into RGB images, and the training step learns and stores the transformations used in rendering.
Dataset
Our input dataset are the original scene NIR images with 6 fields (mcCOEF, basis, comment, illuminant, fov, dist). We started with 26 such images. With some adaptation from the script in L3(https://github.com/ISET/isetL3), we first created a RGB sensor, which generates the corresponding IR sensor with a irPassFilter. Then these two sensors read in the NIR images, along with the padded spectral radiance optical image computed from those spectral irradiance NIR images (to allow for light spreading from the edge of the scene), and output from sensorCompute(irSensor/rgbSensor, oi) the sensor volts(electrons) at each pixel from the optical image. Finally we used an ipCompute on these sensor volts images for the final sensor data images after demosaicing, sensor color conversion, and illuminant correction.
Note that our output images for training shows a reddish effect on the RGB images. We couldn't find the right way to get the normally colored RGB images and IR images with the same size - saving the images directly from the optical images gives the right hue on a complete size image (606*481), however the corresponding IR images cannot be saved this way. So for the purpose of the L3 training we decided to keep our original output from the image processing pipeline(ipCompute), which includes IR and RGB images of size 198*242, with the RGB images showing a reddish effect.
 |
 |
 |
Model and Attempted Improvements
We did 4 experiments on our L3 model: (1) The general L3 model: training all images from the dataset, with the exception of 4 images each from one different category as the testing images. (2) Single category training: training the 4 categories from the dataset (fruit, female, male, scenery) separately. (3) Single channel training: training the RGB channels with IR images separately. (4) RGB to IR reverse training: "decolorization" of images to get a sense of the difficulty on the mapping of both directions.
Results are demonstrated in the 'Results' section below.
CNN Method
Due to the great performance of CNN models in image processing tasks [1], we proposed an integrated approach based on deep-learning techniques to perform a spectral transfer of NIR to RGB images. As inspired by Matthias et al. [2], a Convolutional Neural Network (CNN) is applied to directly estimate the RGB representation from a normalized NIR image. Then, to obtain better image quality, the colorized raw outputs of CNN would go through an edge enhancement filter to transfer details from the high resolution input IR images.
Dataset
For the complexity of spectrum mapping, a deep CNN model with thousands or even millions of trainable variables would be desired. Thus, a large amount of data is required to prevent overfitting, where the model perfectly fits the training data set but loses the ability to inference on a new IR image. For CNN model, We use the RGB-NIR Scene Dataset [3] which consists of 477 images in 9 categories captured in both RGB and Near-infrared (NIR) sensor.
Due to the comparatively positive result of single category training in L3, we only used Urban Building dataset that consists of 102 high resolution RGB/NIR images in the CNN model. The dataset was split into 80% training data and 20% test data, then all the images were cropped into 64 × 64 patches to feed into the model.
 |
 |
| Number of 64 × 64 Patches used in CNN Model | ||
|---|---|---|
| Data Type/ Data Use | Train | Test |
| Input IR (64 × 64) | 7572 | 1044 |
| Output RGB (64 × 64 × 3) | 7572 | 1044 |
CNN Architecture
Illustration of our CNN network architecture is given in Figure X. We use many convolution/deconvolution layers and relatively few pooling layers to increases the total amount of non-linearities in the network, which would help us learn complex mapping from IR to RGB. The activation function of each convolution layers is the ReLu function:
, batch normalization is followed afterwards to avoid overfitting.
In the training process, we use stochastic gradient descent to minimize the mean squared error (MSE) between the pixel estimates of normalized output (oRGB) and normalized ground truth RGB images(GT). MSE is defined as follows:
Post Processing
The raw output of the CNN is blurry and has visible noise, which might be caused by inaccurate pixel-wise estimations. The subsampling property of the pooling layers combined with the correlation property of the convolution layers amplify this effect.[[4]] So post-processing is necessary to recover the lost details.
In the postprocessing step, the raw CNN output would go through guided filter[5], guided by high frequency details from the input IR image to produce the final colorized image. In the result section, you could see that compared to the raw output Eμ, object contours and edges are clearly visible.
Results
L3
L3 model result
Main result for our L3 model (22 images for training set, 4 images for testing set):
| Category | Data Amount | DeltaE |
|---|---|---|
| Training+Testing | 22+4 | 7.1602 |
| Test: Scenery | 4 | 15.7399 |
| Test: Male | 10 | 5.9912 |
| Test: Female | 4 | 5.1804 |
| Test: Fruit | 8 | 7.1626 |
 |
 |
Attempt 1 : Single Category Training
| Category | Data Amount | DeltaE - Training | DeltaE - Testing |
|---|---|---|---|
| Test: Scenery | 4 | 5.8738 | 5.2851 |
| Test: Male | 10 | 5.1895 | 4.8235 |
| Test: Female | 4 | 2.8871 | 8.0136 |
| Test: Fruit | 8 | 5.7514 | 3.9328 |
 |
Attempt 2 : Single Color Channel Training
| Category | Data Amount | DeltaE - Single Channel | DeltaE - Original |
|---|---|---|---|
| Training | 22 | 7.1603 | |
| Test: Scenery | 4 | 15.7399 | 5.2851 |
| Test: Male | 10 | 5.9901 | |
| Test: Female | 4 | 5.1816 | |
| Test: Fruit | 8 | 7.1626 | |
Attempt 3 : RGB to IR Reverse Training
| Category | mse - RGB to IR | mse - IR to RGB |
|---|---|---|
| Training | 0.05556 | 0.07572 |
| Test: Scenery | 0.04430 | 0.14670 |
| Test: Male | 0.06464 | 0.04639 |
| Test: Female | 0.03845 | 0.06328 |
| Test: Fruit | 0.04301 | 0.06706 |
CNN
CNN model result
To compare our result with the ground truth RGB images, we used CIELAB ΔE metric to measure difference in human perception level and MSE metric to quantitatively measure pixels differences across image. Moreover, we also plot the distribution of true R and predict R to measure how close we are to the ground truth image. The training process lasted 1000 epoches in Nvdia GPU, using CUDA Toolkit 9.0 [6]. The following images gives the results for both train and test set colorization.
(1) Train set results
MSE = 0.119
CIELAB ΔE = 2.3265
 |
|
(2) Test set result
MSE = 0.0134
CIELAB ΔE = 4.8342
 |
|
We could see that training set with smaller MSE has best performance compared to the test set in the following three aspects:
a) the predicted RGB values locates more concentrated around the true line b) the smaller CIELAB ΔE, which indicates smaller visual difference c) visually more colorful
Also, in the test set, the first two images has better performance than the third image with contain artificial colorful objects. Since the training mainly contains neural colorful images. That might be result from the different color distribution of test set, so the trained model lose the inference ability when it encounter a brand new IR image.
Guided Filter result
One more thing to notice is that the raw output image of CNN is burry, it seems that we obtain color at the expense of resolution. So we put the results into an edge enhance filter. Guided filter main two parameters, a) local window size r and b) epsilon. Local window determined the smooth region of a local filter and epsilon indicates the denoising ability of the filter. In the following experiment, we try 3 three common window size and tuned the epsilon values. The experiment results is given below, we found that when r = 5, and epsilon = 1e-5, the edge are sharper and more details are saved.
CNN Final result
In this section, we give the final results of the integrated approach.
(1) Train set
(2) Test set
Conclusions
After 1000 epoch iteration, MSE keep getting smaller, the model has not face any overfitting problem.
Reference
[1] A. Levin, D. Lischinski, and Y. Weiss, “Colorization using optimization,” ACM Transactions on Graphics, vol. 23, no. 3, pp. 689–694, 2004.
[2] L. Yatziv and G. Sapiro, “Fast image and video colorization using chrominance blending,” IEEE Transactions on Image Processing, vol. 15, no. 5, pp. 1120–1129, 2006.
[3] S. Iizuka, E. Simo-Serra, and H. Ishikawa, “Let there be color!: Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification,” Proceedings of ACM SIGGRAPH, vol. 35, no. 4, 2016.
[4] G. Larsson, M. Maire, and G. Shakhnarovich, “Learning representations for automatic colorization,” Tech. Rep. arXiv:1603.06668, 2016.
[5] Limmer, M., Lensch, H.P.A.: Infrared colorization using deep convolutional neural networks. In: ICMLA 2016, Anaheim, CA, USA, 18–20 December 2016, pp. 61–68, 2016.
[6] isetL3, Github source code: https://github.com/ISET/isetL3.
Appendix I
Github Page for this project: [7]
Appendix II
Todo: conclusion + future work + contribution
Todo: polish words
Todo: image label size arrangement
Todo: code organization
Todo: Reference rrorganize
Todo: email professor