Irtorgb

Introduction

In this project, we applied several machine learning algorithms like L3 (Local, Linear and Learned) and a neural network based model to find the appropriate mapping from NIR to RGB visible spectrum which human eyes are more sensitive to. We evaluate the results based on the CIELAB delta E metric for measuring difference in human perception level and the RMSE metric for quantitatively measuring pixels differences across image.

Background

Near-Infrared (NIR) images have broad application in remote sensing and surveillance for its capacity to segment images according to object’s material. Although NIR images made object detection an easier task, its monochrome nature is conflicted with human visual perception and thus might not be user friendly. Lack of color discrimination or wrong color representation on NIR images would limit people’s understanding towards the image or even lead to wrong judgement. So colorizing the NIR grayscales images would be desired.

Colorization of NIR images is a difficult and challenging task since a single channel is mapped into a three dimensional space without interchannel correlation, which greatly reduces the effect of traditional color correction/transfer method. Moreover, since surface reflection in the NIR spectrum band is material dependent, some object might be missing from the NIR scenes due to their transparency to NIR. Therefore, IR colorization requires estimating not only chromiance, but also illuminance, which add a lot complexity to the problem.

Method

CNN Method

Due to the great performance of CNN models in image classification tasks [1], we propose an integrated approach based on deep-learning techniques to perform a spectral transfer of NIR to RGB images. As inspired by Matthias el [2], a Convolutional Neural Network (CNN) is applied to directly estimate the RGB representation from a normalized NIR image. Then the colorized raw output of CNN model would go through an edge enhancement filter to transfer details from the high resolution IR input image and obtain better image quality.

Dataset

A deep CNN model might have thousands or even millions of trainable variables, so a large amount of data is required to prevent overfitting, where the model perfectly fits the training data set but loses the ability to inference the spectrum transfer on a new IR image. We use the RGB-NIR Scene Dataset [3] which consists of 477 images in 9 categories captured in RGB and Near-infrared (NIR). Based on promising result of single category training in L3, we only used one category data that consists of 102 high resolution Urban Building images in our CNN model. All the images were cropped into a 64 * 64 patches and then split into 4/5 training data and 1/5 test data.

\\ image + table

CNN Model Description

A recent trend for CNNs is the usage of many convo- lution layers with small convolution kernels and relatively few pooling layers. This increases the total amount of non- linearities in the network. The model consists of nc convolution layers and np max-pooling layers. The pooling layers are distributed between the convolution layers so that each convolution layer block has the same amount of convolution layers. The activation function of the convolution layers is the ReLU function: ReLU(x) = max(0,x). Batch normalization is added after each relu.

Stochastic gradient descent using the backpropagation algorithm [30] is performed to minimize the mean squared error (MSE) between the pixel estimates F (p′i , Θ) of the ith normalized pixel p′i and the corresponding pixels qi of the mean filtered RGB image Tμ:

Loss Function

Since the input and output of the neural network were sensor voltage values, mean squared error was calculated between the known high resolution sensor voltage data generated by ISETCAM and the output generated by the neural network, which can be defined as:

$\mathrm{MSE}=\frac{1}{n}\sum _{i=1}^n(Y_i-\widehat{Y_i}{)}^2.$

While numerous loss functions exist, mean squared error was chosen due to its simplicity in understanding and ease of calculation. We must acknowledge however mean squared error does not factor in any elements of how a human would compare two images, which include sharpness, noise, dynamic range, color difference, illumination, etc. of which some will be discussed in the results section.

Post Processing

The raw output of the inference step Eμ shows visible noise, caused by inaccurate pixel-wise estimations. The subsampling property of the pooling layers combined with the correlation property of the convolution layers amplify this effect. For a network, which contains three 2 × 2 pooling layers, the coherence gap is sπ = 23 = 8 in both image dimensions. Postprocessing is necessary to remove this incoherence and recover the lost details. [reference]

In the postprocessing step, the filtered image is augmented by the high frequency details Ih from the input image to produce the final colorized image. In the result section, you could see that compared to the raw output Eμ, object contours and edges are clearly visible, but the textures of the surfaces are still missing. By using the detail component of the input image Ih, textures and resolution can be partially recovered.

Appendix II

You can write math equations as follows: $y=x+5$

You can include images as follows (you will need to upload the image first using the toolbox on the left bar, using the "Upload file" link).

2016 Summer Olympics bidding results<ref>Template:Cite web</ref> File:Bids for the 2016 Summer Olympics.svg
City	NOC	Round 1	Round 2	Round 3
Rio de Janeiro	Template:BRA	26	46	66
Madrid	Template:ESP	28	29	32
Tokyo	Template:JPN	22	20	—
Chicago	Template:USA	18	—	—