Cnnprediction
Introduction
Background
Methods
We attempt to predict how a given sensor with a Bayer filter responds to a given scene as if the width of the pixels were reduced by half - without performing a full simulation. Two linear transformations, inspired by demosaicking techniques, are used to predict the upsampled sensor response to establish baselines, representing a naive approach. In addition, we attempt to apply state-of-the-art neural network architectures, which have demonstrated success at a variety of image processing tasks, including image upsampling. We then compare the quality of the predicted upsampled sensor voltage data using several metrics.
Dataset and Software Packages
A ~2700 subset of the COCO 2017 Train images was collected which included a variety of objects in various settings under different lighting scenarios, which would otherwise reflect the wide range of scenes that may be be captured by a camera. In addition, a few images of various facial profiles were derived from The Image System Engineering Toolbox for Biology (ISETBIO). For the neural network training, the sensor data was split into 2/3 for training and 1/3 for testing respectively.
Keras, [(https://www.tensorflow.org/ TensorFlow], and Scikit-learn were used to construct, train, visualize, and test the convolutional neural network. NumPy was used for data manipulation.
Generating Sensor Data
These images were then processed as scenes using ISETCAM to produce a sensor response for both a 100 x 120 and a 202 x 242 sensor represented in voltages, which were used as the input and the target respectively. The two extra pixels in each dimension represent the fact that when the pixel size is halved, ISETCAM produced half a Bayer tile at the edges of the sensor response.
Upsampling Algorithms
A naive technique used for predicting upsampled sensor voltage data, the nearest neighbor algorithm simply takes the prior sensor response for a 4x4 "tile" and copies the values towards the right, beneath, and diagonally towards the bottom-right to double the size of the sensor response data. Nearest neighbor simply assumes that the additional pixels in the more dense sensor would have captured the same information regardless (which is obviously untrue). We use the predicted values of the nearest neighbor algorithm as a performance baseline.
Bilinear Interpolation
A slightly less naive technique, bilinear interpolation is inspired by the basic demosaicking algorithm of similar name. This algorithm averages the sensor voltage values of adjacent prior 4x4 tiles to produce new voltage values. Adjacency of a prior tile relative to a predicted tile is determined by whether the prior tiles is immediately present in the vertical, horizontal, or diagonal directions with a distance of 1. Tiles which are predicted at the edge of the sensor data will consist of the average of prior tiles which exist.
Convolutional Neural Networks
Inspired by state-of-the-art advances in image processing, we develop a convolution neural network architecture that is believed to represent being close to the extent to which a given sensor can be upsampled given no further information. Popularized in 2012 by Krizhevksy et. al [1], convolutional neural networks have been successfully applied to many other image processing tasks, including semantic segmentation [2], object detection, and image upsampling [3]. In contrast to traditional neural networks, convolutional neural networks are especially suited towards image processing tasks due to lower number of weights to train (a naive 1-layer neural network for the task here would require the number of weights to train to be in the number of the hundreds of millions, while a single 3x3 kernel would only result in 9, minus any bias), as well as the validated assumption that the information required to predict the upsampled sensor data can be derived locally.
Deconvolutional Neural Networks
The original introduction of deconvolutional neural networks was intended to extract additional low and mid-range image representations beyond edge primitives - in effect an input is mapped to increasingly sparser reprsentations which can form the elements of an image [4]. We leverage this concept in our neural network design to naturally increases the size of an input using kernels, illustrated below:
Architecture
Since the popularization of convolutional neural networks, a number of architectures have been published for image super-resolution (or upsampling) tasks [3][5]. While we can leverage the more advanced neural network concepts such as network-in-network, skip connections, simply adding a large number of layers, prefer to pursue a more simple architecture to focus on developing other parts of the project. The model parameters used are described below, with the somewhat novel introduction of a deconvolutional neural network layer.
| layer type | kernel size | filters | padding | activation |
|---|---|---|---|---|
| deconvolution | 8x8 | 32 | - | ReLU |
| 2D convolution | 3x3 | 32 | same | ReLU |
| 2D convolution | 5x5 | 1 | same | linear |
The selection of the 8x8 kernel size for the initial deconvolution layer was chosen due to a capture of information 4 complete prior 4x4 Bayer tiles, much in the same way bilinear interpolation would.
Loss function
Since the input and output of the neural network were sensor voltage values, mean squared error was calculated between the known high resolution sensor voltage data generated by ISETCAM and the output generated by the neural network.
While numerous loss functions exist, mean squared error was chosen due to its simplicity in understanding and ease of calculation. We must acknowledge however mean squared error does not factor in any elements of how a human would compare two images, which include sharpness, noise, dynamic range, color difference, illumination, etc which we will illustrate in the results section.
Training Parameters
The neural network was trained with the following parameters, which is due to the simple result:
- Adam optimizer with a learning rate set to 0.03
- 40 epochs with early stopping (set to 0.001 with a patience of 1), with a batch size of 16
- no validation set used
Postprocessing
The upsampled sensor data was then transformed into PNG images using the ISETCAM function ipCompute with the following parameters:
- bilinear demosaicing
- conversion to XYZ representation
- no illuminant correction
Results
| Algorithm | MSE | Delta E 2000 (S-CIELAB) |
|---|---|---|
| Nearest Neighbor | 4.618 x 10^-3 | 4.0266 |
| Bilinear Interpolation | 3.518 x 10^-3 | 5.3534 |
| CNN | 0.5970 x 10^-3 | 3.7440 |
Failure Cases
Conclusions
Future Work
Developing a custom loss function which equalizes the weighting of each RGB value - since Bayer tiles produce two green values for every red and blue value, green is overrepresented in the loss function which tends for neural network to produce a greenish hue. Alternatively an approximation of deltaE loss function could be used to force the weights to be sensitive to color differentiation.
Loss functions could involve an additional processing step generating an array of XYZ values from the predicted voltage values, to which
Appendix
References
[3] http://personal.ie.cuhk.edu.hk/~ccloy/files/eccv_2014_deepresolution.pdf