Cnnprediction
Introduction
Predicting sensor response can be expensive
Machine learning can approximate sensor response with different parameters given a baseline sensor response
Primary problem: predicting sensor response when pixel size are halved (or resolution is doubled)
Background
Methods
We attempt to predict how a given sensor with a Bayer filter responds to a given scene as if the width of the pixels were reduced by half - without performing a full simulation. Two linear transformations, inspired by demosaicking techniques, are used to predict the upsampled sensor response to establish baselines, representing a naive approach. In addition, we attempt to apply state-of-the-art neural network architectures, which have demonstrated success at a variety of image processing tasks, including image upsampling. We then compare the quality of the predicted upsampled sensor voltage data using several metrics.
Dataset and Software Packages
A ~2700 subset of the COCO 2017 Train images was collected which included a variety of objects in various settings under different lighting scenarios, which would otherwise reflect the wide range of scenes that may be be captured by a camera. In addition, a few images of various facial profiles were derived from The Image System Engineering Toolbox for Biology (ISETBIO). For the neural network training, the sensor data was split into 2/3 for training and 1/3 for testing respectively.
Keras, TensorFlow, and Scikit-learn were used to construct, train, visualize, and test the convolutional neural network. NumPy was used for data manipulation.
Generating Sensor Data
These images were then processed as scenes using ISETCAM to produce a sensor response for both a 100 x 120 and a 202 x 242 sensor represented in voltages, which were used as the input and the target respectively. The two extra pixels in each dimension represent the fact that when the pixel size is halved, ISETCAM produced half a Bayer tile at the edges of the sensor response.
Upsampling Algorithms
A naive technique used for predicting upsampled sensor voltage data, the nearest neighbor algorithm simply takes the prior sensor response for a 4x4 "tile" and copies the values towards the right, beneath, and diagonally towards the bottom-right to double the size of the sensor response data. Nearest neighbor simply assumes that the additional pixels in the more dense sensor would have captured the same information regardless (which is obviously untrue). We use the predicted values of the nearest neighbor algorithm as a performance baseline.
Bilinear Interpolation
A slightly less naive technique, bilinear interpolation is inspired by the basic demosaicking algorithm of similar name. This algorithm averages the sensor voltage values of adjacent prior 4x4 tiles to produce new voltage values. Adjacency of a prior tile relative to a predicted tile is determined by whether the prior tiles is immediately present in the vertical, horizontal, or diagonal directions with a distance of 1. Tiles which are predicted at the edge of the sensor data will consist of the average of prior tiles which exist.
Convolutional Neural Networks
Inspired by state-of-the-art advances in image processing, we develop a convolution neural network architecture that is believed to represent being close to the extent to which a given sensor can be upsampled given no further information. Popularized in 2012 by Krizhevksy et. al [1], convolutional neural networks have been successfully applied to many other image processing tasks, including semantic segmentation [2], object detection, and image upsampling [3]. In contrast to traditional neural networks, convolutional neural networks are especially suited towards image processing tasks due to lower number of weights to train (a naive 1-layer neural network for the task here would require the number of weights to train to be in the number of the hundreds of millions, while a single 3x3 kernel would only result in 9, minus any bias), as well as the validated assumption that the information required to predict the upsampled sensor data can be derived locally.
Deconvolutional Neural Networks
The original introduction of deconvolutional neural networks was intended to extract additional low and mid-range image representations beyond edge primitives - in effect an input is mapped to increasingly sparser reprsentations which can form the elements of an image [4]. We leverage this concept in our neural network design to naturally increases the size of an input using kernels, illustrated below:
Architecture
Since the popularization of convolutional neural networks, a number of architectures have been published for image super-resolution (or upsampling) tasks [3][5]. While we can leverage the more advanced neural network concepts such as network-in-network, skip connections, simply adding a large number of layers, prefer to pursue a more simple architecture to focus on developing other parts of the project. The model parameters used are described below, with the somewhat novel introduction of a deconvolutional neural network layer.
| layer type | kernel size | filters | padding | activation | params |
|---|---|---|---|---|---|
| deconvolution | 8x8 | 32 | - | ReLU | 2080 |
| 2D convolution | 3x3 | 16 | same | ReLU | 12816 |
| 2D convolution | 5x5 | 1 | same | linear | 401 |
The selection of the 8x8 kernel size for the initial deconvolution layer was chosen due to a capture of information 4 complete prior 4x4 Bayer tiles, similar to the bilinear interpolation algorithm. Also notably there are only about 15,300 trainable params, indicating that the neural network is relatively easy to train.
Loss function
Since the input and output of the neural network were sensor voltage values, mean squared error was calculated between the known high resolution sensor voltage data generated by ISETCAM and the output generated by the neural network.
While numerous loss functions exist, mean squared error was chosen due to its simplicity in understanding and ease of calculation. We must acknowledge however mean squared error does not factor in any elements of how a human would compare two images, which include sharpness, noise, dynamic range, color difference, illumination, etc. of which some will be discussed in the results section.
Training Parameters
The neural network was trained with the following parameters, which is due to the simple result:
- Adam optimizer with a learning rate set to 0.03
- 40 epochs with early stopping (set to 0.001 with a patience of 1), with a batch size of 16
- no validation set used
Postprocessing
The upsampled sensor data was then transformed into PNG images using the ISETCAM function ipCompute with the following parameters:
- bilinear demosaicing
- conversion to XYZ representation
- no illuminant correction
From the image production, we are able to provide further analysis between the images generated from the simulated high resolution sensor voltage data, and those predicted by the algorithms.
Analysis
OpenCV was used to perform edge detection and to generate color histograms. S-CIELAB was used to generate delta E calculations between the predicted and reference values.
Results
The results of the predicted values of nearest neighbor and bilinear interpolation were rather unsurprising. Qualitatively the nearest neighbor algorithm produced a large number of false color artifacts, introduced unnatural edges, and "pixelated" the image. Bilinear interpolation produced similar color artifacts but to lesser degree - however due to the naive averaging of priors, edges are drastically reduced as they become smoothed over. In contrast, despite the convolutional neural network being shallow, the convolutional neural network performed remarkably well. Moreover, the sharpness of the image was close to that of the simulated sensor values, with few obvious false color artifacts and edges that appear to be natural.
Quantitatively, we can assess the quality of the predicted sensor values using a few metric, including mean squared error and delta E 2000.
| Algorithm | MSE | Delta E 2000 (S-CIELAB) |
|---|---|---|
| Nearest Neighbor | 4.618 x 10^-3 | 4.0266 |
| Bilinear Interpolation | 3.518 x 10^-3 | 5.3534 |
| CNN | 0.5970 x 10^-3 | 3.7440 |
Unsurprisingly, the mean squared error for sensor values produced by the CNN was an order of magnitude lower than that of bilinear interpolation and the nearest network, given that the MSE was optimized for as the loss function for training. However, the low mean squared error did not necessarily translate to good color reproduction - we see from the delta E 2000 values (generated from S-CIELAB) that the CNN did not perform appreciably better nearest network despite the images produced by CNN to be qualitatively far superior. We can detect this by observing a green overlay on some of the images, which is hypothesized to be the result of the frequency of green pixel values in the sensor data - that is the mean squared error loss function naively emphasizes the error seen in green pixel values over red or blue values by a factor of 2. Moreover, since coco presumes to capture our everyday environment, we can demonstrate that the dominant color is green, biasing our neural network to produce green hues.
Sample Output
Below we can see the sample outputs from the algorithms that we developed.
Other examples can be found here, ordering is the same as above:
Kernel Visualization
The visualization of the kernels allows us to qualitatively observe what appear to the most pertinent features that may be derived from sensor voltage data to produce higher resolution data. In other words, the kernels represent sparse elemental representations of the voltage data that may be reconstituted to form an enlarged representation. Output from hidden layer reveals that the convolution kernels consume local information in geometrically grid like pattern. Moreover the individual kernels appear to extract information about blue, red, and green values as well as lightness.
Other examples:
Failure Cases
While the convolution neural network performed fairly well, there were a peculiar phenomenon where strong blue tones disappeared from the image and would result in a completely different color:
The thesis around this failure can be due to the infrequency of strong blue tones in the dataset, however red hues should be similarly affected.
Conclusions
Convolutional neural networks can perform remarkably well at image processing tasks - despite the number of trainable parameters being relatively low, the training unoptimized, and the architecture being quite simple (i.e no network-in-network layers, dropout, etc.) the CNN developed in this project upscaled sensor voltage data which, after post-processing, produced images that had low numbers of artifacts and visually pleasing.
Future Work
Developing a custom loss function which equalizes the weighting of each RGB value - since Bayer tiles produce two green values for every red and blue value, green is overrepresented in the loss function which tends for neural network to produce a greenish hue. Alternatively a Taylor approximation of deltaE loss function could be used to force the weights to be more sensitive to color differentiation using the XYZ information generated by ISETCAM. Moreover, we may also seek to add additional quantitative metrics to objectively determine the quality of prediction.
In addition, other non-neural network based algorithms should be used to predict an upsampled version of sensor data since nearest neighbor and bilinear interpolation have known faults that are obvious even prior to implementation - a more fair comparison with more complex upsampling approaches would better bolster (or undermine) the argument that convolutional neural networks are meaningfully more performant than other algorithms.
Appendix
Github: https://github.com/gnedster/psych221
References
[3] http://personal.ie.cuhk.edu.hk/~ccloy/files/eccv_2014_deepresolution.pdf