Cnnprediction

Methods

A voltage data was used

Dataset and Software Packages

• A ~2700 subset of the COCO 2017 Train images was collected which included a variety of objects in various settings under different lighting scenarios, which would otherwise reflect the wide range of scenes that may be be captured by a camera. In addition, a few images of various facial profiles were derived from The Image System Engineering Toolbox for Biology (ISETBIO). For the neural network training, the sensor data was split into 2/3 for training and 1/3 for testing respectively.

• Keras, [(https://www.tensorflow.org/ TensorFlow], and Scikit-learn were used to construct, train, visualize, and test the convolutional neural network. NumPy was used for data manipulation.

Generating Sensor Data

• These images were then processed as scenes using ISETCAM to produce a sensor response for both a 100 x 120 and a 202 x 242 sensor represented in voltages, which were used as the input and the target respectively. The two extra pixels in each dimension represent the fact that when the pixel size is halved, ISETCAM produced half a Bayer tile at the edges of the sensor response.

Future Work

Developing a custom loss function which equalizes the weighting of each RGB value - since Bayer tiles produce two green values for every red and blue value, green is overrepresented in the loss function which tends for neural network to produce a greenish hue. Alternatively an approximation of deltaE loss function could be used to force the weights to be sensitive to color differentiation.

References

[1] https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf

[2] https://www.cv-foundation.org/openaccess/content_cvpr_2015/html/Long_Fully_Convolutional_Networks_2015_CVPR_paper.html

[3] http://personal.ie.cuhk.edu.hk/~ccloy/files/eccv_2014_deepresolution.pdf

[4] https://ftp.cs.nyu.edu/~fergus/papers/matt_cvpr10.pdf

[5] https://arxiv.org/abs/1707.05425