Cnnprediction - Revision history

imported>Student2018: /* Methods */

2018-12-15T17:58:13Z

Methods

← Older revision		Revision as of 17:58, 15 December 2018
Line 10:		Line 10:

	== Methods ==		== Methods ==
	We attempt to predict how a given sensor with a Bayer color filter array responds to a given scene as if the width of the pixels were reduced by half without performing a full simulation. Two linear transformations, inspired by upsampling techniques, are used to predict the upsampled sensor response to establish baselines as naive approaches. In addition, we attempt to apply state-of-the-art neural network architectures which have demonstrated success at a variety of image processing tasks, including image upsampling. We then compare the quality of the predicted upsampled sensor voltage data to the target sensor voltage using mean squared error, delta E 2000, color histograms, and edge detection.		We attempt to predict how a given sensor with a Bayer color filter array responds to a given scene as if the width of the pixels were reduced by half without performing a full simulation. Two linear transformations, inspired by upsampling techniques, are used to predict the upsampled sensor response to establish baselines as naive approaches. In addition, we attempt to apply state-of-the-art neural network architectures which have demonstrated success at a variety of image processing tasks, including image upsampling. We then compare the quality of the predicted upsampled sensor voltage data to the target sensor voltage using mean squared error, delta E 2000, color histograms, and edge detection.

	=== Dataset and Software Packages ===		=== Dataset and Software Packages ===
Line 97:		Line 97:

	=== Analysis ===		=== Analysis ===
	[https://docs.opencv.org OpenCV] was used to perform edge detection and to generate color histograms ~~and~~ S-CIELAB was used to generate delta E 2000 calculations between the predicted and target on the generated images.		[https://docs.opencv.org OpenCV]'s canny edge detection algorithm was used to perform edge detection and to generate color histograms. ISETCAM's S-CIELAB implementation was used to generate delta E 2000 calculations between the predicted and target on the generated images.

	== Results ==		== Results ==

imported>Student2018: /* Conclusions */

2018-12-15T17:47:50Z

Conclusions

← Older revision		Revision as of 17:47, 15 December 2018
Line 164:		Line 164:
	== Conclusions ==		== Conclusions ==

	Convolutional neural networks can perform remarkably well at image processing tasks - despite the number of trainable parameters being relatively low, the training unoptimized, and the architecture being quite simple (i.e no network-in-network layers, dropout, etc.) the CNN developed in this project upscaled sensor voltage data which, after post-processing, produced images that had low numbers of artifacts and visually pleasing. We suggest that convolutional neural networks, with sufficient hyperparameter tuning and training parameters may prove to sufficiently predict sensor responses of various topologies given a prior sensor data response. Additional investment into tuning the hyperparameters of the neural network, or with the introduction of novel architectures may prove to be useful to eliminate some of the aberrations observed in the predicted response.		Convolutional neural networks can perform remarkably well at image processing tasks - despite the number of trainable parameters being relatively low, the training unoptimized, and the architecture being quite simple (i.e no network-in-network layers, dropout, etc.) the CNN developed in this project upscaled sensor voltage data which, after post-processing, produced images that had low numbers of artifacts and were visually pleasing. We suggest that convolutional neural networks, with sufficient hyperparameter tuning and training parameters may prove to sufficiently predict sensor responses of various topologies given a prior sensor data response. Additional investment into tuning the hyperparameters of the neural network, or with the introduction of novel architectures may prove to be useful to eliminate some of the aberrations observed in the predicted response.

	=== Future Work ===		=== Future Work ===

imported>Student2018: /* Generating Sensor Data */

2018-12-15T17:44:57Z

Generating Sensor Data

← Older revision		Revision as of 17:44, 15 December 2018
Line 18:		Line 18:

	=== Generating Sensor Data ===		=== Generating Sensor Data ===
	These collected images were then processed as scenes using [https://github.com/ISET/isetcam ISETCAM] to produce a sensor response for both a 100 x 120 and a 202 x 242 sensor represented in ~~voltages~~, which were used as the input and the target respectively. The two extra pixels in each dimension represent the fact that when the pixel size is halved, ISETCAM produced half a Bayer tile at the edges of the sensor response which we believed to be errata. ~~Regardless, this~~ sensor data was split into two-thirds for training and a third for testing respectively.		These collected images were then processed as scenes using [https://github.com/ISET/isetcam ISETCAM] to produce a sensor response for both a 100 x 120 and a 202 x 242 sensor represented in volts, which were used as the input and the target respectively. The two extra pixels in each dimension represent the fact that when the pixel size is halved, ISETCAM produced half a Bayer tile at the edges of the sensor response which we believed to be errata. This sensor data was split into two-thirds for training and a third for testing respectively.

	=== Upsampling Algorithms ===		=== Upsampling Algorithms ===

imported>Student2018: /* Conclusions */

2018-12-15T17:41:50Z

Conclusions

← Older revision		Revision as of 17:41, 15 December 2018
Line 164:		Line 164:
	== Conclusions ==		== Conclusions ==

	Convolutional neural networks can perform remarkably well at image processing tasks - despite the number of trainable parameters being relatively low, the training unoptimized, and the architecture being quite simple (i.e no network-in-network layers, dropout, etc.) the CNN developed in this project upscaled sensor voltage data which, after post-processing, produced images that had low numbers of artifacts and visually pleasing. We suggest that convolutional neural networks, with sufficient hyperparameter tuning and training parameters may prove to sufficiently predict sensor responses of various topologies given a prior sensor data response. Additional investment into tuning the hyperparameters of the neural network, or with the introduction of novel ~~archiectures~~ may prove to be useful to eliminate some of the aberrations observed in the predicted response.		Convolutional neural networks can perform remarkably well at image processing tasks - despite the number of trainable parameters being relatively low, the training unoptimized, and the architecture being quite simple (i.e no network-in-network layers, dropout, etc.) the CNN developed in this project upscaled sensor voltage data which, after post-processing, produced images that had low numbers of artifacts and visually pleasing. We suggest that convolutional neural networks, with sufficient hyperparameter tuning and training parameters may prove to sufficiently predict sensor responses of various topologies given a prior sensor data response. Additional investment into tuning the hyperparameters of the neural network, or with the introduction of novel architectures may prove to be useful to eliminate some of the aberrations observed in the predicted response.

	=== Future Work ===		=== Future Work ===
	Developing a custom loss function which equalizes the weighting of each RGB value (since Bayer tiles produce two green values for every red and blue value, green is overrepresented in the loss function) may prove to reduce the tendency for the neural network to produce scenes with a green hue. Moreover, if one were to implement a loss function which considered the mean delta E 2000 ~~of an~~ image, a Taylor approximation may be used and would likely result in significantly better color reproduction. Moreover, we may also seek to add additional quantitative metrics to objectively determine the quality of prediction, which may also be used as part of a composite loss function.		Developing a custom loss function which equalizes the weighting of each RGB value (since Bayer tiles produce two green values for every red and blue value, green is overrepresented in the loss function) may prove to reduce the tendency for the neural network to produce scenes with a green hue. Moreover, if one were to implement a loss function which considered the mean delta E 2000 between the target and predicted image, a Taylor approximation may be used to reduce processing time and would likely result in significantly better color reproduction. Moreover, we may also seek to add additional quantitative metrics to objectively determine the quality of prediction, which may also be used as part of a composite loss function to reduce specific aberrations.

	In addition, other non-neural network based algorithms should be used to predict an upsampled version of sensor data since nearest neighbor and bilinear interpolation are considered to be naive. A comparison with more complex upsampling approaches, such as bicubic interpolation, would better bolster (or undermine) the argument that convolutional neural networks are meaningfully more performant than other algorithms.		In addition, other non-neural network based algorithms should be used to predict the upsampled version of sensor data since nearest neighbor and bilinear interpolation are considered to be naive. A comparison with more complex upsampling approaches, such as bicubic interpolation, would better bolster (or undermine) the argument that convolutional neural networks are meaningfully more performant than other algorithms.

	== Appendix ==		== Appendix ==

imported>Student2018: /* Failure Cases */

2018-12-15T17:38:25Z

Failure Cases

← Older revision		Revision as of 17:38, 15 December 2018
Line 156:		Line 156:

	=== Failure Cases ===		=== Failure Cases ===
	While the convolution neural network performed fairly well, there ~~were~~ a peculiar phenomenon where strong blue tones disappeared from the image and would result in a completely different color:		While the convolution neural network performed fairly well, there was a peculiar phenomenon where strong blue tones disappeared from the image and would result in a completely different color:

	[[File:edwardn_failure_cases.png\|center\|frame\|The row above shows the images from the simulated sensor data while those below are produced by the convolutional neural network. Note the mistranslation of strong blue colors.]]		[[File:edwardn_failure_cases.png\|center\|frame\|The row above shows the images from the simulated sensor data while those below are produced by the convolutional neural network. Note the mistranslation of strong blue colors.]]

imported>Student2018: /* Kernel Visualization */

2018-12-15T17:38:05Z

Kernel Visualization

← Older revision		Revision as of 17:38, 15 December 2018
Line 148:		Line 148:
	=== Kernel Visualization ===		=== Kernel Visualization ===

	The visualization of the kernels allows us to qualitatively observe what appear to the most pertinent features that may be derived from sensor voltage data to produce higher resolution data. In other words, the kernels represent sparse elemental representations of the voltage data that may be reconstituted to form an enlarged representation. Output from the second hidden layer reveals that the convolution kernels consume local information in geometrically grid like pattern, with the striping indicating that some kernels appear to ignore or consume certain columnar or row patterns. This implies that the individual kernels naturally appear to extract information about ~~blue, red, and green~~ values ~~as well as lightness~~. We also note that some of the outputs of the second layer are sparse, which means that the number of filters may be reduced for more rapid training and the number of useful features may be limited.		The visualization of the kernels allows us to qualitatively observe what appear to the most pertinent features that may be derived from sensor voltage data to produce higher resolution data. In other words, the kernels represent sparse elemental representations of the voltage data that may be reconstituted to form an enlarged representation. Output from the second hidden layer reveals that the convolution kernels consume local information in geometrically grid like pattern, with the striping indicating that some kernels appear to ignore or consume certain columnar or row patterns. This perhaps implies that the individual kernels naturally appear to extract information about spatial frequency beyond the RGB voltage values. We also note that some of the outputs of the second layer are sparse, which means that the number of filters may be reduced for more rapid training and the number of useful features may be limited.

	[[File:Edwardn_kernal_output.png\|frame\|center\|200px\|The image above shows the output from the second hidden convolutional layer which consist of 16 kernels. Colors represent magnitude of the value only.]]		[[File:Edwardn_kernal_output.png\|frame\|center\|200px\|The image above shows the output from the second hidden convolutional layer which consist of 16 kernels. Colors represent magnitude of the value only.]]

imported>Student2018: /* Nearest Neighbor Interpolation */

2018-12-15T15:19:38Z

Nearest Neighbor Interpolation

← Older revision		Revision as of 15:19, 15 December 2018
Line 21:		Line 21:

	=== Upsampling Algorithms ===		=== Upsampling Algorithms ===
	=== Nearest Neighbor Interpolation ===		==== Nearest Neighbor Interpolation ====
	A naive technique used for predicting upsampled sensor voltage data, the nearest neighbor algorithm simply takes the prior sensor response for a 4x4 "tile" and copies the values towards the right, down, and diagonally towards the bottom-right to double the size of the sensor response data. Nearest neighbor simply assumes that the additional pixels in the more dense sensor would have captured the same information regardless. We use the predicted sensor values of the nearest neighbor algorithm as a performance baseline.		A naive technique used for predicting upsampled sensor voltage data, the nearest neighbor algorithm simply takes the prior sensor response for a 4x4 "tile" and copies the values towards the right, down, and diagonally towards the bottom-right to double the size of the sensor response data. Nearest neighbor simply assumes that the additional pixels in the more dense sensor would have captured the same information regardless. We use the predicted sensor values of the nearest neighbor algorithm as a performance baseline.

imported>Student2018: /* Failure Cases */

2018-12-15T15:18:02Z

Failure Cases

← Older revision		Revision as of 15:18, 15 December 2018
Line 160:		Line 160:
	[[File:edwardn_failure_cases.png\|center\|frame\|The row above shows the images from the simulated sensor data while those below are produced by the convolutional neural network. Note the mistranslation of strong blue colors.]]		[[File:edwardn_failure_cases.png\|center\|frame\|The row above shows the images from the simulated sensor data while those below are produced by the convolutional neural network. Note the mistranslation of strong blue colors.]]

	The thesis around this failure ~~can~~ be due to the infrequency of strong blue tones in the dataset which may bias the neural network to reduce the strength of blue voltage values. The nearest neighbor and bilinear interpolation algorithms did not demonstrate this behavior.		The thesis around this failure may be due to the infrequency of strong, isolated blue tones in the dataset which may bias the neural network to reduce the strength of blue voltage values. The nearest neighbor and bilinear interpolation algorithms did not demonstrate this behavior.

	== Conclusions ==		== Conclusions ==

imported>Student2018: /* Results */

2018-12-15T15:17:24Z

Results

← Older revision		Revision as of 15:17, 15 December 2018
Line 130:		Line 130:
	Unsurprisingly, the mean squared error for sensor values produced by the CNN was an order of magnitude lower than that of bilinear interpolation and the nearest network, given that the MSE was optimized for as the loss function for training. However, the low mean squared error did not necessarily translate to good color reproduction - we can see from the delta E 2000 values that the convolutional neural network did not perform appreciably better than nearest neighbor despite the images produced by the neural network to be apparently qualitatively superior.		Unsurprisingly, the mean squared error for sensor values produced by the CNN was an order of magnitude lower than that of bilinear interpolation and the nearest network, given that the MSE was optimized for as the loss function for training. However, the low mean squared error did not necessarily translate to good color reproduction - we can see from the delta E 2000 values that the convolutional neural network did not perform appreciably better than nearest neighbor despite the images produced by the neural network to be apparently qualitatively superior.

	We can detect this by observing a green overlay on some of the images compared to the target image, which is hypothesized to be the result of the frequency of green pixel values in the sensor data - that is the mean squared error loss function naively emphasizes the error seen in green pixel values over red or blue values by a factor of two. Moreover, since the image from the ~~Coco~~ dataset presumes to capture our everyday environment, by generating a color histogram we can demonstrate that the dominant color in our dataset is green, which would bias our neural network to produce green hues. We can visualize and quantitatively extract this information by producing the color histograms of the predicted and simulated images (notably we could have also performed this on the sensor response itself) using OpenCV.		We can detect this by observing a green overlay on some of the images compared to the target image, which is hypothesized to be the result of the frequency of green pixel values in the sensor data - that is the mean squared error loss function naively emphasizes the error seen in green pixel values over red or blue values by a factor of two. Moreover, since the image from the COCO dataset presumes to capture our everyday environment, by generating a color histogram we can demonstrate that the dominant color in our dataset is green, which would bias our neural network to produce green hues. We can visualize and quantitatively extract this information by producing the color histograms of the predicted and simulated images (notably we could have also performed this on the sensor response itself) using OpenCV.

	=== Sample Output ===		=== Sample Output ===

imported>Student2018: /* Analysis */

2018-12-15T15:13:49Z

Analysis

← Older revision		Revision as of 15:13, 15 December 2018
Line 97:		Line 97:

	=== Analysis ===		=== Analysis ===
	[https://docs.opencv.org OpenCV] was used to perform edge detection and to generate color histograms. S-CIELAB was used to generate delta E 2000 calculations between the predicted and target ~~values~~.		[https://docs.opencv.org OpenCV] was used to perform edge detection and to generate color histograms and S-CIELAB was used to generate delta E 2000 calculations between the predicted and target on the generated images.

	== Results ==		== Results ==