NickNoahMegan - Revision history

imported>Student221 at 05:37, 14 December 2019

2019-12-14T05:37:09Z

imported>Student221 at 05:32, 14 December 2019

2019-12-14T05:32:04Z

← Older revision		Revision as of 05:32, 14 December 2019
Line 1:		Line 1:
	== Introduction ==		== Introduction ==
	The goal of our project was to gain an understanding of the modern state of demosaicing techniques and how to reduce demosaicing artifacts. We explore existing algorithms to discover their strengths and weaknesses, and implement some of them to improve our understanding. We also look into the future possibilities for the ISP by implementing the demosaicing portion of the DeepISP paper.		The goal of our project was to gain an understanding of the modern state of demosaicing techniques and how to reduce demosaicing artifacts. We explore existing algorithms to discover their strengths and weaknesses, and implement some of them to improve our understanding. We also look into the future possibilities for the ISP by implementing the demosaicing portion of the DeepISP paper. Our explored methods can broadly be described in two sets of classes: classical vs machine learning methods, and methods that perform direct image interpolation vs cleaning artifacts from bilinear interpolation. In this work, we explore one classical, direct interpolation method and two machine learning methods for cleaning interpolation artifacts.

	== Background ==		== Background ==

imported>Student221: /* ISO 12233 Slanted Bar Metric */

2019-12-14T05:23:56Z

ISO 12233 Slanted Bar Metric

← Older revision		Revision as of 05:23, 14 December 2019
Line 179:		Line 179:
	[[File:dnnslantedbar.png\|500px\|thumb\|center\|alt text\| Figure 33: ISO 12233 DNN Output]]		[[File:dnnslantedbar.png\|500px\|thumb\|center\|alt text\| Figure 33: ISO 12233 DNN Output]]

	You'll notice that the output of PPG ~~is reddish~~. This is the result of the camera sensor simulated in ISETCam, which seems to be more sensitive to red light than green and blue. We saw similar results in our bilinearly interpolated images. We are not sure of the cause of this red tint at this time.		You'll notice that the output of PPG has a red tint. This is the result of the camera sensor simulated in ISETCam, which seems to be more sensitive to red light than green and blue. We saw similar results in our bilinearly interpolated images. We are not sure of the cause of this red tint at this time.

	One important thing to discuss is why KNN and DNN failed here. Both of these algorithms rely on images from the training set to learn. None of the images within our training set look like our slanted bar image. Most of them look like natural scenes. As a result, a possible reason for failure here could be a result of overfitting on our training dataset. A way to overcome this would be to generate more images like this and retrain our algorithm on both natural and synthetic images.		One important thing to discuss is why KNN and DNN failed here. Both of these algorithms rely on images from the training set to learn. None of the images within our training set look like our slanted bar image. Most of them look like natural scenes. As a result, a possible reason for failure here could be a result of overfitting on our training dataset. A way to overcome this would be to generate more images like this and retrain our algorithm on both natural and synthetic images.

imported>Student221: /* ISO 12233 Slanted Bar Metric */

2019-12-14T05:23:23Z

ISO 12233 Slanted Bar Metric

← Older revision		Revision as of 05:23, 14 December 2019
Line 167:		Line 167:
	[[File:slantedbaroi.png\|500px\|thumb\|center\|alt text\| Figure 29: ISO 12233 Example]]		[[File:slantedbaroi.png\|500px\|thumb\|center\|alt text\| Figure 29: ISO 12233 Example]]

	We ran the raw pixel values of these images through our algorithms and used the ISO 12233 metric on the resulting images. We were successfully able to run this metric on the ground truth slanted bar images as well as the output of PPG. These values are listed below in lines per mm.		We ran the raw pixel values of these images through our algorithms and used the ISO 12233 metric on the resulting images. We were successfully able to run this metric on the ground truth slanted bar images as well as the output of PPG. These values are listed below in lines per mm (Figure 30).

	[[File:slantedbartable.png\|500px\|thumb\|center\|alt text\| Figure 30: ISO 12233 Results]]		[[File:slantedbartable.png\|500px\|thumb\|center\|alt text\| Figure 30: ISO 12233 Results]]
Line 173:		Line 173:
	As expected, PPG has a bit of image sharpness reduction in comparison to the original ground truth image. The only exception is the 9 micron sensor, where we actually see better sharpness from PPG.		As expected, PPG has a bit of image sharpness reduction in comparison to the original ground truth image. The only exception is the 9 micron sensor, where we actually see better sharpness from PPG.

	We were unable to obtain the same metric on our KNN and DNN models though. Our models were trained on the images from the MSR Demosaicing dataset. The Slanted Bar image is very different from the images seen in this dataset which is composed of more natural-looking scenes. As a result, our outputs looked very different from our expected output. As a result, we were unable to obtain this slanted bar metric either because the ISO 12233 metric code from ISETCam couldn't identify the slanted line or because the MTF resulting from images output from these methods looked completely off. We include example outputs from all three algorithms below.		We were unable to obtain the same metric on our KNN and DNN models though. Our models were trained on the images from the MSR Demosaicing dataset. The Slanted Bar image is very different from the images seen in this dataset which is composed of more natural-looking scenes. As a result, our outputs looked very different from our expected output. As a result, we were unable to obtain this slanted bar metric either because the ISO 12233 metric code from ISETCam couldn't identify the slanted line or because the MTF resulting from images output from these methods looked completely off. We include example outputs from all three algorithms below (Figures 31-33).

	[[File:ppgslantedbar.png\|500px\|thumb\|center\|alt text\| Figure 31: ISO 12233 PPG Output]]		[[File:ppgslantedbar.png\|500px\|thumb\|center\|alt text\| Figure 31: ISO 12233 PPG Output]]

imported>Student221: /* PSNR */

2019-12-14T05:22:38Z

PSNR

← Older revision		Revision as of 05:22, 14 December 2019
Line 152:		Line 152:
	In our research, we found that several papers also looked at the PSNR of the outputs to as a metric for judging the images. We compared the PSNR of our images versus the ground truth for each individual color channel, as well as for the image as a whole, and found the following.		In our research, we found that several papers also looked at the PSNR of the outputs to as a metric for judging the images. We compared the PSNR of our images versus the ground truth for each individual color channel, as well as for the image as a whole, and found the following.

	For images without noise:		For images without noise (Figure 27).:

	[[File:Results table 1 demosaicing.png\|500px\|thumb\|center\|alt text\| Figure 27: Results (No Noise)]]		[[File:Results table 1 demosaicing.png\|500px\|thumb\|center\|alt text\| Figure 27: Results (No Noise)]]

	For images with noise:		For images with noise (Figure 28).:

	[[File:Results table 2 demosaicing.png.png\|500px\|thumb\|center\|alt text\| Figure 28: Results (With Noise)]]		[[File:Results table 2 demosaicing.png.png\|500px\|thumb\|center\|alt text\| Figure 28: Results (With Noise)]]

imported>Student221: /* DNN */

2019-12-14T05:21:49Z

DNN

← Older revision		Revision as of 05:21, 14 December 2019
Line 131:		Line 131:

	===DNN===		===DNN===
	~~Again, looking~~ at the same bus image, we see that the DNN model does a better job of reducing false color artifacts. It does an okay job of reducing zippering, but false color artifacts have substantially decreased, especially in the tree branches in the image. One downside though is that the image seems very blocky, almost as if it has lost resolution. This effect is more prevalent in certain images than others.		Figures 21 and 22 look at the same bus image as before, and we see that the DNN model does a better job of reducing false color artifacts. It does an okay job of reducing zippering, but false color artifacts have substantially decreased, especially in the tree branches in the image. One downside though is that the image seems very blocky, almost as if it has lost resolution. This effect is more prevalent in certain images than others.


Line 137:		Line 137:
	[[File:dnnresult1.png\|500px\|thumb\|center\|alt text\| Figure 22: Example 6 DNN Output]]		[[File:dnnresult1.png\|500px\|thumb\|center\|alt text\| Figure 22: Example 6 DNN Output]]

	~~You can also see that~~ the ~~output~~ sometimes tends to dull colors a little bit~~. We can see an example of this in the image below~~.		Figures 23 and 24 show an example where the network sometimes tends to dull colors a little bit.

	[[File:dnnsrc2.png\|500px\|thumb\|center\|alt text\| Figure 23: Example 7 Ground Truth]]		[[File:dnnsrc2.png\|500px\|thumb\|center\|alt text\| Figure 23: Example 7 Ground Truth]]
	[[File:dnnresult2.png\|500px\|thumb\|center\|alt text\| Figure 24: Example 7 DNN Output]]		[[File:dnnresult2.png\|500px\|thumb\|center\|alt text\| Figure 24: Example 7 DNN Output]]

	~~Here is another~~ example which has good colors but has a very blocky output.		Figures 25 and 26 show an example which has good colors but has a very blocky output.

	[[File:dnnsrc3.png\|500px\|thumb\|center\|alt text\| Figure 25: Example 8 Ground Truth]]		[[File:dnnsrc3.png\|500px\|thumb\|center\|alt text\| Figure 25: Example 8 Ground Truth]]

imported>Student221: /* KNN */

2019-12-14T05:20:27Z

KNN

← Older revision		Revision as of 05:20, 14 December 2019
Line 115:		Line 115:
	===KNN===		===KNN===

	Looking at the same bus image for KNN (~~Left is ground truth, right is PPG output~~), we see that KNN does not solve the zippering issue and there are overall more color artifacts than the PPG method.		Looking at the same bus image for KNN (Figures 15 and 16), we see that KNN does not solve the zippering issue and there are overall more color artifacts than the PPG method.

	[[File:GT output 1.png\|500px\|thumb\|center\|alt text\| Figure 15: Example 3 Ground Truth]]		[[File:GT output 1.png\|500px\|thumb\|center\|alt text\| Figure 15: Example 3 Ground Truth]]
	[[File:Knn bus.png\|500px\|thumb\|center\|alt text\| Figure 16: Example 3 KNN Output]]		[[File:Knn bus.png\|500px\|thumb\|center\|alt text\| Figure 16: Example 3 KNN Output]]

	~~An image~~ where KNN performed well ~~with little~~ artifacts ~~is shown below (left is ground truth, right is KNN output)~~. The reason KNN performs better on this image is simply because it is a much easier image to demosaic since it is fairly monochromatic. The main color is red which is why we don't see issues of zippering or false coloring. It also makes it easier for KNN to identify closer neighbor patches.		Figures 17 and 18 show an example where KNN performed well and has few artifacts. The reason KNN performs better on this image is simply because it is a much easier image to demosaic since it is fairly monochromatic. The main color is red which is why we don't see issues of zippering or false coloring. It also makes it easier for KNN to identify closer neighbor patches.

	[[File:Flower gt.png\|500px\|thumb\|center\|alt text\| Figure 17: Example 4 Ground Truth]]		[[File:Flower gt.png\|500px\|thumb\|center\|alt text\| Figure 17: Example 4 Ground Truth]]
	[[File:Flower knn.png\|500px\|thumb\|center\|alt text\| Figure 18: Example 4 KNN Output]]		[[File:Flower knn.png\|500px\|thumb\|center\|alt text\| Figure 18: Example 4 KNN Output]]

	~~An image~~ where KNN ~~performed very poorly is shown below (left is ground truth, right is KNN output)~~. We see this behavior from our KNN model when the weight parameters are not tuned for this specific case. For a better result, we would need to reduce the weights of the patches we add to prevent the overexposure we see here. This shows the disadvantage of a single set of parameters in the KNN model not working well for all images in our dataset.		Figures 19 and 20 show an example where KNN fails completely. We see this behavior from our KNN model when the weight parameters are not tuned for this specific case. For a better result, we would need to reduce the weights of the patches we add to prevent the overexposure we see here. This shows the disadvantage of a single set of parameters in the KNN model not working well for all images in our dataset.

	[[File:Tower gt.png\|500px\|thumb\|center\|alt text\| Figure 19: Example 5 Ground Truth]]		[[File:Tower gt.png\|500px\|thumb\|center\|alt text\| Figure 19: Example 5 Ground Truth]]

imported>Student221: /* PPG */

2019-12-14T05:18:58Z

PPG

← Older revision		Revision as of 05:18, 14 December 2019
Line 99:		Line 99:
	===PPG===		===PPG===

	For PPG, we looked at the following images (~~Left is ground truth, right is KNN output~~)		For PPG, we looked at the following images (Figures 11 and 12)

	[[File:GT output 1.png\|500px\|thumb\|center\|alt text\| Figure 11: Example 1 Ground Truth]]		[[File:GT output 1.png\|500px\|thumb\|center\|alt text\| Figure 11: Example 1 Ground Truth]]
Line 106:		Line 106:
	We can see some places where PPG succeeds and others where it fails. One place where it succeeds is on the bottom right of the bus where the window is divided by a metal bar. This bar is a common place where simple algorithms like bilinear interpolation would have just combined colors from both windows and the bar to get a blurry off-colored line. Here, PPG was able to avoid the zippering artifact, although there is some false color. One place where PPG does particularly poorly is the branches of the tree. These branches have a very high spatial frequency and there wasn't enough data to correctly resolve edges and colors.		We can see some places where PPG succeeds and others where it fails. One place where it succeeds is on the bottom right of the bus where the window is divided by a metal bar. This bar is a common place where simple algorithms like bilinear interpolation would have just combined colors from both windows and the bar to get a blurry off-colored line. Here, PPG was able to avoid the zippering artifact, although there is some false color. One place where PPG does particularly poorly is the branches of the tree. These branches have a very high spatial frequency and there wasn't enough data to correctly resolve edges and colors.

	An example where PPG performs well throughout is ~~here.~~		An example where PPG performs well throughout is seen in figures 13 and 14

	[[File:PPG GT output 2.png\|500px\|thumb\|center\|alt text\| Figure 13: Example 2 Ground Truth]]		[[File:PPG GT output 2.png\|500px\|thumb\|center\|alt text\| Figure 13: Example 2 Ground Truth]]

imported>Student221: /* K-Nearest Neighbors (KNN) */

2019-12-14T05:17:10Z

K-Nearest Neighbors (KNN)

← Older revision		Revision as of 05:17, 14 December 2019
Line 66:		Line 66:
	===K-Nearest Neighbors (KNN)===		===K-Nearest Neighbors (KNN)===

	[[File:Knn.png\|500px\|thumb\|right\|alt text\| Figure 6: KNN Overview]]		[[File:Knn.png\|500px\|thumb\|right\|alt text\| Figure 6: KNN Overview [3]]]

	The second algorithm we implemented is K-Nearest Neighbors (KNN). This method uses bilinear interpolation for the demosaicing and uses KNN for reducing the demosaicing artifacts. A set of ground truth "training" images are stored in an OpenCV KNN model. The way these ground truth images are stored is by dividing each image up into equal patches (for example 6x6 pixel squares) and also storing their bilinear interpolated result of their bayer patterned image.		The second algorithm we implemented is K-Nearest Neighbors (KNN). This method uses bilinear interpolation for the demosaicing and uses KNN for reducing the demosaicing artifacts. A set of ground truth "training" images are stored in an OpenCV KNN model. The way these ground truth images are stored is by dividing each image up into equal patches (for example 6x6 pixel squares) and also storing their bilinear interpolated result of their bayer patterned image.
Line 73:		Line 73:

	The idea behind this method is to add in missing detail resulting from just bilinear interpolation alone from a set of similar stored ground truth images. For example, if a test image patch we are working on is of the sky, we would most likely find all the closest neighbor patches to also be areas of the sky from our stored images. The advantages of this method are that it is a relatively simple machine learning technique to apply and it also doesn't require us to train a model. However, the disadvantages are that we see low performance if our dataset isn't specifically catered to this particular method. For example, if we do not have a large enough dataset for our ground truth "training" images, or if the images are too diverse, this method will struggle to find neighbor patches similar enough to add meaningful detail to the image. Another disadvantage is that if we want this algorithm to work well, we would need a relatively large amount of storage to store enough images.		The idea behind this method is to add in missing detail resulting from just bilinear interpolation alone from a set of similar stored ground truth images. For example, if a test image patch we are working on is of the sky, we would most likely find all the closest neighbor patches to also be areas of the sky from our stored images. The advantages of this method are that it is a relatively simple machine learning technique to apply and it also doesn't require us to train a model. However, the disadvantages are that we see low performance if our dataset isn't specifically catered to this particular method. For example, if we do not have a large enough dataset for our ground truth "training" images, or if the images are too diverse, this method will struggle to find neighbor patches similar enough to add meaningful detail to the image. Another disadvantage is that if we want this algorithm to work well, we would need a relatively large amount of storage to store enough images.


	~~''KNN Illustration [3]''~~

	===Deep Neural Network (DNN)===		===Deep Neural Network (DNN)===

imported>Student221: /* Dataset */

2019-12-14T05:15:25Z

Dataset

← Older revision		Revision as of 05:15, 14 December 2019
Line 51:		Line 51:
	=== Dataset ===		=== Dataset ===

	For our project, we used a demosaicing dataset provided by Microsoft Research Cambridge [1]. This dataset was designed for machine learning based demosaicing, and includes train, validation, and test sets of low-resolution images to learn a model and verify results. The images were generated using dcraw, which simulates the camera ISP pipeline for some given camera and can convert RAW images to PNG images by pushing the image through this pipeline. It includes images with and without noise. The results are verified using provided ground truth images, which were generated by downsampling the output of the camera in such a way as to reduce aliasing and other unwanted effects. To reintroduce noise into the dataset, they use the commonly accepted noise models that combine a Poisson and Gaussian distribution. The researchers fit the noise model to the original images, and then applied noise sampled from this model to the downsampled versions. The camera model that was used to generate the images we demosaic on is for the Panasonic Lumix DMC-LX3 CCD camera with a Bayer Pattern CFA. Examples of input and ground truth images are shown below. Input images are provided as ~~pngs~~ which are then converted to raw pixel values within our code.		For our project, we used a demosaicing dataset provided by Microsoft Research Cambridge [1]. This dataset was designed for machine learning based demosaicing, and includes train, validation, and test sets of low-resolution images to learn a model and verify results. The images were generated using dcraw, which simulates the camera ISP pipeline for some given camera and can convert RAW images to PNG images by pushing the image through this pipeline. It includes images with and without noise. The results are verified using provided ground truth images, which were generated by downsampling the output of the camera in such a way as to reduce aliasing and other unwanted effects. To reintroduce noise into the dataset, they use the commonly accepted noise models that combine a Poisson and Gaussian distribution. The researchers fit the noise model to the original images, and then applied noise sampled from this model to the downsampled versions. The camera model that was used to generate the images we demosaic on is for the Panasonic Lumix DMC-LX3 CCD camera with a Bayer Pattern CFA. Examples of input and ground truth images are shown below. Input images are provided as PNGs which are then converted to raw pixel values within our code.

	[[File:inputimgmsr.png\|500px\|thumb\|center\|alt text\| Figure 4: Dataset Inputs]]		[[File:inputimgmsr.png\|500px\|thumb\|center\|alt text\| Figure 4: Dataset Inputs]]