WanlingLiuQianDong

Introduction

Image deblurring is an important task in the area of image processing and computer vision. The goal of image deblurring is to recover a sharp image for a blurry image. Motion blur, which happens when there is a relative motion between a camera and a scene during image exposure time, is a common source of blurring when we take photos. We can reduce the effect of undesirable motion blur by reducing exposure time, and it is a fairly straight-forward way to solve the problem. However, this method can increase image noise significantly especially in dark scenes. We can also use ISP (Image Signal Process) to deblur image, though the conventional ISP pipeline solves certain subproblem at each step and requires certain engineering work (refer to Fig. 1). There are other traditional methods that can deal with motion blur, but they commonly set various constraints to model the blur, and those constraints limit the generalization of the model (i.e. when in real life, the blur is more complex, and the model can fail). Sometimes fine-tuning model parameters is also an issue since it may need certain level of prior knowledge.

Fig. 1 ISP pipeline

Due to the complexity of designing traditional approaches, people are trying to use machine learning to deal with those problems. Machine learning is able to learn and remove the blur without knowing prior information (e.g. kernel functions). In this project, we study and find a robust machine learning method for blind motion deblurring. There are two kinds of blurry images, one is blind images, knowing the sharp images while not knowing the source of blurring, the other one is non-blind, knowing both the sharp image and blurring source. We focus on the blind images in our project. To address the blurry image problem, CNN and GAN are commonly used. For example, [1] used multi-layer perceptron (MLP) for non-blind images, [2] designed a deep multi-scale CNN for dynamic scene deblurring, and [3] applied GAN for blind images. Therefore, we mainly focus on these two kinds of models and by comparing their pros and cons, we choose the model with best performance and robustness. Furthermore, we combine and modify the existing model to boost the performance of resulting images. We calculated MSE (Mean Square Error), PSNR (Peak Signal to Noise Ratio), SSIM (Structural Similarity) and S-CIELAB (spatial CIELAB) to evaluate our results. Our dataset is the GOPRO dataset [2], which is widely used for training and testing the models for motion deblurring. The dataset can be found here.

Methods

GAN model

• cGAN structure

A generative adversarial network is a popular deep learning approach to generate new realistic images. The architecture is comprised of a pair of competing generator and discriminator. The generator learns how to generate plausible fake images that ideally are indistinguishable from real examples in the dataset. The discriminator model is trained to classify whether a given image is real(real world images) or fake(generated images). By competing with each other, the two models grows together and finally, we will have good generator and discriminator.

Fig. 2 Conditional generative adversarial network architecture.

• Generator architecture

Fig. 3 The generator architectures used in Pix2Pix[1]

1. Simple generator

A simple generator is comprised of a downsampling encoder and a upsampling decoder. Both networks are based on convolutional networks typically one or two convolutional layers following by a maxpooling layer.

1. Unet-based generator

Unet is a convolutional neural network which was developed for image segmentation at first. The architecture is still based on the standard convolutional neural network, but with the internal connection between encoder and decoder, the modified generator is extended to smaller datasets.

1. VGG16-based generator

Fig. 4 VGG16-based generator architecture [2]

To accelerate the GAN converging speed and reduce the overfit due to limited dataset, the encoder of the generator is replaced by a pretrained VGG16 with all parameters locked for transfer learning. Based on the architecture, the time for each epoch is only half of the Unet generator and it also converges much earlier than Unet generator.

CNN model

File:Cnn1.png

Fig. 5 Different CNNs for image processing [4]

Fig. 6 SRN-DeblurNet framework [4]

Image Quality Evaluation

• Metrics

The mean-square error computes the pixel-based difference between two images which represents the cumulative squared error between the deblurred and the original image. A lower value of MSE represents a lower error and a high image quality.

$MSE=\frac{1}{mn}\sum _{i=0}^{m-1}\sum _{j=0}^{n-1}[I(i,j)-K(i,j){]}^2$

PSNR computes the peak signal-to-noise ratio between two images. The ratio representing a measure of the peak error can be used as a reference for image quality evaluation between the original and the deblurred image. An image with a higher PSNR has a higher image quality.

$PSNR=10{\log }_{10}(\frac{MA{X}_I^2}{MSE})$

SSIM is used for measuring the similarity between two images. SSIM is designed to improve on traditional methods such as peak signal-to-noise ratio (PSNR) and mean squared error (MSE).

$SSIM(x,y)=\frac{(2{\mu }_x{\mu }_y+C_1)+(2{\sigma }_{xy}+C_2)}{({\mu }_x^2+{\mu }_y^2+C_1)+({\sigma }_x^2+{\sigma }_y^2+C_2)}$

• Yolo object detection

Results

• GAN
• CNN

Conclusions

• In general, our CNN model beats pix2pix GAN model, and CNN model yields practical results
• (GAN) Pix2pix may not be a suitable model for image deblurring but other GAN models may work
• (CNN) It is difficult for the models to retrieve the high-frequency information (edges, fine details...). But we can design better architectures specifically for high-frequency information or we can incorporate perceptual loss.
• The metrics we used (MSE, PSNR, SSIM, object detection, S-CIELAB) in general indicate the trend, though sometimes they have limitations when compared with what human perceive.

Reference

[1] Schuler, C. J., Christopher Burger, H., Harmeling, S., & Scholkopf, B. (2013). A machine learning approach for non-blind image deconvolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1067-1074).

[2] Nah, S., Hyun Kim, T., & Mu Lee, K. (2017). Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3883-3891).

[3] Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., & Matas, J. (2018). Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 8183-8192).

[4] Tao, X., Gao, H., Shen, X., Wang, J., & Jia, J. (2018). Scale-recurrent network for deep image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 8174-8182).

Appendix

Model parameters: