WanlingLiuQianDong

Introduction

Image deblurring is an important task in the area of image processing and computer vision. The goal of image deblurring is to recover a sharp image for a blurry image. Motion blur, which happens when there is a relative motion between a camera and a scene during image exposure time, is a common source of blurring when we take photos. We can reduce the effect of undesirable motion blur by reducing exposure time, and it is a fairly straight-forward way to solve the problem. However, this method can increase image noise significantly especially in dark scenes. We can also use ISP (image signal process) to deblur image, though the conventional ISP pipeline solves certain subproblem at each step and requires certain engineering work (refer to Fig. 1). There are other traditional methods that can deal with motion blur, but they commonly set various constraints to model the blur, and those constraints limit the generalization of the model (i.e. when in real life, the blur is more complex, and the model can fail). Sometimes fine-tuning model parameters is also an issue since it may need certain level of prior knowledge.

Fig. 1 ISP pipeline

Due to the complexity of designing traditional approaches, people are trying to use machine learning to deal with those problems. Machine learning is able to learn and remove the blur without knowing prior information (e.g. kernel functions). In this project, we study and find a robust machine learning method for blind motion deblurring. There are two kinds of blurry images, one is blind images, knowing the sharp images while not knowing the source of blurring, the other one is non-blind, knowing both the sharp image and blurring source. We focus on the blind images in our project. To address the blurry image problem, CNN (convolutional neural network) and GAN (generative adversarial network) are commonly used. For example, [1] used multi-layer perceptron (MLP) for non-blind images, [2] designed a deep multi-scale CNN for dynamic scene deblurring, and [3] applied GAN for blind images. Therefore, we mainly focus on these two kinds of models and by comparing their pros and cons, we choose the model with best performance and robustness. Furthermore, we combine and modify the existing model to boost the performance of resulting images. We calculated MSE (Mean Square Error), PSNR (Peak Signal to Noise Ratio), SSIM (Structural Similarity) and S-CIELAB (spatial CIELAB) to evaluate our results. Our dataset is the GOPRO dataset [2], which is widely used for training and testing the models for motion deblurring. The dataset can be found here.

Dataset

We used the GOPRO dataset [2]. The dataset contains 3214 pairs of blurry and sharp images at 1280x720 resolution and is publicly available at this website. We randomly selected 2103 images as our training set and 500 images as our testing set (training set: testing set $\approx$ 8:2).

As for the generation of image pairs, they adopted kernel-free methods for generating blurry images, which is to approximate camera imaging process instead of designing complex kernels. They used GOPRO4 Hero Black camera to take 240 fps videos, averaging (integrating, more details are described in section 2 of [2]) various number (7-13) of successive latent frames to produce different strengths of blurs (the motions are caused by both camera shake and object movement). The sharp image corresponding to each blurry image is defined as the mid-frame among sharp images used to produce blurry images.

Methods

ISP

The traditional way to do image signal processing is based on the prior of the point spread function and noise in the image. The algorithm is used to find the blur kernel and then recover to the original image. The blur can be caused by various reasons such as defocus aberration, motion blur, gaussian blur, and in real cases, there isn't enough information for finding the blur kernel. Therefore, traditional deblurring methods are easy to suffer from intensive parameters tuning and poor performance.

GAN model

GAN structure

A generative adversarial network is a popular deep learning approach to generate new realistic images. The architecture is comprised of a pair of competing generator and discriminator. The generator learns how to generate plausible fake images that ideally are indistinguishable from real examples in the dataset. The discriminator model is trained to classify whether a given image is real(real world images) or fake(generated images). By competing with each other, the two models grows together and finally, we will have good generator and discriminator.

Fig. 2 Conditional generative adversarial network architecture.

Different generator architectures

• Simple generator

A simple generator is comprised of a downsampling encoder and a upsampling decoder. Both networks are based on convolutional networks typically one or two convolutional layers following by a maxpooling layer.

Fig. 3 The generator architectures used in Pix2Pix[1]

• Unet-based generator

Unet is a convolutional neural network which was developed for image segmentation at first. The architecture is still based on the standard convolutional neural network, but with the internal connection between encoder and decoder, the modified generator is extended to smaller datasets.

• VGG16-based generator

Fig. 4 VGG16-based generator architecture [2]

To accelerate the GAN converging speed and reduce the overfit due to limited dataset, the encoder of the generator is replaced by a pretrained VGG16 with all parameters locked for transfer learning. Based on the architecture, the time for each epoch is only half of the Unet generator and it also converges much earlier than Unet generator.

CNN model

• SRN (scale-recurrent network) structure

Fig. 5 SRN-DeblurNet framework [4]

Xin Tao et al. proposed the scale-recurrent network (SRN) which is shown in Fig. 5. There are five basic modules that are worth noticing:

1. Encoder-decoder: this is adapted from U-Net [6]. This structure improves the network’s regression ability, and is widely used in image and video processing.

2. Residual Block: [7] proposed a structure that has an additional shortcut (identity mapping) connection compared with conventional structure that stacks layers. This structure can deal with the gradient vanishing problem [7] and is useful for deep networks.

3. Skip connection: skip connection is used for combining different levels of features and also good for gradient flow and accelerating convergence.

4. Scale connection: the model can benefit from scale connection since the scale connection makes it possible for the model to train for different scales of images, solving the problem of overfitting specific scales.

5. RNN: at each scale level, we are doing similar work, therefore, we can use RNN to connect different levels. The characteristics of RNN is to share weights between each hidden state, which has the effect of reducing parameters to decrease the risk of overfitting and increasing efficiency.

• Modified model

Fig. 6 Our basic block

Though Tao et al. proposed the SRN model, they also discussed the influence of RNN layers and they suggested that without using RNN, the model can still generate comparable results. Therefore, we decided to modify the SRN model by removing the RNN layers and change their basic block to further simplify the model and explore the potential of pure CNN. As shown in Fig. 6, our block also stacks convolutional layers and has the basic structure of residual module but would be computationally cheaper. In general, this simplified model combines the advantages mentioned in previous section, and further improves the efficiency.

Image Quality Evaluation

Reference Metrics

The mean-square error computes the pixel-based difference between two images which represents the cumulative squared error between the deblurred and the original image. A lower value of MSE represents a lower error and a high image quality.

$MSE=\frac{1}{mn}\sum _{i=0}^{m-1}\sum _{j=0}^{n-1}[I(i,j)-K(i,j){]}^2$

PSNR computes the peak signal-to-noise ratio between two images. The ratio representing a measure of the peak error can be used as a reference for image quality evaluation between the original and the deblurred image. An image with a higher PSNR has a higher image quality.

$PSNR=10{\log }_{10}(\frac{MA{X}_I^2}{MSE})$

SSIM is used for measuring the structural similarity between two images. SSIM is designed to improve on traditional methods such as peak signal-to-noise ratio (PSNR) and mean squared error (MSE).

$SSIM(x,y)=\frac{(2{\mu }_x{\mu }_y+C_1)+(2{\sigma }_{xy}+C_2)}{({\mu }_x^2+{\mu }_y^2+C_1)+({\sigma }_x^2+{\sigma }_y^2+C_2)}$

These three metrics above are commonly used for papers evaluating the results of deblurring. Combining those three metrics we expect they can offer a general trend about the results from the aspects of pixel-based difference, noise level, and structural difference. However, those parameters do not include much information related with human perception. Therefore, we investigated more about the metrics.

Yolo object detection

Yolo object detection network [3] is one of the most popular networks for object detection task. The algorithm is based on a neural network and can detect all the objects on the image. The network divides the image into grids and output detected objects with bounding boxes and confidence. Since Yolo object detection is trained based on real people object detection, it can be used to justify where the image quality is increased. A better quality image is supposed to increase objects detected and detection confidence.

Fig. 7 Yolo object detection block [3]

Perceptual Metric

We also include s-cielab to evaluate different models. It merges CIELAB and human spatial sensitivity and provides a difference between reference and test image. The calculation step is shown in Fig. 8. For the S-CIELAB error measure. The first step is to implement a spatial filtering to the image data which simhlates the human visual system. Then, make is consistent with the basic CIELAB calculation.

Fig. 8 A fow-chart for S-CIELAB computation.[5]

Results

• GAN

Model Comparison (averaged data for 100 epochs on the test set)
Model	Blurry Image	GAN with Unet generator	GAN with VGG16 generator	Sharp image
MSE	222.75	337.44	415.21	N/A
PSNR	25.75	23.44	22.50	N/A
SSIM	0.79	0.77	0.74	N/A
s-cielab	755	1003	1623	N/A
Objects detected	18	13	13	22
Detection Confidence	75%	82.23%	79.92%	82.77%
Runtime (s/epoch)	N/A	375.71	163.67	N/A
Trainable parameters	N/A	54,414,979	13,753,091	N/A

• CNN

Model Comparison (averaged data for 100 epochs on the test set)
Model	Blurry Image	Tao et al.	Ours
MSE	222.75	190.48	162.91
PSNR	25.75	26.35	27.04
SSIM	0.785	0.793	0.816
Runtime (s/epoch)	N/A	245	187

Discussion

The image produced our current CNN model are clearer but not perfectly deblurred. The reason can be that due to the time and budget limitation, the model has not converged yet and we expected the output would be more desirable if we ran our model for more epochs. The other reason is that the current loss function only takes content loss into account, which can be improved to include perception loss using S-CIELAB. We expected that the model performance would be boosted especially when dealing with high-frequency data.

When dealing with deblurring, we can notice that for images showing the effect of depth-of-field (DOF), the deblurring model would not influence on the DOF effect, which is a nice characteristics since the DOF effect is what we want to preserve (see figure bird and leopard). This also suggests that our model is practical for real life usage.

Conclusions

• In general, our CNN model beats pix2pix GAN model, and CNN model yields practical results
• (GAN) Pix2pix may not be a suitable model for image deblurring but other GAN models may work
• (CNN) It is difficult for the models to retrieve the high-frequency information (edges, fine details, etc.). But we can design better architectures specifically for high-frequency information or we can incorporate perceptual loss.
• The metrics we used (MSE, PSNR, SSIM, object detection, S-CIELAB) in general indicate the trend, though sometimes they have limitations when compared with what human perceive.

Reference

[1] Schuler, C. J., Christopher Burger, H., Harmeling, S., & Scholkopf, B. (2013). A machine learning approach for non-blind image deconvolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1067-1074).

[2] Nah, S., Hyun Kim, T., & Mu Lee, K. (2017). Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3883-3891).

[3] Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., & Matas, J. (2018). Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 8183-8192).

[4] Tao, X., Gao, H., Shen, X., Wang, J., & Jia, J. (2018). Scale-recurrent network for deep image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 8174-8182).

[5] Zhang, X., & Wandell, B. A. (1996, May). A spatial extension of CIELAB for digital color image reproduction. In SID international symposium digest of technical papers (Vol. 27, pp. 731-734). SOCIETY FOR INFORMATION DISPLAY.

[6] Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234-241). Springer, Cham.

[7] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).

Acknowledgment

Thanks Professor Brian Wandell for sharing the insight related with depth of field, Dr. Joyce Farrell for the advice of metrics, and Zheng Lyu for discussing the loss functions and project idea.

Appendix

Model parameters: