Po-Hsiang Wang

Introduction

Super-Resolution Generative Adversarial Networks (SRGAN) is a deep learning application to generate high resolution (HR) images from low resolution (LR) image [1-6]. In this work, we use SRGAN to up-scale 32x32 images to 128x128 pixels. Meanwhile, we evaluate the impact of different camera parameters on the quality of final up-scaled (high resolution) images and infer from these stimuli to understand what the network is able to learn.

SRGAN converts a LR image to HR through a generator/discriminator network.

Data processing and model training pipeline: Original image is processed with different camera parameters using ISET Camera Designer. These images are resized to 32x32x3 and served as the (LR) input to the generator. The target HR images are the original ones which are not processed. Total four models were trained:
1. Model_SR: SRGAN model that does super resolution only
2. Model_SR_Color: SRGAN model that does super resolution and color correction
3. Model_SR_Pixel: SRGAN model that does super resolution and restore spatial resolution due to reduction of system MTF
4. Model_SR_Deblur: SRGAN model that does super resolution and deblur

Background

SRGAN network

SRGAN takes a low-resolution image and up-scale it to become a higher resolution image. Input to the generator is a LR image, the generator will then create a 'fake' HR image. Input to the discriminator can be either a 'real' HR image or a 'fake' HR image. The job of the discriminator is to tell whether the input image is generated from the generator or not and give feedback (generator did a good job if discriminator got fooled)/back-propagate the GAN loss to train the generator.

Generator and Discriminator in SRGAN
Ledig, Christian, et al. "Photo-realistic single image super-resolution using a generative adversarial network." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. link

The Figure shown in the left describes the networks from the original paper. In this work, small modifications are done as follows

• • PixelShuffler x2: This is for feature map upscaling. We use 2x ‘deconv2d’ Keras built-in function for implementation.
    
  • PRelu(Parameterized Relu): PRelu introduces learnable parameter that makes it possible to adaptively learn the negative part coefficient. We use Relu as activation function for

The rest parts of our SRGAN model used in this work follows the original implementation

• • k3n64s1 means 3x3 kernel filter, outputting 64 channels with stride of 1.
    
  • Residual blocks: Since deeper networks are more difficult to train. The residual learning framework eases the training of these networks, and enables them to be substantially deeper, leading to improved performance.
    

Loss function is a measure of how good prediction the model does. Different loss functions are designed for different tasks.

Loss function of discriminator

• Mean squared error (MSE) loss,

$\mathrm{MSE}=\frac{1}{n}\sum _{i=1}^n(Y_i-\widehat{Y_i}{)}^2.$

where y_i is the ground-truth: (0: fake, 1: real) and ŷ_i is the predicted label from the discriminator.

Loss function of generator

• Adversarial loss: We use 'binary cross-entropy' loss in Keras. The goal is to have the discriminator output as "1" when the input is 'fake' HR such as the loss term is minimized.
  
• Content loss: L2 distance between 'real' and 'fake' HR feature maps. The feature maps are extracted from pre-trained VGG₁₉ network. We use the 9^th convolution layer (3x3x256 convolution) for extraction. In the example below, we can see these feature maps are quite similar to images filter by different spatial frequencies. The generator is optimized to maintain such perceptual similarity between a real HR image and its fake HR image.

Example of feature maps in 4 different channels (out of 256) extracted from 9^th layer of VGG₁₉ network

SSIM

We use Structural Similarity (SSIM) index to measure the similarity between the model generated HR image and the original HR image (target image). Higher number indicates higher similarity (better quality of the model generated images).

Appendix

You can write math equations as follows: $y=x+5$

You can include images as follows (you will need to upload the image first using the toolbox on the left bar, using the "Upload file" link).

Reference

[1] Ledig, Christian, et al. "Photo-realistic single image super-resolution using a generative adversarial network." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. link
[2] Wang, Xintao, et al. "Esrgan: Enhanced super-resolution generative adversarial networks." Proceedings of the European Conference on Computer Vision (ECCV). 2018. link
[3] Ciolino, Matthew, David Noever, and Josh Kalin. "Training set effect on super resolution for automated target recognition." Automatic Target Recognition XXX. Vol. 11394. International Society for Optics and Photonics, 2020. link
[4] Nagano, Yudai, and Yohei Kikuta. "SRGAN for super-resolving low-resolution food images." Proceedings of the Joint Workshop on Multimedia for Cooking and Eating Activities and Multimedia Assisted Dietary Management. 2018. link
[5] Pathak, Harsh Nilesh, et al. "Efficient super resolution for large-scale images using attentional GAN." 2018 IEEE International Conference on Big Data (Big Data). IEEE, 2018. link
[6] Takano, Nao, and Gita Alaghband. "Srgan: Training dataset matters." arXiv preprint arXiv:1903.09922 (2019). link