Po-Hsiang Wang

Introduction

Super-Resolution Generative Adversarial Networks (SRGAN) is a deep learning application to generate high resolution (HR) images from low resolution (LR) image [1-6]. In this work, we use SRGAN to up-scale 32x32 images to 128x128 pixels. Meanwhile, we evaluate the impact of different camera parameters on the quality of final up-scaled (high resolution) images and infer from these stimuli to understand what the network is able to learn.

SRGAN converts a LR image to HR through a generator/discriminator network.

Data processing and model training pipeline: Original image is processed with different camera parameters using ISET Camera Designer. These images are resized to 32x32x3 and served as the (LR) input to the generator. The target HR images are the original ones which are not processed. Total four models were trained:
1. Model_SR: SRGAN model that does super resolution only
2. Model_SR_Color: SRGAN model that does super resolution and color correction
3. Model_SR_Pixel: SRGAN model that does super resolution and restore spatial resolution due to reduction of system MTF
4. Model_SR_Deblur: SRGAN model that does super resolution and deblur

Dataset

Training

1800 cat and dog images were downloaded from Flickr and Pixabay. These images were processed using three different camera settings using ISET Camera Designer developed by David Cardinal, 2020, F20-PSYCH-221, Stanford University, under the framework of ISETCAM [7-9] link

• Camera setting 1- F/4; aperture diameter: 1mm, pixel size: 1umx1um; under exposing; default image processor
  
• Camera setting 2- F/4; aperture diameter: 1mm, pixel size: 25umx25um
  
• Camera setting 3- F/22; aperture diameter: 0.176mm; pixel size: 1umx1um
  

Objective for choosing these settings

• Camera setting 1- To produce images that require color correction as we use default image processor while under exposure, these images have warm tone
  
• Camera setting 2- In general, large pixel size is desirable because it results in higher dynamic range and signal-to-noise ratio. However, the reduction in spatial resolution and system MTF introduce severe pixelated effect
  
• Camera setting 3- Images looks much blurry compared with original images because they becomes diffraction limited at small aperture value
  

Original(baseline) and processed images through ISET Camera Designer.

Methods

SRGAN model

SRGAN structure

SRGAN takes a low-resolution image and up-scale it to become a higher resolution image. Input to the generator is a LR image, the generator will then create a 'fake' HR image. Input to the discriminator can be either a 'real' HR image or a 'fake' HR image. The job of the discriminator is to tell whether the input image is generated from the generator or not and give feedback (generator did a good job if discriminator got fooled)/back-propagate the GAN loss to train the generator.

Generator and Discriminator in SRGAN
Ledig, Christian, et al. "Photo-realistic single image super-resolution using a generative adversarial network." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. link

The Figure shown in the left describes the networks from the original paper. In this work, small modifications are done as follows

• • PixelShuffler x2: This is for feature map upscaling. We use 2x ‘deconv2d’ Keras built-in function for implementation.
    
  • PRelu(Parameterized Relu): PRelu introduces learnable parameter that makes it possible to adaptively learn the negative part coefficient. We use Relu as activation function for

The rest parts of our SRGAN model used in this work follows the original implementation

• • k3n64s1 means 3x3 kernel filter, outputting 64 channels with stride of 1.
    
  • Residual blocks: Since deeper networks are more difficult to train. The residual learning framework eases the training of these networks, and enables them to be substantially deeper, leading to improved performance.
    

SRGAN loss functions

Loss function is a measure of how good prediction the model does. Different loss functions are designed for different tasks.

Loss function of discriminator

• Mean squared error (MSE) loss,

$\mathrm{MSE}=\frac{1}{n}\sum _{i=1}^n(Y_i-\widehat{Y_i}{)}^2.$

where y_i is the ground-truth: (0: fake, 1: real) and ŷ_i is the predicted label from the discriminator.

Loss function of generator

• Adversarial loss: We use 'binary cross-entropy' loss in Keras. The goal is to have the discriminator output as "1" when the input is 'fake' HR such as the loss term is minimized.
  
• Content loss: L2 distance between 'real' and 'fake' HR feature maps. The feature maps are extracted from pre-trained VGG₁₉ network. We use the 9^th convolution layer (3x3x256 convolution) for extraction. In the example below, we can see these feature maps are quite similar to images filter by different spatial frequencies. The generator is optimized to maintain such perceptual similarity between a real HR image and its fake HR image.

Example of feature maps in 4 different channels (out of 256) extracted from 9^th layer of VGG₁₉ network

Image quality evaluations

SSIM

We use Structural Similarity (SSIM) index to measure the similarity between the model generated HR image and the original HR image (target image). Higher number indicates higher similarity (better quality of the model generated images).

Reference

[1] Ledig, Christian, et al. "Photo-realistic single image super-resolution using a generative adversarial network." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. link
[2] Wang, Xintao, et al. "Esrgan: Enhanced super-resolution generative adversarial networks." Proceedings of the European Conference on Computer Vision (ECCV). 2018. link
[3] Ciolino, Matthew, David Noever, and Josh Kalin. "Training set effect on super resolution for automated target recognition." Automatic Target Recognition XXX. Vol. 11394. International Society for Optics and Photonics, 2020. link
[4] Nagano, Yudai, and Yohei Kikuta. "SRGAN for super-resolving low-resolution food images." Proceedings of the Joint Workshop on Multimedia for Cooking and Eating Activities and Multimedia Assisted Dietary Management. 2018. link
[5] Pathak, Harsh Nilesh, et al. "Efficient super resolution for large-scale images using attentional GAN." 2018 IEEE International Conference on Big Data (Big Data). IEEE, 2018. link
[6] Takano, Nao, and Gita Alaghband. "Srgan: Training dataset matters." arXiv preprint arXiv:1903.09922 (2019). link
[7] A simulation tool for evaluating digital camera image quality (2004). J. E. Farrell, F. Xiao, P. Catrysse, B. Wandell . Proc. SPIE vol. 5294, p. 124-131, Image Quality and System Performance, Miyake and Rasmussen (Eds). January 2004. link
[8] Digital camera simulation (2012). J. E. Farrell, P. B. Catrysse, B.A. Wandell . Applied Optics Vol. 51 , Iss. 4, pp. A80–A90. link
[9] Image Systems Simulation (2015). J.E. Farrell and B.A. Wandell Handbook of Digital Imaging (Edited by Kriss). Chapter 8. ISBN: 978-0-470-51059-9. link