View source for Jaeseong

== Introduction ==


In semiconductor industry, optical microscopy me suffers from low resolution as the semiconductor process technology advanced and design rules became much smaller than the optical resolution imit. For example, deep UV light with 190 nm wavelength can resolve ~50nm structure considering the point spread function, while the cutting-edge semiconductor devices have sub 10nm structures.  In Fig1(a), a scanning electron microscope (SEM) image is shown. As shown in the image, semiconductor devices have patterns densely packed in a small area. In Fig1(b), downsampled SEM image is shown to demonstrate how typical optical image looks. Due to the resolution limit of optical imaging system, details are lost compared to SEM image. 
Despite the resolution advantage of SEM, the optical imaging is still widely used because it is much faster, and it does not require complicated environment to acquire images such as a vacuum changer.  In this regard, I explored image up-sampling techniques to enhance the optical image resolution.     

[[File:Jaeseong_Fig1.png]]

Fig1. (a) SEM image of typical logic device layer [1] (b) down sampled by a factor of 5 to mimic optical image

== Data Preparation ==

I chose scanning electron microscope (SEM) images as the ground truth. SEM can resolve ~10nm structures so that it is widely used in the industry and research.  I used SEM image dataset by GFZ data services [2], which are the images of various types of particles.  
All images in the data set are cropped in 500*500 pixel for the consistent image size as the original images did not have same size. Moreover, by cropping images, a greater number of images are generated and with the smaller image size, time required for computation can be saved. In this way I have generated 70 high resolution images.  
Then to mimic the lower resolution of the optical microscopy, I downsampled high resolution images by a factor of 4. In each image, 4*4-pixel block is taken and the mean pixel value is calculated to downsample. With the downsampling, the same set of 70 images are generated with the lower resolution of 125*125 pixel. To implement this data preparation, Images in the dataset is cropped using the Python PIL module crop mothod and downsampled using Python skimage module downscale_local_mean method. 
In Fig2, example SEM images in the data set are presented. Various types of sand particles are imaged 

[[File:Jaeseong_Fig2.png]]  

Fig2. Sand Particles SEM image examples in the data set.  

----

== Upsampling Methods ==


I have first tried upsampling interpolation methods 
Nearest Neighbor (NN) is the simplest and fastest image scaling method. When upsampling images, NN algorithm creates empty space between pixels in the source images. The pixel value is simply determined by taking the closest input pixel. In Fig3, the process is shown for upsampling a 4*4pixel image by a factor of 2 and generate an 8*8 pixel image. In the middle picture, black space represents empty pixels where interpolation is required.  

[[File:Jaeseong_Fig3.png]]

Fig3. Nearest Neighbor interpolation for upsampling images [3] 

Bilinear and bicubic interpolation methods take a greater number of pixels into calculation while NN takes only one pixel. [4],[5].They require more processing time but also generate smoother images. Bilinear considers the closest 2*2 neighboring pixels. Then it takes weighted average of neighboring 4pixels to determine the empty pixel value.  Bicubic interpolation calculates in a similar way to Bilinear but it considers more number of pixels. It takes the closest 4*4 neighboring pixels instead of 2*2. Fig4 illustrates how Bilinear and Bicubic interpolation takes pixels to calculate unknown pixel values. To implement these upsampling interpolation method, Python PIL module resize method is used. 

[[File:Jaeseong_Fig4.png]]

Fig4. (a) Bilinear interpolation (b) Bicubic interpolation [4] 

In addition to the traditional image upscaling interpolation methods introduced thus far, I have explored modern machine learning based Super Resolution(SR) method as well. 
I also tried Super-Resolution Generative Adversarial Network (SRGAN) model [6], [7]. Single Image SR has been attractive study area and many different approaches are proposed. Among those I chose to try SRGAN as this is known to be able to generate texture and details in high quality. 
In this method, Residual Dense Network (RDN) is used to construct high resolution images. It generates a large number of upscaled intermediate representations of the input low resolution images by Residual Dense Blocks (RDB). And then these inter mediate representations are reordered and recombined through a few convolutions.  Fig 5 shows the architecture of RDN. To implement this work existing Python module idealo/image-super-resolution is used [8]

[[File:Jaeseong_Fig5.png]] 

Fig5. The architecture of residual dense network[7]

----

== Results and Analysis ==


Data set images are processed and high resolution images are compared in Fig6. Ground Truth images are down sample by a factor of 4, so that down sampled image size is 4 times smaller. This low resolution image is processed through the 4 different algorithms discussed above, which are Nearest Neighbor, Bilinear, Bicubic interpolations and lastly SRGAN. 
For the performance comparison, zoomed in high resolution images and difference images are presented. Difference images are generated by taking the absolute values of each pixel level difference followed by the normalization. 
From the zoomed in images, NN image clearly shows ‘blockiness’ of texture. It is much noticeable on the edge of features where contrast is high.  Bilinear generated smoother image compared to NN but overall image looks blurry.  Bicubic shows further improved smoothness, and it also recovers some texture information. SRGAN shows the sharpest edges, visibly it has the best quality among all 4 results. However, this method over-sharpens the edge and it lost smoothness particle edges in the ground truth image. And it has created texture artifacts especially on the rough particle surface which do not exist in the ground truth image. 
From the difference images, it is shown that the overall noise level is reduced as more complex algorithm is applied. Also, for all interpolation method, noise is stronger on the edge of features.   

[[File:Jaeseong_Fig6.png]]

Fig. 6 Result image comparison for NN, Bilinear, Bicubic and SRGAN. Difference and Zoomed in result images are presented. 

For the quantitative analysis, root mean square error (RMSE), peak signal-to-noise ratio (PSNR) and structural similarity index method (SSIM) are measured, which are all widely used measures to check the image quality. RMSE is the standard deviation of the residuals, so that it can show how much the predictions differ from during the process. The lower valued of the RMSE means higher quality.   PSNR represents the maximum power signal compared to noise level in dB unit. The noise introduced during the upscaling process make the original signal weak. The higher value of PSNR indicates higher quality. SSIM has been developed to incorporate human visual perception to evaluate the image quality. SSIM value is ranged between 0 and 1, and higher value of SSIM indicates higher quality.  
In Fig7, RMSE, PSNR, SSIM are calculated and compared for all 4 types of upsampling methods. As more complicated interpolation method used, from NN to bilinear to bicubic, RMSE value is reduced and PSNR and SSIM are increased. This clear trend indicates complex interpolation improve the image quality.  However, SRGAN result shows higher RMSE and lower PSNR and SSIM compared to all other interpolation methods, which did not match qualitative comparison analysis presented in Fig6. 
It is understandable that SRGAN can have higher RMSE as it introduced noise by over sharpening and generating artifacts. However, it is not expected for SSIM also indicates lower image quality for SRGAN result. More study is required to further understand the performance of SRGAN training model and analysis methods. 

[[File:Jaeseong_Fig7.png]]

Fig7. Qualitative performance comparison. (a) RMSE, (b) PSNR, and (c) SSIM


----

== Conclusion ==

In this study, computational upsampling methods are presented to overcome the optical microscopy resolution limit. In total 70 ground truth images are prepared by cropping large SEM images to 500*500 pixel size. In this way more number of image samples, consistent image size, and shorter computation time can be achieved. To completed data set, ground truth images are down sampled by a factor of 4, resulting in 125*125 pixel low resolution counterparts.  
Nearest Neighbor, Bilinear, Bicubic interpolations as well as SRGAN based model training and prediction is performed to upsample low resolution images and restore the original size high resolution images. Later mentioned methods use more complicated computational algorithms.
The performance was analyzed by comparing difference images and calculating RMSE, PNSR, and SSIM. Visually, including high resolution and difference images, as more complicated method is applied, better-quality high-resolution image is achieved. Calculated image quality measures such as RMSE, PSNR and SSIM also agree the trend within interpolation methods that higher complexity algorithms produce better quality resolution images. However, calculated quality measures show that SRGAN is not as good as other interpolation methods. This maybe due to over-sharpened edge and artifacts in rough surface in SRGAN image that ground truth do not have. To understand and improve the result, more study can be done on training model and SSIM analysis.  


----

== Code ==

[[File:code_jaeseong.zip]]

== References ==

[1] Digital image. http://sanco-me.net/about/semiconductors-microelectronics/

[2]Klinkmüller, Matthias; Kemnitz, Helga; Schreurs, Guido; Rosenau, Matthias (2016): GeoMod2008 materials benchmark: The SEM image dataset. GFZ Data Services. http://doi.org/10.5880/GFZ.4.1.2016.004

[3] http://tech-algorithm.com/articles/nearest-neighbor-image-scaling/

[4] https://www.cambridgeincolour.com/tutorials/image-interpolation.htm
 
[5] J. A. Parker, R. V. Kenyon and D. E. Troxel, "Comparison of Interpolating Methods for Image Resampling," in IEEE Transactions on Medical Imaging, vol. 2, no. 1, pp. 31-39, March 1983.
doi: 10.1109/TMI.1983.4307610

[6] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, Xiaoou Tang, "ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks”
arXiv:1809.00219 [cs.CV]

[7] Yulun Zhang1 , Yapeng Tian2 , Yu Kong1 , Bineng Zhong1 , Yun Fu1 Residual Dense Network for Image Super-Resolution https://arxiv.org/pdf/1802.08797.pdf

[8] https://github.com/idealo/image-super-resolution