Uriel Rosa

Introduction

Cameras designed for robotic, and/or autonomous vehicle vision applications have typically been adapted from existing human-intended pipelines. However, robotic handling of images does not necessarily have to emulate the human vision systems for achieving high performance, or reduced costs. The semantic labeling of images classified by CNN (convolutional neural network) approaches might be substantially influenced by the design parameters of the cameras acquiring the images.

In this study, camera pipeline parameters were modified to investigating the effects of replacing typical in focus RGB images with similar images reprocessed as monochrome, defocused to include chromatic aberration effects and defocused monochrome images.

The ieCameraDesigner ISET [1] application software was configured in four distinct camera designs to generate rgb, monochromatic, defocused and a combination of these effects in distinct pipelines.

Natural images of African mammals downloaded from David Cardinal’s data set [2] were screened for close similarity to produce a base dataset of containing images processed by a state-of-the-art CNN, the Resnet-50.

The goal of this study is to evaluate simulated effects of the camera parameters monochromatic and chromatic defocusing on the performance of semantic labeling computed by convolutional neural networks.

Background

Relevant literature referring to the monochrome, chromatic defocusing and semantic labeling classification of current convolutional neural network frameworks, and autonomous vision, is cited and partially edited in this section, as follows.

“Recent years have witnessed amazing progress in AI related fields such as computer vision, machine learning and autonomous vehicles. Since the first successful demonstrations in the 1980s, great progress has been made in the field of autonomous vehicles. However, fully autonomous navigation in arbitrarily complex environments also require informed decisions made by CNN. Accurate perception systems, autonomous vision, are required in autonomous navigation [3].

An object detection task can be addressed with a variety of different sensors in an integrated approach. However, cameras are the cheapest and most commonly used type of sensors for the detection of objects. The visible spectrum of light is typically used for daytime detections, whereas the infrared spectrum can be used for nighttime detection [3]. A traditional detection pipeline includes object classification and verification/refinement.

Semantic segmentation is a fundamental topic in computer vision. The goal of semantic segmentation is to assign each pixel in the image a label from a predefined set of categories. Semantic segmentation is the first step towards scene understanding. It is mainly based on low-features, such as color, edges, and brightness. The methods for feature selection have been reported in the above subtasks of lane and road detection, traffic sign recognition, and vehicle detection. Wu et al. (2016b) have proposed a more efficient ResNet architecture by analyzing the effective depths of 21 residual units. They point out that ResNets behave as linear ensembles of shallow networks. Based on this understanding they design a group of relatively shallow convolutional networks for the task of semantic image segmentation [3].

The modern era of neural networks began with the pioneering work of McCulloch and Pitts(1943). They described a logical calculus of neural networks that united the studies of neurophysiology and mathematical logic. With a sufficient number of simple units, neurons, and synaptic connections set properly and operating synchronously, they showed that a network would compute any computable function. The disciplines of neural networks and artificial intelligence were born. The properties of the machines and their behavior are inspired by facts about animal brains. In 1986 the development of the back-propagation algorithm was reported by Rumelhart, Hinton and Williams. Back-propagation learning was discovered independently in two other places about the same time (Parker, 1985; LeCun, 1985). Convolutional Neural Networks allowed a significant improvement in the performance of object detection [4]; and machine vision.

Deep Residual Learning for Image Recognition, “ResNet-50”, are currently the state-of-the-art in image classification. Deeper neural networks are more difficult to train. Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun presented a residual learning framework to ease the training of networks that are substantially deeper than those used previously. They explicitly reformulated the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. They provided comprehensive empirical evidence showing that these residual networks are easier to optimize, and gain accuracy from considerably increased depth [5,6]. The method was evaluated on the ImageNet 2012 classification dataset that consisted of 1000 classes. The models are trained on the 1.28 million training images and evaluated on the 50k validation images. It was tested on 100k test images.

Pre-trained Model: A pre-trained model has been previously trained on a dataset and contains the weights and biases that represent the features of whichever dataset it was trained on. Learned features are often transferable to different data. For example, a model trained on a large dataset of animal images will contain learned features like edges or horizontal lines that you would be transferable to our dataset. Pre-trained models are beneficial for many reasons. Using a pre-trained model saves time. Time and compute resources has already been spent to learn a lot of features that the model will likely benefit from [5,6,7].

Monochrome images: Technologies used in autonomous vehicles typically include lane detection. Typical images collected by the on-board camera are color images [8,9]. Each pixel in the image is made up of R, G, and B three color components, which contains large amount of information. Processing these images directly makes the algorithm consume a lot of time. Image preprocessing includes grayscale conversion of color image, gray stretch, median filter to eliminate the image noise and other interference information. Gray stretch can increase the contrast between the lane and the road, which makes the lane lines more prominent. Equation (1) represents the function which is to be applied to an RGB image to convert it to Gray Scale.

L(x,y) = 0.21 R(x,y) + 0.72G(x,y) + 0.07 B(x,y) (1)

Where R - Red component of the image G - Green component of the image B - Blue component of the image x,y - position of a pixel [8]. In the current study, gray stretch is not performed. However, color images are replaced by monochrome images to evaluating this effect in CNN classification performances.

Defocus aberration: In optics, defocus is the aberration in which an image is simply out of focus. Optically, defocus refers to a translation of the focus along the optical axis away from the detection surface. In general, defocus reduces the sharpness and contrast of the image. What should be sharp, high-contrast edges in a scene become gradual transitions. Fine detail in the scene is blurred or even becomes invisible. Nearly all image-forming optical devices incorporate some form of focus adjustment to minimize defocus and maximize image quality [10]. Figure 1 demonstrates typical cases of chromatic aberrations.

File:D:\F20-PSYCH221\project\pics\ca.png

Figure 1. Left: intense effect of chromatic aberration (mouth); Center: diagram indicating chromatic aberration produced by the lens; Right: text image shows strong chromatic aberrations [10,11].

Image Signal Processing Pipelines: Jiang et. al [12] introduced a method that combines machine learning and image systems simulation that automates the image processing pipeline design. The approach is based on a new way of thinking of the image processing pipeline as a large collection of local linear filters. The method has been used to design pipelines for novel sensor architectures in consumer photography applications. It applies a learning-based approach to the ISP pipeline design using affine mapping frameworks. Image patches are clustered based on simple features and then a per-class affine mapping learns to map the raw patches to the sRGB patches. This work combines image systems simulation technology and modern computational methods into a methodology that creates image processing pipelines.”

Methods

In the current study defocused aberrations are introduced in the simulated ISET/isetcam pipeline for monochrome and rgb images. The processed images allow quantifying these effects on the performance of the CNN. The pre-trained Resnet-50 upper classification layers are re-trained with four classes of wild animal images obtained from David Cardinal’s African mammals data set. The simulated camera pipeline designs are obtained from the ISET/isetcam ieCameraDesigner ISE 2020 application. Original dataset: Natural images of African mammals downloaded from David Cardinal’s data set [2] were screened for close similarity to produce a total of 100 images per class (50% original and 50% reflected images). A sample of animal classes containing original JPEG images is shown in figure 2.

Figure 2. Original images representing the four classes of animals selected from David Cardinal’s data set; from left to right: cheetah, hyenas, leopards and lions.

ISET dataset simulated with the ieCameraDesigner RGB: the original images were processed through the ieCameraDesigner pipeline using the default optical image, sensor and ISP designs, except the sensor color image was selected as rgb. A sample of the same animal classes containing the monochrome images is shown in figure 3.

Figure 3. ISET simulated RGB images representing the four classes of animals selected from David Cardinal’s data set; from left to right: cheetah, hyenas, leopards and lions.

Defocused RGB: the original images were processed through the ieCameraDesigner pipeline using the default optical image, sensor and ISP designs, except the sensor color image was selected as rgb. The cameraTweak file shown in Appendix I was loaded in the ieCameraDesigner pipeline to produce the defocus effect of 5.5 diopters on the images. A sample of the same animal classes containing the defocused RGB images is shown in figure 4.

Figure 4. ISET simulated defocused RGB images representing the four classes of animals selected from David Cardinal’s data set; from left to right: cheetah, hyenas, leopards and lions.

Monochrome: The original images were processed through the ieCameraDesigner pipeline using the default optical image, sensor and ISP designs, except the sensor color image was selected as monochrome. A sample of the same animal classes containing the monochrome images is shown in figure 5.

Figure 5. ISET simulated monochrome images representing the four classes of animals selected from David Cardinal’s data set; from left to right: cheetah, hyenas, leopards and lions.

Defocused monochrome: the original images were processed through the ieCameraDesigner pipeline using the default optical image, sensor and ISP designs, except the sensor color image was selected as monochrome. The cameraTweak file shown in Appendix I was loaded in the ieCameraDesigner pipeline to produce the defocus effect of 5.5 diopters on the images. A sample of the same animal classes containing the defocused monochrome images is shown in figure 6.

Figure 6. ISET simulated defocused monochrome images representing the four classes of animals selected from David Cardinal’s data set; from left to right: cheetah, hyenas, leopards and lions.

Figure 7 shows a block diagram of the dataset obtained after processing the original data through the modified pipelines created by the ISET/isetcam ieCameraDesigner application.

Figure 7. Block diagram showing the dataset obtained after original dataset is processed by the ISET/isetcam ieCameraDesigner application.

Results

Table 1. Resnet-50 classification performance results. Dataset classes: cheetah, hyenas,

              leopards, lions	Resnet-50 / 200 epochs 

Resnet-50 / 100 epochs

Validation accuracy, % Validation loss Validation accuracy, % Validation loss Original 91 0.24 87 0.34 RGB 85 0.43 80 0.51 Defocused RGB 68 0.88 53 1.15 Monochrome 88 0.47 79 0.51 Defocused monochrome 68 0.98 62 0.88

Table 2. SqueezeNet classification performance per class results. SqueezeNet Pre-trained Network Correctly classified counts /250 cheetah hyenas leopards lions RGB 107 60 93 80 Defocused RGB 61 35 29 51 Monochrome 99 60 65 38 Defocused monochrome 41 24 13 20

References

[1] ISET/isetcam “aiCameraDesigner” ISE 2020 in: https://github.com/ISET/isetcam [2] David Cardinal’s Natural images of African mammals:

      https://canvas.stanford.edu/files/6540020/download?download_frd=1

[3] J. Janaia, F. Guney, A. Behla and A. Geigera. “Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art”. Preprint submitted to ISPRS Journal of Photogrammetry and Remote Sensing, April 20, 2017. [4 ] H. Simon. “Neural Networks: a comprehensive foundation”. Prentice Hall 2nd ed. 1999. [5] K. He, X. Zhang, S.Ren and J. Sun. “Deep Residual Learning for Image Recognition”. https://arxiv.org/abs/1512.03385 [6] Resnet-50 Kaggle Pre-trained models for keras: https://www.kaggle.com/keras/resnet50 [7] Resnet-50 Matlab: https://www.mathworks.com/help/deeplearning/ref/resnet50.html?s_tid=srchtitle [8] V. Viswanathan , R. Hussein. “Applications of Image Processing and Real-Time embedded Systems in Autonomous Cars: A Short Review”. International Journal of Image Processing (IJIP), Volume (11): Issue (2): 2017. [9] H. Zhu, K. Yuen, L. Mihaylova and H. Leung. “Overview of Environment Perception for Intelligent Vehicles”. IEEE Transactions on Intelligent Transportation Systems, vol.18, No.10, October 2017. [10] https://en.wikipedia.org/wiki/Defocus_aberration [11] https://digital-photography-school.com/chromatic-aberration-what-is-it-and-how-to-avoid-it/ [12] H. Jiang, Q. Tian, J. Farrell and B. Wandell. “Learning the image processing pipeline”. IEEE Transactions on Image Processing, vol. 26, no. 10, pp. 5032–5042, Oct 2017. [13] MLTransferLearning guidelines: https://stanford-pilot.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=d0f21cb7-7111-43d0-a647-ac4c017dfba0

Appendix

You can write math equations as follows: $y=x+5$

You can include images as follows (you will need to upload the image first using the toolbox on the left bar, using the "Upload file" link).