Image classification with a five band camera

Introduction

Image classification is a central topic in computer vision in which images are lumped into categories via comparison and prediction. Many state-of-the-art classification systems follow a somewhat ubiquitous procedure of feature extraction via dense grid descriptor sampling, coding into higher dimensions, pooling, and finally classification. Integrated computer vision techniques like object and scene recognition are also gaining traction in consumer-grade electronics. Two now common examples of this in digital cameras are (1) scene recognition, in which the appropriate aperture, shutter speed, and nonlinear gamma settings are selected and (2) face detection, which is used to determine the appropriate autofocus parameters and locations for redeye removal.

Aside from pure software-based developments, data acquisition is central to the heart of this problem. Most silicon based digital camera sensors use CCD or CMOS sensor chips that are inherently sensitive to a broad spectrum of light, starting at short visible wavelengths, and reaching into the near-infrared. Many of these sensors have a series of filters that discretize or remove light, ending with three channels of digital data, each corresponding to an RGB color channel: red, green or blue. Such cameras are often termed "three band". It is argued here that a camera with more sensitivity over a broader spectrum of wavelengths could be helpful to many computer vision techniques.

To test this hypothesis, a set of 72 images of four different pieces of fruit were taken on a prototype Olympus brand five band camera donated to Stanford's Center for Image Systems Engineering (SCIEN) lab, sampled using SIFT feature descriptors, and compared using RANSAC.

Background

In 2011, Brown and Susstrunk [1] used a modified single lens reflex (SLR) camera to capture several hundred three band color and NIR image pairs to show that the addition of NIR image information leads to significantly improved performance in scene recognition tasks. The authors tested multiple feature descriptors in their analysis, including color-specific SIFT, multispectral SIFT, GIST (feature descriptors generated using three scales and eight orientations per scale), and HMAX (a hierarchy of filtering and max-pooling operations computed independently and concatenated for each band). In each case, the authors found a general trend that adding more information lead to better recognition performance, and that the SIFT based descriptors showed the greatest improvements (recognition rates of 59.6% vs. 73.1% for RGB versus RGB + NIR, respectively).

In 2012, Namin and Petersson [2] presented a method of distinguishing between different materials occurring in natural scenes using a seven band camera. Instead of using SIFT for object detection and localization, the authors used a texture based approach considering Fourier spectrum features from gray-level co-occurrence matrices (GLCMs), and classifiers built using support vector machines (SVM) and AdaBoost. Such an approach considers the entire image as one class of data, such as an image of grass or bushes, rather than identifying certain objects in an image. They found that adding extra spectral information helped them discriminate properties better, achieving an average classification accuracy of 91.9% and 89.1%, respectively for a ten class problem.

In support of recent success of image classification using multispectral imaging, many groups have begun adapting commonly used techniques for multi-band image data. For example, groups Saleem and Sablatnig [3] Xiao, Wu, and Yuan [4] developed adapted feature sampling and extraction techniques for multi-band images. As another example, groups Raja and Kolekar [5] and Salamanti, Germain, and Susstrunk [6] both developed improved shadow removal and image restoration techniques based on the use of extra spectral information.

Methods

Instruments

The camera used in this experiment was a prototype

Results

- Organize your results in a good logical order (not necessarily historical order). Include relevant graphs and/or images. Make sure graph axes are labeled. Make sure you draw the reader's attention to the key element of the figure. The key aspect should be the most visible element of the figure or graph. Help the reader by writing a clear figure caption.

Conclusions

- Describe what you learned. What worked? What didn't? Why? What should someone next year try?

References

[1] Brown, M.; Susstrunk, S., "Multi-spectral SIFT for scene category recognition," Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on , vol., no., pp.177,184, 20-25 June 2011, doi: 10.1109/CVPR.2011.5995637. < http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5995637&isnumber=5995307>

[2] Namin, S.T.; Petersson, L., "Classification of materials in natural scenes using multi-spectral images," Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on , vol., no., pp.1393,1398, 7-12 Oct. 2012, doi: 10.1109/IROS.2012.6386074. < http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6386074&isnumber=6385431>

[3] Saleem, S.; Sablatnig, R., "A Robust SIFT Descriptor for Multispectral Images," Signal Processing Letters, IEEE , vol.21, no.4, pp.400,403, April 2014, doi: 10.1109/LSP.2014.2304073. <http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6730675&isnumber=6732989>

[4] Yang Xiao; Jianxin Wu; Junsong Yuan, "mCENTRIST: A Multi-Channel Feature Generation Mechanism for Scene Categorization," Image Processing, IEEE Transactions on , vol.23, no.2, pp.823,836, Feb. 2014, doi: 10.1109/TIP.2013.2295756. < http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6690151&isnumber=6685907>

[5] Lloyds Raja, G.; Kolekar, M.H., "Illumination normalization for image restoration using modified retinex algorithm," India Conference (INDICON), 2012 Annual IEEE , vol., no., pp.941,946, 7-9 Dec. 2012, doi: 10.1109/INDCON.2012.6420752. < http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6420752&isnumber=6420575>

[6] Salamati, N.; Germain, A.; Susstrunk, S., "Removing shadows from images using color and near-infrared," Image Processing (ICIP), 2011 18th IEEE International Conference on , vol., no., pp.1713,1716, 11-14 Sept. 2011, doi: 10.1109/ICIP.2011.6115788. <http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6115788&isnumber=6115588>

Appendix I

Presentation

- Upload source code, test images, etc, and give a description of each link. In some cases, your acquired data may be too large to store practically. In this case, use your judgement (or consult one of us) and only link the most relevant data. Be sure to describe the purpose of your code and to edit the code for clarity. The purpose of placing the code online is to allow others to verify your methods and to learn from your ideas.