Hypercube Waveband Registration

Introduction

Hyperspectral Imaging

Hyperspectral imaging allows us to visualize a vast portion of the electromagnetic spectrum and detect information that would otherwise be invisible to the naked eye. Hyperspectral sensors are able to extend its spectral footprint significantly beyond the visible red, green, and blue bands into the infrared region. The availability of sensor data at a large number of spectral bands generates a three-dimensional hypercube whose levels represent the different spectral bands and whose values at each level represent the sensor-detected light intensity at the corresponding pixel location at that specific band. Analyzing information presented by these additional spectral bands may lead to additional insight on a particular object or scene. For example, different materials have different spectral signatures. The existence of certain material in a scene or an object may be much more evident in a particular waveband than others.

HySpex hypersectral imaging system

Available Hyperspectral System

Hyperspectral cameras are defined by their spectral and spatial resolutions. The hyperspectral imaging system used to capture data for this project consists of two individual hyperspectral cameras mounted side-by-side. One is responsible for the visible and near-infrared (VNIR) portion of the spectrum and operates in the range of 400 to 1000 nm. The other captures the short-wavelength infrared (SWIR) portion and works in the 1000 to 2500 nm range. While there are many obvious advantages of using a hyperspectral imaging system, there are imperfections inherent in the design of the system. In this project, we consider the misregistration (pixel misalignment) between bands within either the VNIR or SWIR camera, as well as misregistration between the two cameras. We make an effort to quantify the extent of the misregistration using the mutual information measure in probability and information theory. In the consideration of misregistration, we also encounter other imperfections in the spectral images, of which we pay particular attention to how noise at certain bands affects the processing of the hyperspectral data.

Methods

Urban scene from hyperspectral remote sensing

Feature vs. Area Based Approach

There exist two general categories of registration techniques: area-based and feature-based. Feature-based approaches work under the assumption that the images in question contain distinct features such as corners and edges that can be easily detected using methods such as Harris Corner Point or Zero-Cross Edge Detection. In the urban scene shown, for example, feature-based methods generally work better in areas with houses and worse in the areas with vegetations. On the other hand, area-based methods rely on the statistical correlations of pixel intensities between corresponding regions of the two images. Cross correlation is a popular area-based measure used to compute the similarity between two images. In this project, we focus on the superior area-based mutual information similarity measure. Mutual information, as proposed by Viola, requires no information on the object's surface properties besides its shape, and has been shown to be robust against variation in illumination. In other words, mutual information works well on scenes that lack distinct features, and when gradient-based corner and edge detection methods fail. Mutual information is also superior to correlation-based methods because it does not degrade with contrast reversal, in which case correlation-based methods provide no well-defined optimal solution.

Mutual Information

Mutual information measures the dependency and redundancy between two images, each represented by a two-dimensional matrix filled with intensity value at each pixel location (matrix entry). By letting the two images to be registered be two random variables $A$ and $B$, the mutual information between these images can be calculated by finding the marginal probabilities $P_{A}(a)$ and $P_{B}(b)$ with the joint probability $P_{A,B}(a,b)$. In the calculation of $I(A,B)$, $a$ denotes a certain intensity value in Image A, and $b$ denotes intensity value in Image B.

$I(A,B)=\sum _{a,b}P_{A,B}(a,b)log{\frac {P_{A,B}(a,b)}{P_{A}(a)P_{B}(b)}}$

The marginal and joint probabilities are as defined as

$P_{A,B}(a,b)={\frac {h(a,b)}{\sum _{a,b}h(a,b)}}$

$P_{A}(a)=\sum _{a}P_{A,B}(a,b){\frac {}{}}$

$P_{B}(b)=\sum _{b}P_{A,B}(a,b){\frac {}{}}$

While the calculation of the marginal probabilities $P_{A}(a)$ and $P_{B}(b)$ is quite straightforward, the computation of the joint probability $P_{A,B}(a,b)$ brings out a two-dimensional joint histogram matrix $h$ used to tabulate the occurrences of different intensity values across the two images. Each entry $h(a,b)$ in the joint histogram matrix denotes the number of corresponding pixels with intensity value $a$ in Image $A$ and intensity value $b$ in Image $B$.

$h={\begin{bmatrix}h(0,0)&h(0,1)&\ldots &h(0,N-1)\\h(1,0)&h(1,1)&\ldots &h(1,N-1)\\\ldots &\ldots &\ddots &\ldots \\h(M-1,0)&h(M-1,1)&\ldots &h(M-1,N-1)\\\end{bmatrix}}$

The dependence on only intensity values categorizes mutual information as an area-based (or intensity-based) method that works well when there is a lack of features on the image. The inclusion of joint probability statistic helps make mutual information robust against variation in illumination and random noise.

Bosch painting in RGB from Cantor Museum

MATLAB^® Implementation

During the course of the project, scripts have been written to evaluate the registration of captured hyperspectral images and register (align) the different wavebands within each hypercube if necessary. Among the scripts that were written based on the interest of different individuals as related to hyperspectral research stands the core of this package: mutualInformation.m and jointHistogram.m used to efficiently calculate the mutual information between two images. Other functions, including one used to register two images, build upon the result from the mutual information calculation. Here we attempt to highlight some important steps in mutualInformation.m and jointHistogram.m.

In mutualInformation.m, the pixel intensity values are first normalized and quantized into a number of bins based on user specification. For the hyperspectral Bosch painting data used in this project, we use 8-bit sampling of pixel intensity, resulting in 256 different intensity levels from 0 to 255. Normalization and quantization are not required by the nature of the mutual information formulation. However, they help simplify and standardize the process of numerical computation. Following this standardization, marginal probability density functions can be easily obtained with built-in histogram functions, and joint probability density function can be computed with only a few manipulations. It is important to note here that because we are implementing mutual information with bins, we should make sure that the bins are not sparsely populated in order to produce good results. The data used in generating these distributions should be large enough to avoid producing a sparsely populated histogram.

jointHistogram.m computes the joint probability density function $h$ of Image $A$ and Image $B$. The algorithm is as followed: For all pixels in Image $A$ with a normalized quantized value of $a$, find the spatially corresponding pixels in Image $B$ and obtain the histogram of intensity value $b$ among these pixels. This algorithm is iterated over all possible values of $a$. For the discrete case of joint probability distribution, $P(A,B)=P(B|A)P(A)=P(A|B)P(B)$. Creation of the joint histogram $h$ basically implements $P(B|A)$ part of the equation. Dividing $h$ by the exhaustive sum over all its entries results in $P(A,B)$.

Here are a sample set of histogram generated by the program on two wavebands of a hyperspectral image.

Bosch painting in different spectral bands

Currently, the following functions are included in the attached registration software:

• mutualInformation.m: Computes the mutual information of two grayscale images.
• jointHistogram.m: Computes the joint histogram to two normalized quantized grayscale images.
• imageRegistration.m: Locates the optimal rigid transformation that maximizes the mutual information between two images.
• findHighSNR.m: Identifies the waveband with the highest normalized mean SNR.
• scanLineRegister.m: Analyzes the longitudinal variation of misregistration across the image.
• crossBandMI.m: Compares the mutual information between different sets of wavebands.
• crossBandRegister.m: Compares the misregistration between different sets of wavebands.

Results

The availability of the mutual information algorithm opens the door for various registration-related applications. For this project, we can evaluate the inherent quality of registration within and between the SWIR and VNIR cameras. In addition, since the hyperspectral system uses two separate cameras to cover the desired spectral bands, the resulting hypercube image is by nature multimodal, used to describe images captured by different lenses, at different angles, and with different perspectives. There exists a non-rigid, non-affine, projective transformation between images from the two cameras.

Noise Consideration

Before digging into mutual information and registration, it is important to consider the noise properties of the hyperspectral imaging system. In particular, it is known that low light level and low sensitivity often lead to high level of photon noise. Photon noise may be more pronounced in some wavebands over others, but acts as the fundamental limit on the signal-to-noise ratio (SNR) of an image. The original hyperspectral data that we have on-hand are photon counts $n$ for each pixel at each waveband. Photon counts are rather useful in estimating SNR. Because the arrival of photons and $n$ are Poisson distributed, $SNR\approx {\sqrt {n}}$.

Using a white point on the image, we can plot the photon count as a function of waveband and observe the change in SNR across different bands. For the VNIR camera, data suggest that its SNR peaks at around 630 nm, which is the center of the VNIR spectrum. For the SWIR data, SNR peaks at around the wavelength of 1250 nm, which is toward the lower wavelength end of the SWIR spectrum.

VNIR White Point Spectral Profile

SWIR White Point Spectral Profile

Knowing that the SNR peaks at 630 nm for VNIR and at 1250 nm for SWIR, we can safely use these two bands as the reference bands in the registration works that follow, rest assured that we have taken at least one step in minimizing the impact of noise. Without having to run a denoising algorithm through every single band, capturing the highest-SNR band is a intuitively viable option. While both 630 nm and 1250 nm serve as the local peak SNR band for their corresponding spectrum (VNIR or SWIR), 1250 nm serves as the global peak SNR band when we are dealing with all available spectral bands at the same time.

Unimodal Waveband Registration

Using 1000 nm and 1350 nm as reference wavebands for their corresponding spectrum, we evaluate the quality of registration between bands for the VNIR and SWIR cameras individually. The algorithm attempts to register each band in the hypercube with the reference band and outputs the xy-translation necessary to bring the band in alignment with the reference. A negative value in the x-direction represents a left shift, while a negative value in the y-direction denotes a up shift. We need to keep in mind that some bands may have a low SNR that would negatively affect the registration result. However, we have taken the step to ensure that at least the reference band has high SNR. This does not guarantee that noise would not play a role in the experimental results, but only helps to minimize it.

Comparing all the VNIR wavebands to the high SNR VNIR band, we find that there is no misregistration among those bands, as shown in the VNIR crossband registration plots. This is consistent with the claim that any misregistration should be at sub-pixel level. In addition, we also compared all the SWIR wavebands to the high SNR SWIR band and found that misregistration only occurs at the last spectral index with an x-offset of only 1 pixel. This suggests that the SWIR camera is designed to flavor the lower-end spectrum toward 100 nm. It matches our expectation that any misregistration should happen on the edges of the spectrum. The result is mostly consistent with the claim that any misregistration offset should be less than 1 pixel. It would be interesting to examine the root cause of this 1-pixel offset, for it be noise, imperfection in data processing, or other factors. Initial investigation reveals that it might have been an artifact of the algorithm. The algorithm uses a portion of the entire image for comparison in order to speed up the computation. The size of this portion can be adjusted with a scaling factor. The result becomes less accurate as this scaling factor increases.

Bimodal Waveband Registration

When we attempt to register wavebands in one camera to a reference band from another camera, the problem becomes a bimodal one because the two cameras may have different optical properties and may be set up at different angles to capture the same object or scene. Because of the bimodal nature of the setup, there is high probability that the same object or scene captured by the two cameras may be misaligned. In this project, we attempt to register the VNIR wavebands to the global high SNR band from the SWIR camera at around 1350 nm.

The difficulty in performing this bimodal registration stems from fact that VNIR and SWIR are essentially two separate data hypercubes with different spatial resolutions. Either spatial down-sampling or up-sampling is required as a pre-processing step. Here we elected to down-sample the VNIR data by a factor of around 4 using bicubic interpolation to bring it into the same spatial resolution as the SWIR data. In addition, since the cropping of the two data sets were done individually in two separate occasions, there is an inherent xy-offset between VNIR and SWIR data. Knowing that any misregistration across the entire hyperspectral imaging system should be at most a few pixels or fewer, we need to cleverly eliminate any offsets reported by the registration tool that are common to a large majority of pixels. The remaining offsets represent the actual misregistration.

We noted that there is a spike in misregistration around spectral of index of 38 and attempted to analyze how these bands are special compared to others. So far we have not been able to explain this phenomenon. Because there is also uncertainty in the inherent misregistration between the two cameras, data obtained in these plots have been called into question. That's why we have collected data in a more accurate manner from the raw hyperspectral data file for re-running this experiment.

Scan Line Variations

It is also interesting to look at the registration between bands within different groups of scan lines and examine if there is a systematic variation from one end of the the field of view to the other. This amounts to scanning from the top of our image to the bottom and looking at the euclidean offset as a function of scan line. Scan line variations can be analyzed in both the unimodal and bimodal cases. Here we consider groups of 100 scan lines.

For the unimodal case, we scan and compare between the first and last VNIR band. The former has the lowest SNR in VNIR, and the latter has the highest. We also scan and compare between the 58th and last SWIR band. The former has the highest SNR in SWIR, and the latter has the lowest. The results shown below were a little surprising. They tell us that even though the entire image registers well as a whole, corresponding groups of scan lines do not. Moreover, it is difficult to spot any patterns on the plot, and there is no evidence of bias in terms of spatial location toward any scan line groups. Looking at the euclidean offset, it seems that group misregistration is more sever in SWIR than in VNIR. Otherwise, everything else seems sudo-random. The results shown here deserve further investigation. With the availability of the newly extracted data, we are hopeful that we will get some results that are more inline with the physical specifications of the cameras.

Conclusions

Mutual information is a powerful measure for measuring the similarity between two images or two spectral bands in the context of waveband registration. It is based on the dependency and redundancy of pixel intensities for the two images. It is robust to noise and works even if only a small portion of the two images are similar. Note that the images must be large enough to avoid a sparsely populated histogram in the computation of mutual information.

In using mutual information for waveband registration, we found that the spectral bands align well within each camera, and most misregistrations across the two cameras are on the scale of only a few pixels. As an extension of this project, we can take advantage of the difference in spatial resolution between the VNIR and SWIR cameras to detect any sub-pixel misregistration.

What was most surprising is the misalignment between groups of scan lines, and on top of that, the lack of misalignment as a function across the field of view of the camera. If this phenomenon is in fact true and verified, it suggests that a possible solution of registration would involve a combination of both feature and area-based methods. One intuitive approach would be use mutual information as a feature matching algorithm in cases where feature points cannot be readily extracted. Mutual information would identify corresponding patches of the two images to be registered, and using the coordinates of these patches we can compute either an affine or projective transformation matrix that maps in between the two images.

As I continue this project, the next step would be to add a stochastic gradient optimization to the registration algorithm to efficiently search for the optimal misregistration offset. With this optimization on-hand, we can then explore ways to implement combined feature and area-based methods. Of course, we would also re-run the experiments previously discussed on this page with the newly extracted data to verify the integrity of our results.

References

1. HySpex Main Specifications. Norsk Elektro Optikk A/S (NEO).
2. Skauli, Torbjorn. Hyperspectral Sensor Technology. Norwegian Defence Research Establishment.
3. Viola, Paul A. Alignment by Maximization of Mutual Information. Massachusetts Institute of Technology. 1995.
4. Kern, Jeffrey P, Marios Pattichis, and Samuel D. Stearns. Registration of Image Cubes Using Multivariate Mutual Information IEEE 0-7803-8 104-1 (2003): 1645-1649.
5. HyperCube Spectral Research Program. Computer software. Vers. 10.6. US Army Geospatial Center.

Appendix I

Scripts

File:HypercubeWavebandRegistration.zip

Data

File:BoschSWIR.zip
File:BoschVNIR.zip

Appendix II

A majority of the project was carried out by Steve Dai, with contributions from Qiyuan Tian on the understanding of the algorithm, testing methodology, and related discussions.

Link to our group's other project: A Versatile Image Fusion Method