ChoudharyRajbhandarySutton

From Psych 221 Image Systems Engineering
Jump to navigation Jump to search


Background

Hyperspectral images contain hundreds of spectral samples from across the electromagnetic spectrum per pixel. Appropriate rendering of such information in the RGB or grayscale monitor can provide extra information about the images which have been exploited in applications such as agriculture, minerology, surveillance and surgery. The spectral bands of the images must first be projected to a lower dimensional space, typically the RGB color space or grayscale of a monitor for the display.

Figure 1 - Realization of Hyperspectral Images in 3 dimensions. x and y determine the size of the image being captured. z depends on the number of unique wavelengths bins that have been acquired

Biomedical applications of hyperspectral images include real-time and non-invasive diagnosis of skin cancers [1], features to enhance contrast between different organs, and determination of blood concentrations to identify regions of ischemia [2]. Our aim is to choose three to four spectral bands, which provide maximum contrast, to apply these algorithms in retrofitted conventional cameras.

Approach

The project presents various methods and their results of using optimal information from hyperspectral images to be displayed in a RGB color monitor with highest contrast. The application of hyperspectral images so far has been to reveal patterns that are not visible with normal vision. We are proposing that we can use information from hyperspectral images of pig organs to select about 3-4 bands that are not necessarily in the conventional RGB range to display images on a RGB monitor, and thereby providing enhanced contrast between different organs. Taking real time hyperspectral images during surgery to show high contrast between different organs would be ideal; however, the current state of technology poses several limitations. Hyperspectral images themselves are taken by stitching a sweeping scan of one-dimensional images (aka push-broom scanning technique). Hence, using such a camera during surgery can be impractical. Additionally, the camera is much more expensive compared to conventional camera, and therefore not likely to be used by regular operating rooms. However, if the color filters on top of the sensors can be strategically chosen to display maximum contrast between various organs, a camera can be both easily operable and affordable while still providing sufficient contrast. Different organs each have their own characteristic reflectance curve. Hence, reflectance is used as differentiating parameter. Since hyperspectral images acquired in this project contain 148 bands between approximately 400-1000 nm, multitude of bands provides opportunity to parameterize reflectances at various bands, or their combination to obtain maximum distance between the observed organs. The project aims at showing that strategically selecting color filters pass bands in conventional camera can allow rendering of organs during surgery to provide maximum contrast. We are proposing to create a feature that would render the four organs studied in distinct colors making both organs and their boundaries more conspicuous.

Project Aims

  • Aim 1. Evaluate appropriate methods to display the four pig organs (colon, liver, spleen, stomach) with maximum contrast using hyperspectral images (all 148 bins).
  • Aim 2. Develop a method to select a limited number of wavebands that provide maximum contrast between different images
  • Aim 3. Display the post-processed image to display in RGB monitor to show distinction between different organs.

Hyperspectral Image Rendering

Reference Organ Acquisition

Figure 2 - Demonstration of how hyperspectral images from pig surgeries were cropped to isolate organ tissues for further analysis.
Figure 3 - Average reflectance values over 148 wavelength bins for four isolated pig organ tissues: colon, liver, spleen, stomach

A pig was anesthetized and hyperspectral images of various organs were taken using the NORSK Elektro Optikk AS hyperspectral camera. The hdr and img raw data files of four organs – liver, spleen, colon and stomach -- are parsed through data extraction script to obtain photons, illuminants, and other parameters of interest. We isolated representative 30x30 pixel region of interest (RoI) of four organs -- colon, liver, spleen and stomach as seen in Figure 2. We avoided using parts of the organ that had an obvious aberration due to non-uniform illumination. For example, the image below shows a section of the liver that we could have chosen; however, due to the aberration on the top left of the chosen section, we would have to move our selection around to choose another patch.


For each chosen patch, we can isolate the the radiance and illuminant intensities (in photons) at each pixel. We can then calculate the mean reflectance of the organ for all the 148 wavelength bins by averaging over the 30x30 pixel image. Figure 3 shows a plot of the reflectances vs wavelength for the 4 organs. Reflectances were normalized for each wavelength bin (scaled such that they covered a range of 0 to 1). Each organ does have a unique reflectance spectrum. During our experiments we attempt to exploit these variances in multiple ways. There is one caveat that we must consider. These spectra are mapped over 148 wavelength bins. While there may be sufficient data in the hyperspectral realm, we would ideally like to choose only a few bins so that the our rendering techniques could be performed with a regular camera instead of requiring hyperspectral images. Using the methods outlined in a later section, we decide the 3-4 wavelengths that exhibit the maximum variance across these wavelength bins, and thus lead to a higher contrast in organ tissues across the rendered images.


Simply selecting the photon intensities of the chosen wavelengths and mapping them to RGB does not yield a good contrast across the images, this lead us to use some post-processing techniques for contrast enhancement and organ identification in order to utilize the variance across the reflectances. In most cases, the post processing would result in a matrix with the RGB intensities for each pixel, and we would use the MATLAB function “image” to display the images. In some case, we would need to scale are intensities so as to ensure that they were in the (0,1) range for “image” to work.

Method I: Maximum Standard Deviation and PCA Weighting

In this method, we treat all the wavelength bins as independent, and find the standard deviation for the normalized values in each wavelength bin. Figure 4 shows the plot of standard deviation vs wavelength (blue curve). We can also perform the Principal Component Analysis (PCA) using the normalized mean reflectances as sample points, and the wavelengths as variables. The first component is the one that has the maximum variance and the wavelengths that have the maximum co-efficients for the first component contribute the most to the variance. The plot of the coefficients of the first component vs the wavelengths is also shown below (green curve).

Figure 4 - Blue: Normalized Standard Deviation of the four pig organ tissue reflectance values for each wavelength bin. Green: Coefficients from the first principal component after PCA on the 148 wavelength four pig organ reflectance values

We identified 720,616 and 480 nm as the wavelength bins having the maximum coefficients in the green curve and decided to map the photon intensities for these wavelengths to R,G, and B respectively (scaled to obtain values between 0 and 1 for “image” to render them). Figure 5 shows the difference between the standard RGB (620,564,448 nm) and our chosen bins (720,616,480 nm)

Figure 5 – Left: Radiance intensities from wavelength bins 620, 564, and 448nm assigned to RGB. Right: Photons from wavelength bins 776, 604, and 496nm assigned to RGB.

It can be noticed from Figure 5 that the contrast is not greatly enhanced when directly mapping photon intensities of the selected bins to RGB. This is because we are not really using the reflectance differences while rendering the images. The following methods perform Post-Processing to enhance contrast.

Method II: Principal Component Analysis (PCA) Based Algorithm

To begin our investigation, we isolated 30x30 pixel images of each organ. The organs used in this study were the colon, liver, spleen, and stomach. These organ tissues are good specimens for our experiments because under standard displays the liver and spleen show very little contrast, and similarly the stomach and colon show little distinction between each other.

Figure 6 – 30x30 pixel Isolated Pig Organs Rendered in RGB (620, 564, and 448nm)

We investigated many post-processing techniques to render these organs with high contrast. Our initial investigations involved performing principal component analysis (PCA) on the raw hyperspectral pig organ data. PCA is a mathematical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. In our experiment, the observations are the intensities of each hyperspectral wavelength bin at each pixel. One additional step we took before calculating the PCA was to first normalize the data using a standard score. This standard score indicates how many standard deviations our observation is above or below the mean. It is a dimensionless quantity derived by subtracting the population mean from an individual raw score and then dividing the difference by the population standard deviation.


Eq. 1 – Standard Score. represents the score, the observation, the mean of the observations, and the standard deviation of the observations.


We analyzed the data in two ways: analyzing using radiance in photons; analyzing using reflectance. First, we examined the principal components computed using raw radiance values. We visualized the first vector of principal components to find the bins that have the maximum weight.

Figure 7 - Normalized coefficients of the first principal component of the isolated organ tissue radiance intensities from wavelength bins 400 to 1000nm.. Top bins were: PCA_R=776nm,PCA_G=604nm,PCA_B=496nm,PCA_IR=912nm.

We anticipated that the principal components with the highest weights would contribute the most to the variance. The PCA transformation is defined in such a way that the first principal component has the largest possible variance (that is, accounts for as much of the variability in the data as possible), and each succeeding component in turn has the highest variance possible under the constraint that it be orthogonal to (i.e., uncorrelated with) the preceding components. We investigated using the principal components as a multiplicative transform for our RGB values. For rendering a 2-D color image, we must supply an i-j-3 matrix, where i and j determine the size of an image, and the last dimension defines to which color the intensities of the i x j array are mapped: R, G, or B. In rendering new images, we use the first three principal components to distort the radiance data to produce new R, G, and B intensities (Eq.2).


Eq. 2 – Rendering New RGB from Top 3 Principal Components


We relied on the idea that the coefficients of each principal component were orthogonal. Thus, when we multiply the hyperspectral radiance data by the top three PCA coefficients we should obtain 3 intensity spectrums. These intensities were then mapped as RGB values and imaged.

Figure 8: Left: Control: raw isolated organ tissue radiance intensities from wavelength bins 620, 564, and 448nm to RGB. Middle 3: The intensities (in gray-scale) of all 148 wavelengths of radiance data scaled by the first 3 principal component coefficients. Right: The prior three intensities assigned to RGB

The newly equated RGB values in Figure 8 provide a much higher contrast than the standard RGB intensities used previously. Now, the colon and stomach which both look dark red under standard RGB are drastically transformed to look blue and yellow, respectively. However, this transform utilizes all 148 wavelength bins of the hyperspectral image. We aimed to find a solution that only involves acquiring 3-4 wavelength bins of data, while still maintaining sufficient contrast between similar looking organs. So, we continued by performing a similar transform, but now composing our transform matrix with only the wavelengths with the highest variance. To do this, first the photons from these wavelengths were isolated.



Recall that were the wavelengths obtained from Figure 7. The 148-coefficient matrix from before was then fragmented, and a new transform matrix was formed. Now only coefficients pertaining to the top 4 highest weighted are used. This 4-coefficient transform can now be used to distort our proposed 4 bin radiance dataset.


Eq. 3 – Obtaining new coefficient matrix by extracting the top 4 wavelength bins with the highest weighted coefficients from the original 148-coefficient transform.


Finally, the radiance data from the new wavelength bins were transformed using the first three principal components of our new coefficient matrix. These intensities were then assigned to RGB.


Eq. 4 – Rendering New RGB from Top 3 Principal Components from new transform matrix


Figure 9: Left: Control: raw isolated organ tissue radiance intensities from wavelength bins 620, 564, and 448nm to RGB. Right: The intensities of the 4 wavelengths of radiance data scaled by the first 3 principal component coefficients and assigned to RGB.

The new PCA coefficient transform matrix is now much more realistic. Typical cameras can easily acquire photons at 3-4 wavelengths. Although not as great as the previous 148 coefficient transform, the new RGB image in Figure 9 acquired from the 4 coefficient transform still provides sufficient contrast to easily distinguish between similar organs such as the liver and spleen which look dark purple before and green and navy blue, respectively, after transformation. The previous studies have all been performed on isolated tissue samples. The images, when rendered, were each scaled so that the R, G, and B channels filled the entire range from 0 to 1. We wanted to utilize the entire color gamut to display our isolated images with the greatest contrast. We continued this approach on images without isolated tissues.


The same 148-coefficient and 4-coefficient PCA transforms were performed on the image of the pig stomach.The newly acquired images, to our dismay, do not enhance the contrast between organs as greatly as when the transform was performed on isolated organs. There is some additional contrast seen between the stomach and intestines after the 148-wavelength transformation, but not to the degree that we had predicted. The 4-wavelength transform shows similar color modulations, but the contrast, understandably, is far less than that of its 148-coefficient counterpart.


We surmised that image scaling brought about this problem. During the previous studies on isolated tissues, we were able to multiply by our coefficients and scale from 0 to 1 in each channel thereby increasing the contrast significantly. Now we are not able to scale the tissues to the same level of contrast because the extraneous objects, such as the clinician’s glove or white coat have hindered our scaling. While the organs may have been distorted to have different properties they cannot be scaled as drastically and therefore do not exude as extreme contrast as previously observed.

Figure 10: Left: Control: raw pig stomach radiance intensities from wavelength bins 620, 564, and 448nm to RGB. Middle: The top three intensities of the 148 wavelengths of radiance data scaled by the 148-coefficient transform and assigned to RGB. Right: The top three intensities of the 4 wavelengths of radiance data scaled by the 4-coefficient transform and assigned to RGB. 4 selected wavelengths: 912, 776, 604, 496 nm

In our second series of experiments, we decided to analyze the reflectance at each pixel rather than the photon intensities. We had discovered that the reflectance spectra of each organ differed quite significantly, and if we could capture these wavelength bins rather than the radiance bins that had the greatest variance, we may be able to better enhance contrast when displaying images to RGB. First, we examined the principal components computed using raw reflectance values. We visualized the first vector of principal components to find the bins that have the maximum weight.

Figure 11 - Normalized coefficients of the first principal component of the isolated organ tissue reflectance values from wavelength bins 400 to 1000nm. Top bins were: PCA_R=756nm,PCA_G=604nm,PCA_B=500nm,PCA_IR=912nm.

We performed the same principal component transforms using all 148-wavelength bins and then 4-wavelenght bins on isolated and un-isolated organ images. The results are shown in Figures 12, 13, and 14.

Figure 12: Left: Control: raw isolated organ tissue reflectance values mapped from wavelength bins 756, 604, and 500nm to RGB. Middle 3: The intensities (in gray-scale) of all 148 wavelengths of radiance data scaled by the first 3 principal component coefficients. Right: The prior three intensities assigned to RGB


Figure 13: Left: Control: raw isolated organ tissue reflectance values mapped from wavelength bins 756, 604, and 500nm to RGB. Right: The intensities of the 4 wavelengths of radiance data scaled by the first 3 principal component coefficients and assigned to RGB.


Figure 14: Left: Control: raw pig stomach reflectance values from wavelength bins 756, 604, and 500nm to RGB. Middle: The top three intensities of the 148 wavelengths of radiance data scaled by the 148-coefficient transform and assigned to RGB. Right: The top three intensities of the 4 wavelengths of radiance data scaled by the 4-coefficient transform and assigned to RGB. 4 selected wavelengths: 912, 756, 604, and 500nm.

Similar to the results obtained from radiance intensity transforms, the reflectance transforms did not sufficiently enhanced organ contrast in un-isolated images as shown in Figure 14.

Method III: Raw Reflectance Value Rendering

A simple approach to use the reflectance values to enhance contrast is to use them as the intensity itself. We are no longer concerned with the photon intensity values, but directly plot the normalized reflectance values (scaled to between 0 and 1 for rendering in MATLAB). Thus for our chosen wavelengths of 720, 616 and 480 nm (from Method I), we directly map the reflectance values to the R, G, and B intensities respectively. Since the wavelength bins are chosen to given the maximum variance between the reflectance values, mapping them to RGB intensities should give us some contrast. The images below show the standard RGB rendering of photon intensities vs the rendering using the raw reflectance values for the 4 organs (from top to bottom, colon, liver, spleen stomach—the left is standard RGB, and the right is the raw reflectances rendered).

Figure 15 - Comparison between the image rendering of conventional RGB values to the raw reflectance values directly mapped to RGB values. The first column shows the images rendered by mapping photon intensities to RGB and the second column shows images of raw reflectance values mapped to RGB. Organs from top row to bottom are colon, liver, spleen and stomach


This approach need not be done on a pixel by pixel basis. We could achieve some sort of smoothing by dividing the image into a grid and averaging reflectances over each section of the grid. For example the images above actually have the reflectance values averaged over a 2x2 grid. This griding can be potentially used to average out noise from the image at the cost of making it look pixelated. The images using a 5x5 grid are shown below. However averaging out the reflectance values is not beneficial from a contrast point of view and so we stick to the first scenario.

Figure 16 - Reflectance values averaged over 5x5 grids of pixels and directly mapped to RGB. Top left- Colon, Top right - Liver, Bottom Left - Spleen, Bottom Right - Stomach

This approach does suffer from the drawback that if certain images are far apart in the RGB space, they are not necessarily perceptually different. In order to quantify and enhance perceptual differences, we had to move into CIELab space. One last thing that we tried in this approach was to try and identify tissue patches by comparing the reflectance values of a pixel to the reflectance value of the organs. This did not really give us any useful results mainly because the threshold values for the difference in reflectances were not easy to identify. We invariably would end up identifying random parts of the picture (like the hand or the skin) as an organ. A better approach for this is the correlation based algorithm explained in Method V.

Method IV: Contrast Enhancement in CIELab Colorspace

CIELab provides us with a metric to determine perceptual difference quantitatively. We wanted to utilize this metric to enhance the contrast of the images by essentially using the following flow:

The RGB intensities are given by the scaled photon intensities at 720, 616 and 480 nm respectively. We assume the standard display metrics used in ISET to perform the transformation from RGB to CIELab. After plenty of iterations of trying several transformations, the following works the best:

Here, α is empirically found to be 100, (in the code we calculate the reflectance as Radiance/Illuminant, and the values we got were scaled by a factor of 100 from the mean_reflectance values computed in ISET). This simple transformation itself gives us a pretty good contrast between the organs. Figure 17 shows the before (left) and after (right) of the organ tissues and the deltaE values (plotted as contours depicting gradients across the pixels). Note the change in scale of the contour maps in the “after” plot.

Figure 17 - Left: Organ Tissues and deltaE before CIE Lab transformation, the top left plot renders the tissues by mapping the intensities at 720 nm to R, 616 nm to G and 480 nm to B. Right: after Lab transformation of the organ tissues. The deltaE values are plotted as contours depicting gradients across the pixels.

Given that this simple transformation gives such good results, if we had more time, we would have liked to try other non-linear transformations to further improve the quality of the contrast. We believe that going to CIELab has tremendous potential in terms of a quantitative contrast enhancement methodology. However, due to the computation intensive nature of this transformation, we would like to perform the transformations only on sections of the image which have organs present, or in other words, we would like to identify the organs in the image and then proceed towards contrast enhancement in those areas. To achieve this, we use a correlation based algorithm described in the next section.

Method V: Reflectance Correlation Scored Organ Discrimination

We used correlation based algorithm to categorizes pixels of an image into either different organs or leave them unaltered if not within certain threshold value. Before computing the correlation, we observed erratic patches and white reflection spots along the pictures. To reduce these and other noise, we used a bilateral filter. Bilateral filtering was proposed by Tomasi and Manduchi in 1998 as a non-iterative method for edge-preserving and noise reducing smoothing filter [3] [4]. Using reference reflectance of the four reference organs, we calculate the cross correlation between the pixel under consideration to these four reference organs. We then calculated the absolute difference between the autocorrelation function of each reference organs and the respective cross correlation to these scores to create a score matrix for the image. We obtain four score matrix, providing scores indicating proximity of these reflectance to these four reference organs. We empirically established threshold values, so that each pixels now would either be categorized as an organ if the scores show close match to the reflectance of the organ, or it is left unaltered if the threshold values are not met, indicating that the pixel’s reflectance vector is different from the one of different organs.



If the following condition for the score matrix of an organ is satisfied, the pixel is assigned with red (liver), green (stomach), blue (colon) or white (spleen) values.



Results based on the aforementioned correlation algorithm is shown in the figure below. Correlation algorithm sucessfully distinguished different test organ images. However, as can been seen from the images, skins and shadows were also recognized as being organs (for example: blue pixel overlay on some part of the skin). This artefact, although reduced by bilateral filter, cannot be completely eliminated because of the non-uniform illumination. When estimating reflectance of the surfaces in the field of view, the photon count from the white tape is taken as the illuminant.

Figure 18 - Organ images run through the reflectance correlation algorithm show different organs being shown as different colors. Colon, liver, spleen and stomach are shown as blue, red, white and green, respectively. Notice that some dark area and some skin pixels are also rendered as being particular organ. We believe this artifact is due to non uniform illumination of the image.

Graphic User Interface: Hyperspectral Image Analyzer

Hyperspectral Image Analyzer provides graphical user interface to load hyperspectral images, and compare the results produced by different algorithms. The interface also allows user to select wave bands in the hyperspectral image to be mapped to the standard RGB. In the figure below, speen is being selected, and correlation algorithm is used to render the image by using the wave bands selected from PCA analysis. The original image and rendered images are shown for comparision.

Figure 19 - A snapshot of Hyperspectral Graphic User Interface: GUI with annotation of various features and properties. Images can be loaded and displayed by passing it through different algorithms. The original and rendered images are displayed. The wavebands to be used for selected algorithms can also be manually selected by using the slider.

Conclusions

Based on aforementioned studies performed on the hyperspectral image data of four pig organs, we have shown that it is possible to render images with enhanced contrast between pig organs. Our aim was to evaluate various methods to find the highest contrast bins, and this was performed by maximum standard deviation and PCA algorithms. Both algorithm showed higher contrast compared to conventional RGB pictures. We also showed that by using representative reflectance, which is specific to different organs, we were able to distort the RGB values in the CIELab colorspace. We saw perceptual difference between organs that would otherwise look similar. We also showed that a correlation algorithm using bilateral filter was successfully able to identity different organs based on the training set. We have provided analysis for selecting 3 or 4 wave bands that could now replace the filters in traditional camera. We have shown from our studies that images obtained from these cameras should provide increased contrast when compared to standard RGB filters. Finally, we have made a Hyperspectral Image Analyzer platform, which allows a user to load images, and use the methods studied in this project to distort and render new images. This tool is useful to study the difference between the original image and the enhanced image, and can be used as a user-friendly educational tool.

Future Work

  • The algorithms developed in this project can be further tested with more data sets. In future tests, we would take extreme care to ensure that the photon illumination of our data sets is uniform, as we believe that the reflectance estimation is compromised by non-uniform illumination of the organ.
  • A very interesting application of this work can be to improve early detection of ischemia. Imaging the effects of altered blood supply to organs have been studied previously, and these algorithms can be applied to pre and post-ischemic organ tissues to render them with high contrast, thereby detecting ischemia.
  • The algorithms developed in this project could be further implemented at the hardware level to apply real-time image processing. This would allow physicians to simply turn these enhancement features on to help differentiate between similar looking organs.

References

  • 1. S. G. Kong, Y. R. Chen, I. Kim, and M. S. Kim, “Analysis of Hyperspectral Fluorescence Images for Poultry Skin Tumor Inspection,” Applied Optics, Vol. 43, No. 4, pp.824-833, February 2004.
  • 2. H. Akbari, Y. Kosugi, "Hyperspectral Imaging: a New Modality in Surgery," Recent Advances in Biomedical Engineering
  • 3. C. Tomasi, "Bilateral filtering of gray and color images," IEEE Computer Vision Sixth Conference, 2002
  • 4. http://www.mathworks.com/matlabcentral/fileexchange/12191

Appendix I - Presentation Slides and MATLAB Code

Presentation Slides

File:HyspexPigOrganSlides.pdf

MATLAB Code

File:HyspexPigOrganMATLAB.zip

Appendix II - Work partition (if a group project)

All:

Paurakh:

Swadesh:

Michael: