Hampapur

Background

The goal of this project is to determine if and by how much the blue pigment (color) present in the Sellaio face painting has degraded. To this end, I attempt to use the hyperspectral data of said image.

Methods

The key constraint with this project is that the ground truth is not available i.e we lack information as to how the artist intended the painting to look like. Consequently, all methods attempted employed analysis and heuristics with the hope that we might perhaps make a cogent argument on how the painting could be restored, if it should need restoring. Methods were employed both in the spectral domain and in CIELAB space. [[1]]. Furthermore, this problem is further complicated by the fact that over time, owing to sunlight, moisture, etc, the spectra could change in undefined ways. This is mostly attributed to the fact that the color pigments along with the material over which they're painted, degrade in undefined ways.

Methods in the spectral domain

Principal Component Analysis

An attempt was made to look at the principal components across the various spectra and determine the number of principal components to preserve. However, truncating the number and magnitude of the components proved to yield no useful information. Hence, this method was not pursued further.[[2]]

Amplitude Boosting

The key idea under this method was the assumption that over time, the only change in the spectra was the attenuation of the amplitude associated with said color. Studies of other paintings have shown that the original painting, as intended by the artist, were meant to be garish and bright as opposed to subtle and smooth. Selectively boosting the spectra for blue would perhaps yield the intended result.

Clustering

This employs state-of-the-art unsupervised learning algorithm detailed in [[3]]. The idea that I wished to pursue in this section is that even if the spectra of the image changed in undefined ways, it should be possible to aggregate like colors together. To this end I frame an adjacency matrix A (or similarity matrix as defined in the paper) based on two criteria - one, the spectral proximity and two,the energy content in said spectral bands. So, for instance, if there were four wavelengths at 300 nm, 700 nm, 720 nm, 730 nm at 20, 25, 23, 50 energy units, the adjacency matrix would claim that the 720 nm and 700 nm are related (by putting in a 1 in the corresponding spot in the matrix) and say that the 300 nm, 730 nm and the group of 700 nm and 720 nm are not related/similar by putting in a 0 in the corresponding positions. Note that the 730 nm is not classified as related to the 700 nm and 720 nm because of the large difference in energy levels. Of course, a particular wavelength is related to itself,so all the diagonal entries would be 1. This is then fed into the clustering algorithm. At the end of the algorithm, I expected to get a cluster of the key colors making up the painting which I would then use to construct the painting.

Methods in the CIELAB space

The CIELAB space is constructed to reflect the similarity in the perception of the human visual system. Therefore, by working in this space , the hope was to be able to determine the direction and magnitude of selectively moving the blue pigments so as to achieve maximum visual appeasement. Unfortunately, this method also lacked a key data metric: the delta E [[4]] values for the color pigments used in the creation of the blue color in the painting.

Moving from spectral data to LAB space

Ideally, we would first begin by changing the illuminant in the scene to D65. Unfortunately, owing to memory constraints, ISET refused to do this transformation.We then move from spectral data to XYZ space. From this space, we get the white point of light and then move into LAB space. It is here that the grid search and selective changing of the pigments in blue is achieved.Then the space is moved back to XYZ and then moved into sRGB where we get the final result and evaluate the effect of performing the changes.

Grid search

For lack of practical data detailing the changes in the delta E values of the blue pigment, I had to improvise and use similar data. In particular, I used the degradation data detailed for the red pigment (also found in other paintings) as a baseline estimate for the delta E values for blue. In particular, as detailed in Kirby and Saunders' paper, I bound the upper value of delta E value by 2 and 6. Owing to computational constraints, I ended up using a delta E value of 2 distributed over a logarithmic spacing of values. Furthermore, for each of the values of blue in the LAB space, this method generates a sphere of possible values. As noted by the delta E values, the small change in delta E value should be indicative of the (perhaps) limited change in the perception of the painting.

Some practical experiments

This was one of the first attempts in trying to be able to determine how the spectra of plant based pigments change under natural conditions. I ground raw spinach and brinjal to get green and purple/blue color pigments. I dried the colors using paper towels and then applied the colors to white cotton cloth and left outside to outside conditions. Over a period of 48 hours, the green color had degraded to yellow , perhaps owing to the death of the chlorophyll in the pigments and the blue/purple had degraded to black. The goal was to determine if there was a particular direction in the spectum in which plant based colors moved upon natural degradation.

Results & Comments

When rendered into sRGB space, there is no visible difference. The painting, however, looks lovely when rendered in D65 light. I will try and talk to Prof. Joyce to get this fixed and include said image in here for posterity.

Clustering

The community detection algorithm/clustering algorithm resulted in 84 clusters of approximately 2 wavelengths per cluster. This is significantly larger than what I was expecting. I believe the performance of this algorithm could be improved by better selecting the criteria from which to form the adjacency matrix. In particular, my restriction that consecutive bands must be related should, perhaps, be relaxed to get smaller number of clusters with more members. On success, this should definitely look lovely.

Grid Search

As noted in the method section, this method yields too many feasible points in the LAB space. Visually evaluating all possible combinations of paintings is not a feasible solution. A better heuristic would be to add an additional constraint while performing the search for the direction and magnitude in which to selectively move the blue pigment points. A fine example would be to add in a constraint such that the covariance of the resulting selected blue pigment points is also minimized. This would ensure that all the blue pigment points would be clustered as close to one another as possible. The caveat with this method would be that we're allowing different parts of the same color to be allowed to vary in different ways (directions). Even though this sounds feasible, having the same pigments deteriorate in dramatically different ways within the same painting is unlikely.

Update

After obtaining the painting under D65 lighting, several new methods were implemented and carried out with success. In particular, replacing the blue pigments by their mean, replacing all the pigments by their k-mean values, replacing the pigments by the median, mean to within a given range, replacing the blue pigments with the min, max values within a specified range were carried out. With the exception of k-means, and replacing the blue pigments with over-all mean, the delta E values were well within 20 (others were several orders of magnitude higher). Obtaining delta E values within this range as established by Kirby and Saunders (the max value they describe for degradation is about 21 in delta E). I would like to draw attention to the third (third from top, the smaller one) rendering of the painting. It seems to bring to light the intended blue in the painting. Also note the spot of blue on the left of the painting. What was dimmer is now more distinct.

Extended update

A few more techniques were implemented and tried out involving moving the mean in the blue channel, scaling values in the blue channel within a local bound, moving and minimizing the mean to within a local bound. A few words about the following results are provided below:

Comparison of k-means image with original image. As can be seen, this is most definitely cannot be how the artist intended the painting to look like.

Comparison of moving all blue channel values to the global mean with original image. This is the extreme scenario in which the color is radically changed. Again, not quite what the artist intended.

Comparison of moving only true blue values defined to be in the range of (-7,-2) with original image. Selectively, the values are replaced to yield the following result.This yields a delta value around 20 which is an acceptable variation in change of the painting.

Comparison of moving local values to true minimum value in the locality to original rendering. This yields a result well within acceptable delta E values and also (perhaps) moves the blue pigment in a way the artist envisioned it.

Comparison of selectively scaling blue channel values in a true blue neighborhood with original rendering. This is unrealistic and with the exception of the first two scalings, is also not visually appealing.

Comparison of selectively moving the average and then minimizing said values to within a locality for a range of movements. It appears that this operation ideally yields the best set of possible original renderings for the painting while keeping to within the delta E values of 20, which is indicative of no massive change in the painting.

Updated Comment

It definitely appears there to have been degradation in the painting. It also appears that the artist used blue as a primary color over the painting. Sploshes of said color can be seen surround the woman's hair, in the hillock in the left and also in the skies. However, I still maintain that the degradation is not severe as evidenced by the low delta values attributed from all the pertinent changes presented.

Code

Included in the zip file are the scripts(in conjunction with ISET) used to create said images.Feel free to use them at will. Code for psych 221

Summary

The pigments in the painting deteriorate in ways that cannot determined. Key causes for such deterioration include sunlight, moisture, etc.

Future work & Comments

This project would be very well served by some key additional pieces of information. In particular, having chemical analysis of the blue pigment done in order to determine the constituents would be a first step. We could then use the spectra of the raw materials and then perform an additive analysis of the spectra and come up with an approximate to the current spectra. The difference would be a good indicator as to how the spectra degraded with time. Another method would be to actually try and look at accelerated degradation of the blue pigment. This would perhaps give us an idea on how far the painting has degraded. In all, having the spectra of the raw materials used to create the painting would be an excellent additional piece of information to have.

Conclusion

I learned a lot in doing this project. Since this project has no solution, trying out different techniques from various branches of engineering was an eye-opener.

Acknowledgement

I would like to thank Prof. Joyce Farrell for providing both the hyperspectral data and also referring me to key papers central to attempting to solve this project.