GellineauHuangLee

Introduction

In recent years, advertisers and magazine editors have been accused of over-glamorizing the subjects of their photoshoots. Through photo-editing software like Adobe Photoshop, models have graced ads and magazine covers with seemingly impossibly slim and curvy figures, or impossibly smooth skin. In response to this phenomenon of 'photoshopping to the extreme,' Eric Kee and Hany Farid published a paper in 2011, detailing on using computers and algorithms to judge how much a picture has been altered--something that usually only a human being might be able to perceive properly.

This project is in essence an attempt to follow and replicate the results obtained by Kee and Farid.

Methods

Our methodology was to gather statistics that describe the alterations people notice most and use them to predict user perception of a photo manipulation. These statistics are separated into two methods as described in the source paper of this project, geometric distortions and photometric distortions. Geometric distortions describe modifications such as flattening the stomach enlarging the bust and shaping hips which help form our opinion on the degree of modifications. The other most common distortions are classified as photometric distortions. These describe pixel to pixel differences between images and the effect of blurring and sharpening on images. We gather these statistics and compare them to data collected from real users. In this way, we can train the software to not only collect these statistics but produce a rating that agrees with our perception of photo manipulation.

Data Collection

A series of approximately 137 before and after images were collected off of the internet from various sources including: magazine covers, artist demonstrations and collections on discussion forums. Full body, face and torso masks were individually and manually made for each of these images. These were necessary in order to collect relevant statistics. If an object is added to the after photo that doesn't affect our perception we would only be introducing noise by attempting to fit statistics to it. Therefore, fitting blurring filters are only done over the face and geometric distortions are weighted based on where they occur on the body. This step of image preparation is the only non automatic part of the software and was a problem for our source paper as well.

To collect all of the user data a website was also constructed. This website allowed a user to rate the degree of image manipulation on a scale from 1 to 5 for 70 photos chosen at random from our set of 137. We then used Google's form interface to have that information sent back to us for aggregation.

The data collected to date is summarized above in the figure of the standard error. The images were assigned a random number but are sorted in the image above in order of average rating. The error bars are calculated as: $\frac{\sigma }{\sqrt{n}}$ where $n$ is the number of times a photo has been rated and $\sigma$ is the deviation of those ratings.

The spread of some ratings are very widely spread while others have a very tight distribution. However, there are a few users who disagree heavily with the average results. This implies that a few users may not have carried out the experiment correctly.

Geometric Distortions

We implemented an algorithm for image registration as was described in reference 4. The algorithm seeks to fit an affine distortion to maps the original image to the edited one. An affine fit transformation is described as:

$m_{7}f(\tilde{x},\tilde{y},after)+m_8=f(m_{1}x+m_{2}y+m_5,m_{3}x+m_{4}y+m_6)$

M7 and M8 describe the contrast and brightness alterations in the image and are only included to improve the quality of the geometric fit. The other terms describe the translation and stretching of the image. It is important to note that at the end of the fit, these m values are estimate for 9X9 patches of each image. These values can then describe the vector field which represents the amount of local distortions between the original and edited image.

$v(x,y)=(m_{1}x+m_{2}y+m_5-x)\hat{x}+(m_{3}x+m_{4}y+m_6-y)\hat{y}$

The finer details required to implement this algorithm are in the references and our source code is available in our appendices but there are a few parts of the algorithm that are worth explaining. The image below shows the performance of the algorithm. The algorithm takes the before and after image and calculates the vector field V(x,y) which describes the geometric changes between the before and after images. Notice that the magnitude of v(x,y) has hot spots where the photo editor brought in her hips. The applied fit is a copy of the original that has the calculated vector field applied to it. Notice that this image is now better aligned to the edited image. This is a requirement for the next step.

The algorithm does the following in order to establish a fit:

• Both images are transformed into 255X255 grayscale images. Their borders are padded with random noise and the background in each image is replaced with random noise.
• They are then both histogram equalized.
• The image is then blurred and decimated in a Gaussian pyramid.
• At each level, we do a preliminary align where the entire image is the block size then we break the image into 9X9 blocks and perform a geometric fit.
• To ensure smoothness in m values, we then penalize for smoothness and iteratively solve a least squares problem that minimizes the error in both estimating m and the smoothness of m.
• With this estimate of m, we then calculate the likelyhood of each pixel in the original image belonging to the edited one and again iteratively solve for these m values.
• This overall process is repeated 5 times to improve the fit before the algorithm moves on to the next level of the pyramid.
• In the end we collect all of the transformations and return an estimate for the vector v(x,y).

Not all images were perfectly registered in our testing. For example, in the image on the right we can see that though the algorithm did estimate the distortions that was made by this photo manipulator, it did not distort the image to the same degree. We believe that our settings to penalize smoothness and number of iterations were not set as well for this image as it was for others. Convergence is not unique and for a few images we do notice a slightly better fit for repeated tries.

Photometric (draft)

Besides geometric adjustment, photographers also edit the photometric features to make the images more stunning. For photometric model, we are focusing on face images, which is also the area catch most attentions. There are many tricks we could do for the face images. For example, utilizing sharpness filter to eye region to make eyes more sparkling, and also applying blurring filter to make skin more smooth(as shown in the following image left:original image, right:altered image).

These linear filters would be used in different regions for different purpose(smoothing and sharpening), so we have to divide the images into smaller patches to analyzing local regions. In the ____'s paper, they used two measuring method to evaluate the degree a face image has been altered: structural similarity(SSIM) and perceptual distortion. For both methods, we use luminance channel, which is easier for us to analyze image contrast.

Structural Similarity (SSIM)

SSIM is a methode to measure how similar two images is, which is including structural modification and contrast. File:Ssimformula1.png where File:Ssimformula2.png File:Ssimformula3.png

File:Stdab1.png File:Covab1.png

Perceptual Distortion

Training

The eight parameters obtained from the geometric and photometric analysis were trained on the labels collected from user input. Although Kee and Farid only collected data for regression (that is, the average of all responses for an image), we also calculated labels for the purpose of classification (using the mode of all responses for an image).

The LIBSVM software, developed by Chih-Chung Chang and Chih-Jen Lin at the National Taiwan University was used to train, cross-validate and eventually test our metric.

Results

Organize your results in a good logical order (not necessarily historical order). Include relevant graphs and/or images. Make sure graph axes are labeled. Make sure you draw the reader's attention to the key element of the figure. The key aspect should be the most visible element of the figure or graph. Help the reader by writing a clear figure caption.

Conclusions

Describe what you learned. What worked? What didn't? Why? What would you do if you kept working on the project?

References

1. Kee E, Farid H, 2011. A Perceptual Metric for Photo Retouching.
2. Chih-Chung Chang and Chih-Jen Lin, LIBSVM : a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1--27:27, 2011. [1]
3. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, "Image quality assessment: From error visibility to structural similarity," IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, Apr. 2004.
4. Senthil Periaswamy, Hany Farid, Medical image registration with partial data, Medical Image Analysis, Volume 10, Issue 3, June 2006, Pages 452-464

Appendix I

Upload source code, some result images, etc, and give a description of each link. In some cases, your acquired data may be too large to store practically. In this case, use your judgement (or consult one of us) and only link the most relevant data. Be sure to describe the purpose of your code and to edit the code for clarity. The purpose of placing the code online is to allow others to verify your methods and to learn from your ideas. It should be possible for someone else to generate result images using your code.

Appendix II

Our group met frequently to discuss the direction and requirements of our project. As a result, we assigned the following tasks to ourselves:

Antonio: Geometric Algorithm, website design, Main GUI, Masking in Photoshop Justin: Image Acquisition, SVR,ssim, Data Parsing, Large share of Masking in Photoshop Chun-Wei: D-Value Algorithm, Additional GUIs for testing D-Values and ratings, Image Evaluation, Masking in Photoshop

We would like to note that problems came up at almost every level and we did work together on most problems to get through it all. We also spent a lot of time together just preparing and testing our code. It took a half a day to analyze all 137 images split between 4 workstations.