LawalMcCoy: Difference between revisions
imported>Psych2012 |
imported>Psych2012 |
||
| Line 21: | Line 21: | ||
=== Subtitle Location=== | === Subtitle Location=== | ||
Based on our research into subtitling, we found that subtitles are typically placed near the bottom of the frame and aligned left. We attempted to mimic this position, without spending too much time focusing on this aspect. The subtitle's upper left location was selected to be, in most cases, 10% above the bottom and 25% to the right from the left boundary of the image. This position sometimes had to be adjusted, as did the font size, when the movies read in were small. Instead of trying to create a very general function to find the appropriate location, we made a fairly simple one and just adjusted it as needed. Another issue we ran into was when subtitles were too long for one line. To solve this, we check if the size of the textbox surrounding the subtitle is longer than the image itself. If it is, we split the subtitle into two lines and place the second line directly beneath the first. | Based on our research into subtitling, we found that subtitles are typically placed near the bottom of the frame and aligned left. We attempted to mimic this position, without spending too much time focusing on this aspect. The subtitle's upper left location was selected to be, in most cases, 10% above the bottom and 25% to the right from the left boundary of the image. This position sometimes had to be adjusted, as did the font size, when the movies read in were small. Instead of trying to create a very general function to find the appropriate location, we made a fairly simple one and just adjusted it as needed. Another issue we ran into was when subtitles were too long for one line. To solve this, we check if the size of the textbox surrounding the subtitle is longer than the image itself. If it is, we split the subtitle into two lines and place the second line directly beneath the first. We created a simple MATLAB function that would take in the size of the image and the width of the text and output the top-left corner of the box to place the text within. | ||
''[x y] = pos_text(image_size, textwidth)'' | |||
<div style="text-align: center;"> '''Example of Splitting Captions into Two Lines''' | <div style="text-align: center;"> '''Example of Splitting Captions into Two Lines''' | ||
Revision as of 00:02, 22 March 2012
Improving Subtitle Visibility in Low Budget Movies
Introduction
Subtitles are used as common tools to help audiences understand dialogue that they otherwise would be unable to interpret. However, by simply overlaying white text above the image there is a chance that the subtitles themselves will be unreadable. The main goal of our project was to develop a program in MATLAB that achieves visually pleasing viewing properties of a given subtitle and frame (picture) combination. Although the best solution is to have real people go through a movie, frame by frame, and decide what viewing properties for the subtitles makes them most visible, developing an automated solution may be the next best thing. Many films, particularly foreign films, may not have the budget to have personally add subtitles to their movies and would greatly benefit from such an algorithm.
When deciding how to create the optimal subtitles, we investigated a number of variables, including the location of the subtitles, the color of the subtitles, and any potential shadowing around the subtitles. When comparing colors, our program uses the CIELAB metric of DeltaE as a proxy for perceptual differences between the surrounding image and the subtitles. Color theory states that the larger the DeltaE between two objects, the more discernible they are to the human eye. In this situation we analyzed the average DeltaE difference between text and other pixels in the textbox.
One of our major goals was to answer the following question: do larger DeltaE differences appear more visually appealing to the viewer or do more subjective factors (such as shadowing and location) dominate? Furthermore, is there even really an advantage to larger DeltaE differences (with all other factors held constant) in determining visually pleasing subtitles?
Method
Placing Captions on an Image
First we began by placing captions onto single images. We loaded the images into MATLAB, and used the build-in “text” function in MATLAB to place the text onto the image. We found that the appropriate text size for most images is 16 point. We created a main wrapper function, Place_Captions.m.
image = Place_Captions(im, txt, position, fontsize, fontcolor, shadow_size)
This function takes in an image, a string to overlay onto that image, a position with an x and y coordinate for the top left corner of the textbox, a fontsize (typically 16), a vector representing the RGB values of the fontcolor, and the length of the shadow extending from the subtitles. We used this function to investigate how the various text locations, font colors and shadows impacted how the text appeared on the image.
Subtitle Location
Based on our research into subtitling, we found that subtitles are typically placed near the bottom of the frame and aligned left. We attempted to mimic this position, without spending too much time focusing on this aspect. The subtitle's upper left location was selected to be, in most cases, 10% above the bottom and 25% to the right from the left boundary of the image. This position sometimes had to be adjusted, as did the font size, when the movies read in were small. Instead of trying to create a very general function to find the appropriate location, we made a fairly simple one and just adjusted it as needed. Another issue we ran into was when subtitles were too long for one line. To solve this, we check if the size of the textbox surrounding the subtitle is longer than the image itself. If it is, we split the subtitle into two lines and place the second line directly beneath the first. We created a simple MATLAB function that would take in the size of the image and the width of the text and output the top-left corner of the box to place the text within.
[x y] = pos_text(image_size, textwidth)
Shadow
The goal of shadowing is to have the text blend into the background as seamlessly as possible. Unlike caption boxes, which cover up much of the background image, we designed our shadowing to retain as much background information as possible. In order to place a shadow around the text, we first had to obtain a mask that would allow us to separate the text from the actual image. To achieve this we place the same text at the same position on a completely black image that was the same size as the original image. By calculating the distance from each pixel that was near the text to the actual text, we were able to determine how much shadowing to place on each pixel. For example, a pixel right next to the text would be turned completely black, while a pixel three pixels away would only be darkened. One of the inputs into our place_captions function is shadow_size, which will determine how many pixels beyond the text the shadow will extend. We found that a shadow size of 3-5 pixels was most effective. We created a Make_Shadow function to add the shadow after we have placed the text.
shadow_image = Make_Shadow(image, mask, shadow_size, shadow_factor)
This function takes in the image, a mask containing only the text with a black background, the number of pixels that the shadow will extend from the text, and a scale factor that represents how quickly the shadow fades away. The function works by weighting the distances to nearby text from each pixel and summing these weights. Once normalized, this is the factor we use to darken nearby pixels. Thus if a pixel is very close to dense subtitles, it will be significantly darkened. There are many potential shadowing algorithms, but this one seemed to work fine for our purposes.
With 5 Pixel Shadow
Subtitle Color
We started by looking at both white and yellow subtitles. We decided that in most cases white subtitles looked better and thus stuck with white and shades of gray (equal amounts of each RGB channel) as potential subtitle colors. We did this in spite of the fact that yellow subtitles had higher DeltaE values in general when compared to nearby pixels. We also decided that we would only use brighter shades of gray and not black subtitles. We felt that black would contrast too much with the film and would not be as visually appealing. Thus, we designed an algorithm that would select the optimal color based on comparing the average DeltaE difference between the text color and the other pixels in the text box surrounding the subtitles.
optimal_color = Optimize_DeltaE(im, txt, fontsize)
We decided to stick with only optimizing individual images, because of the computing time required for the algorithm. To find the optimal color for a given frame would take 3-5 minutes depending on the size of the image and amount of text. . This clearly makes maximizing DeltaE impractical for clips of reasonable length using our program. However, for the vast majority of frames we tried this on, we found that a completely white subtitle actually maximized DeltaE. This means that we could assume a white subtitle and perform some sort of quick check to see if optimizing would help a frame. This would allow the algorithm to be effective for clips without imposing an extremely excessive computation time.
After optimizing the color, the shadowing algorithm had to be adjusted to darken the pixels surrounding the text based on the color of the text. However, this algorithm for placing the shadows on subtitles with non-white colors could be significantly improved. We found that after optimizing the color, it was preferred to leave the shadow off.
Placing Captions on a Movie Clip
In order to place captions on actual movie clips, we first found clips online and converted them in MATLAB movie structs. We then manually transcribed the subtitles to be added with when they occurred and placed them in a text file. We then wrote a function to add the captions for each frame that had dialogue, Movie_Captions.m.
movie_out = Movie_Captions( movie_in, text_file, fps)
The function takes in a MATLAB movie struct, and the location of the text file with the number of frames per second to line up the captions with the corresponding frame. We ran into problems by assuming the clips we had were 30 frames per second, when really some were 23 or 29. To add subtitles to even a clip of just roughly 30 seconds, it typically took our program over 4-10 minutes depending on how many subtitles were present in the clip and the resolution of the movie. This was simply producing white subtitles with a shadow behind them, not optimizing the subtitles. After creating the movie struct we had to run the MATLAB movie2avi command and then compress the movie further into .mp4 form.
Watch some clips that we created with audio included:
Clips without audio:
Survey of Optimizing Colors
Challenges
We ran into several challenges using our program, the most of which was simply computing time. To add subtitles to even a clip of just roughly 30 seconds, it typically took our program over 4-10 minutes to accomplish this depending on how many subtitles were present in the clip and the resolution of the images. This was simply running our program to produce white subtitles with a shadow behind them, like most commercial subtitles. Furthermore, when trying to optimize for color, for each individual frame it would take four minutes for the program to give us the correct gray shade that maximized the deltaE difference between the text box and the pixels on the image. This clearly makes maximizing deltaE impractical for clips of reasonable length using our program. However, for the vast majority of frames we tried this on, we found that a completely white subtitle actually maximized deltaE so the base case assumption (which is used in commercial subtitles) was essentially sufficient for a movie thus cutting down computing time. For the interest of this presentation though, we will also present a few images where the deltaE difference was not maximized with completely white subtitles.
Results
The above three images represent from left to right, the bright white subtitle (not deltaE differential maximized) overlaid on an image, the deltaE differential maximized shade of gray for the subtitle with a shadow, and the deltaE maximized shade of gray with the subtitle without the shadow. Upon polling residents in our dorm, we found that consistently people found the deltaE differential maximized subtitle with a shadow as the least visually appealing. Generally, people found the deltaE maximized shade of gray without the subtitle to be the best and the white subtitle with the shadow to be middle best. However, there was not a massive preference between the latter two. This yields us to believe that the effect of the shadow works best with white subtitles, and perhaps can compensate for lack of deltaE differential.
Conclusions
test
References
http://www.dcmp.org/captioningkey/text.html
Appendix I - Code and Data
Appendix II - Work Partition
test