LawalMcCoy
Improving Subtitle Visibility in Low Budget Movies
Introduction
Subtitles are used as common tools to help audiences understand dialogue that they otherwise would be unable to interpret. However, by simply overlaying white text above the image there is a chance that the subtitles themselves will be unreadable. The main goal of our project was to develop a program in MATLAB that achieves visually pleasing viewing properties of a given subtitle and frame (picture) combination. Although the best solution is to have real people go through a movie, frame by frame, and decide what viewing properties for the subtitles makes them most visible, developing an automated solution may be the next best thing. Many films, particularly foreign films, may not have the budget to have personally add subtitles to their movies and would greatly benefit from such an algorithm.
When deciding how to create the optimal subtitles, we investigated a number of variables, including the location of the subtitles, the color of the subtitles, and any potential shadowing around the subtitles. When comparing colors, our program uses the CIELAB metric of DeltaE as a proxy for perceptual differences between the surrounding image and the subtitles. Color theory states that the larger the DeltaE between two objects, the more discernible they are to the human eye. In this situation we analyzed the average DeltaE difference between text and other pixels in the textbox.
One of our major goals was to answer the following question: do larger DeltaE differences appear more visually appealing to the viewer or do more subjective factors (such as shadowing and location) dominate? Furthermore, is there even really an advantage to larger DeltaE differences (with all other factors held constant) in determining visually pleasing subtitles?
Method
How the program works
The program works through the use of a main wrapper function, place_captions. Before running the program though, preliminary work needs to be done. First the clips are converted into movie structs which are Matlab constructs that contain each frame in the movie as an image. Then, the subtitles must be transcribed into a text file in the same directory as place_captions. Now the program is ready to run. The program creates a black shadow around the text which research has shown increases discriminability between text and the image behind it. Next the location of the subtitle is so that it is in the lower left corner, and issues with this are discussed in the subsequent section. The program that we run finally selects the optimal shade of gray (equal amounts of RGB channel) that maximize the deltaE of the subtitles between the image the subtitle is overlaid on.
Shadow
The goal of shadowing is to have the text blend into the background as seamlessly as possible. Unlike caption boxes, which cover up much of the background image, we designed our shadowing to retain as much background information as possible. In order to place a shadow around the text, we first had to obtain a mask that would allow us to separate the text from the actual image. To achieve this we place the same text at the same position on a completely black image that was the same size as the original image. By calculating the distance from each pixel that was near the text to the actual text, we were able to determine how much shadowing to place on each pixel. For example, a pixel right next to the text would be turned completely black, while a pixel three pixels away would only be darkened. One of the inputs into our place_captions function is shadow_size, which will determine how many pixels beyond the text the shadow will extend. We found that a shadow size of 3-5 pixels was most effective.
Subtitle Location
The subtitle's upper left location was selected to be, in most cases, 10% above the bottom and 25% to the right from the left boundary of the image. This was done to replicate the the style in commercial subtitles. Issues came into play when the actual subtitle was either too long, the font needed to be adjusted, or if the images that were read in from the movie were small. This called for adjustments in the code to break up the subtitle into two lines if the width of the subtitle was greater than that in the image as well as allowing font to be an input to the function.
Subtitle Color
Challenges
We ran into several challenges using our program, the most of which was simply computing time. To add subtitles to even a clip of just roughly 30 seconds, it typically took our program over 4-10 minutes to accomplish this depending on how many subtitles were present in the clip and the resolution of the images. This was simply running our program to produce white subtitles with a shadow behind them, like most commercial subtitles. Furthermore, when trying to optimize for color, for each individual frame it would take four minutes for the program to give us the correct gray shade that maximized the deltaE difference between the text box and the pixels on the image. This clearly makes maximizing deltaE impractical for clips of reasonable length using our program. However, for the vast majority of frames we tried this on, we found that a completely white subtitle actually maximized deltaE so the base case assumption (which is used in commercial subtitles) was essentially sufficient for a movie thus cutting down computing time. For the interest of this presentation though, we will also present a few images where the deltaE difference was not maximized with completely white subtitles.
Results
The above three images represent from left to right, the bright white subtitle (not deltaE differential maximized) overlaid on an image, the deltaE differential maximized shade of gray for the subtitle with a shadow, and the deltaE maximized shade of gray with the subtitle without the shadow. Upon polling residents in our dorm, we found that consistently people found the deltaE differential maximized subtitle with a shadow as the least visually appealing. Generally, people found the deltaE maximized shade of gray without the subtitle to be the best and the white subtitle with the shadow to be middle best. However, there was not a massive preference between the latter two. This yields us to believe that the effect of the shadow works best with white subtitles, and perhaps can compensate for lack of deltaE differential.
Conclusions
test
References
test
Appendix I - Code and Data
test
Appendix II - Work Partition
test