App for Programmable Camera in Android

Group Members: Kaitlyn Benitez-Strine, Ronnie Instrella, Joe Maguire

Back to Psych 221 Projects 2014

Introduction

The basic description of our project is as follows:

We have a prototype programmable camera to be used with iOS or Android devices. The project's goal is to make an app that will run on iOS or Android and uses the camera. Think of an interesting camera app, and we can work together to build it. Prior experience in iOS or Android is needed.

Our group's task was to use the power of a programmable camera to make seemingly difficult photos much easier to produce. We immediately started thinking of types of photos that are somewhat easy to do (if you know how to manipulate your camera a little), but that are difficult enough that the average person would not automatically understand how to reproduce. Today there are various cool effects your average digital camera can do, if one understands how to manipulate the numerous modes. However, most people don't understand the concept behind the shot and thus stick to the simple "point and shoot" automatic mode.

We wanted to hone in on one single "cool" effect, and after turning to the internet for inspiration, we found our project - slow shutter speed photography. Inspired by the magical picture to the right, our group set out to make slow shutter speed photography an easy experience through the phone.

What has Been Done in the Past

Our group investigated prior applications on the iPhone/ Android to see what sorts of options were available to users on a typical slow shutter speed photography app. We picked two different apps for the iPhone and analyzed how the average person would approach taking a photo with these apps. We picked a paid app and a free app to analyze: Slow Shutter Cam and LongExpo respectively. I had our subject try to match a photo we had taken with the app while doing a think-aloud protocol.

Mobile Application: Slow Shutter Cam

Featured by Apple in "App Store Essentials: Camera & Photography" and "Extraordinary Photo Apps", Nominated for the 2010 and 2011 "Best App Ever Award - Best Photo App", and Recently mentioned by the NY Times is the best slow shutter speed camera on the market right now for IOS - Slow Shutter Cam [1].

For $0.99 one can take three types of slow shutter speed photography photos.

1. Motion Blur: Basically allowing the user to turn on the the shutter priority mode on the camera, it captures images over an extended period of time for lovely blurring effects (perfect for ghost images, waterfalls, and adding a notion of movement to photos.)
2. Low Light: Used under low light conditions so that camera sensor picks up every last photon of light, this mode allows the user to capture people/ moving objects under low light conditions.
3. Light Trail: Used to capture moving light (such as fireworks or cars at night, this mode allows the user to capture the movement of a light source.

And in addition to these modes, one has the options of tapping to adjust focus/ exposure, locking the exposure/ focus from shot to shot, seeing a live preview of the captured image, and much more during the taking of the photos. After the fact, the user can compensate for exposure, and adjust brightness, contrast, saturation, and hue.

All of this is wonderful for a photographer wishing to take nearly Digital Single Lens Reflex (DSLR) shots with his or her iPhone camera, but for the average person I wondered what they would make of the features.

Review of Slow Shutter Cam

Never having been exposed to the app before, our subject was confused by a couple of things. First she was unaware of what the AF/ AE locks were at the top of the screen and after much experimentation she was still unaware of the purpose of their function (locking exposure and focus for numerous shots). Plus it wasn't exactly intuitive where to find the major modes of the app (which could be found by pressing the shutter looking object in the lower left side of the screen). Once she discovered the button (after having exhausted all other options), she didn't know which option would be best for her purpose of taking a photo of a waterfall in a shady region of the Stanford Shopping Center on an afternoon in March. Furthermore, after experimenting with the modes she was never entirely certain what terms like "exposure", "exposure boost", or "light sensitivity" meant.

As a result, we intend to make the "difficult" terms more easily understood, and pictures to change modes more readily observed for the average non-photographer user. This could be done with better symbols/ pictures, and little blurbs to explain the effects each option has on the captured image.

Mobile Application: Long Expo

LongExpo doesn't have quite as many awards as Slow Shutter Cam, but it is a great free app for long exposure and light trail photos. It's current rating for the latest update is 4.5 stars. [2]

LongExpo also has three modes:

1. Standard mode: Used for the typical blurring effect, this mode is for regular long exposure photography.
2. Low Light Mode: With an adjustable bar for exposure in low light conditions, this mode allows the user to take photos of movable objects/ people in darker settings.
3. Light Trail Mode: Focused on recording moving light sources, this mode captures light trails, and light stream photography.

And beyond these modes, one can edit the photo in numerous ways after taking the shot. The user can adjust brightness, contrast, saturation and where the photo freezes. Plus they can add filters (similar to Instagram's), as well as frames, stickers, focus, drawings, and meme lettering, as well as other interesting effects.

Again we had an average person test the app without having seen it before to see how she would match a photo we had taken previously.

Review of Long Expo

The user had difficulty finding go back/ delete buttons when she wanted to take a different picture. It was also unintuitive how to switch shutter speed from the get go. After tapping random buttons, she eventually discovered the adjusters she had wanted to find, but it took some time to do so. And even then, when she found the variable shutter speed she discovered a B "time" (which basically goes as long as you want to take the picture for), but that was not obvious from the start.

She also never found out that there were several modes (the three listed above), so she just stuck to what she was given from the start and never discovered the other modes to choose from.

All of this made it clear that the buttons for adjusting modes, and going back should be clear and explicit as soon as you open the app. Plus, labeling of what certain adjusters denote would also be helpful.

What We Intend to Do

The best part about our app is that we have a mirrorless, programmable camera to take the photos for our users, which implies that the user will be free to move about while getting high resolution photos. So although we intended to have similar modes to the apps before us, we have a couple advantages that make for a better shot:

• Stability: The programmable camera can be placed on something so that when the user intends to take the image, he/she does not ruin the shot by pressing the camera button on the phone and changing its disposition as the picture begins. The slight change in position would create an unsharp image, but the programmable camera can be placed on a table or tripod.

• Mobility: The programmable camera again can be placed anywhere, so the user does not have to be where the camera is and yet the user can see what the camera sees and control it without being next to it. Thus, the user can be in the photo without running to beat a self timer.

• Fine Tuned Control: The programmable camera has more possibilities than a phone's camera allowing for a better shot - for instance there are finer tuned choices for shutter speed, ISO speed, and aperture.

With these benefits, my group intends to create an app with four modes:

1. Motion Blurring: Great for capturing movement either by blurring the moving objects/ water, or for blurring the background by panning means.
2. Painting With Light: This technique is when either the user is behind the camera and lighting up an object in front of the camera (thus the emitted light becomes a sort of paint brush unto the night scene), or the user is in front of the camera with a light source and the light source is what is painting the scene (light trail).
3. Montage: In this mode the camera takes continuous photos quickly of a moving subject and combines all stills of the movement into one shot.
4. Thresholding: Here the camera takes the high contrast between light and dark pixels and gets rid of all the color to become a black and white photo highlighting the light parts in white and dark parts in black.

And along with these modes the user will be able to adjust the brightness, hue, and contrast after the picture has been taken.

Background

Besides our own basic knowledge on slow shutter speed photography, we also searched the internet for the basics on motion blurring and painting with light, and we interviewed a professor for continuing studies, Joel Simon, for more helpful tidbits. We aimed to make slow shutter speed photography as intuitive as possible for the main photographer, but in the end, practice is what will make anyone better at taking photos under any conditions.

Motion Blurring and Panning

Slow Shutter Speeds introduce two sorts of blurring sources: subject and camera movement. All light is blocked by the shutter - since it sits in front of the camera's sensor - until one presses the shutter button to flip it open for a designated length of time.

Motion Blurring

For motion blurring, ideally the only movement will be of the subject since the camera should be steady on a table or tripod. This will allow the image to be clearer (at least the background will be). Nikon provides a great artist's rendering of what the camera captures with a moving subject at certain shutter speeds at this website here: http://imaging.nikon.com/history/basics/04/03.htm .[3] When a slower shutter speed is selected, a longer time elapses from the instant the shutter opens until the instant it closes. The camera can record as much movement from the subject it can, given the time available (shutter speed value).

One problem we ran in to was that the longer the shutter was open, the more of a risk there was of blowing out or overexposing the photo. To counter that, Darren Rowse of the Digital Photography School and Joel Simon , a continuing Studies professor at Stanford, had some helpful tidbits of information. Both Darren and Joel suggested closing the aperture, decreasing the ISO, or trying a neutral density filter [5][7], however we could not change the aperture due to a fixed lens size, nor could we add a neutral density filter since we would not be able to give everyone with the app a neutral density filter. Thus our only defense was lowering the ISO speed for photos taken during the day.

Panning

Another way of capturing movement is by panning the moving object. Joel suggested we add this feature since it fell perfectly along with motion blurring. This entails rotating the camera to follow the moving object (like turning your head on a swivel to observe a passing bicycle or a child on the swings). This captures movement by streaking the background in a motion opposite to the way the camera was rotated while keeping the subject still. The key idea is to keep the subject in the same position in the frame the entire time the shutter is open.

To find the ideal settings for panning, Darren Rowse's guide to mastering panning was very useful. He suggested keeping the shutter speed slightly lower than you normally would, stating, "Start with 1/30 second and then play around with slower ones. Depending upon the light and the speed of your subject you could end up using anything between 1/60 and 1/8"[6]. So we ensured to have those settings available to the user when they decided to pan the camera.

Painting With Light and Light Trail

Painting With Light

Here the real creativity of the user shines through because in this mode one can use their phone screen, flash light, or laser to "draw" (with light) on the objects before the sensor to highlight or "etch in" things they wouldn't normally dare to impose on the real world. Here however, the camera's sensor is capturing the emission and reflection of the light source (typically by shining the light source from behind the camera).

Joel suggested adding this type of slow shutter speed photography to complement the initial light trail idea we had, but to get more details on what were important features for the user, we turned to the internet. Darlene Hildebrandt suggested: locking the focus so that the camera won't "hunt" in the dark for the best focusing power; setting the exposure to manual or bulb so the camera doesn't keep guessing the best exposure; and putting the ISO as low as possible to minimize noise [10]. She notes, "Basically what you do it set your camera on Bulb, open the shutter using your locking release and walk into your scene and start lighting the objects in the camera view using your flashlight." [10]

Light Trail

While the shutter is open for a certain length of time, the sensor does not stop receiving photons from the light sources in front of it. Thus, it keeps recording where it observes the light so that one can see a pretty trail of said light. Again, a tripod or steady camera position is pivotal for this mode so that the light trails remain crisp and clean and the background illuminated in the light can be clear as well. One can either capture the movement of traffic carnival rides or make their own objects with flashlight, phone screen, or sparkler light in front of the camera. As soon as one hits the camera button, the phone screen can become a color that the camera picks up through the sensors, and the user can move their phone into different letters and shapes. The sensor keeps capturing the photons each light source emits for a beautiful picture.

Darren Rowse once more had some helpful tips for this type of slow shutter speed photography. Like Darlene suggested for Painting with Light, Darren suggested keeping a low ISO value for noise reduction, and having the focus be manually locked for best results.

Thresholding

Two of the members of our project have taken Computer Vision and Digital Image Processing courses at Stanford University. To implement these methods we used the Catalano Framework for Android. This framework contains a wide range of optimized android image processing tools.

Methods

Camera Parameters

To accomplish the defined tasks, it is important to have control over the settings of the camera. This includes the camera's shutter speed, exposure compensation, f stop, and ISO. Motion blurring occurs at nominal levels of illumination so we must minimize the amount of light acquired by the CCD or else the image will appear washed out. To do this we automatically reduce the ISO values, exposure compensation while increasing the camera's f-stop. For light trail and painting with light, the level of background illumination is very low, so we can preserve nominal ISO, exposure compensation, and f stop will increasing the exposure time. This is because the light source is orders of magnitude brighter then background, so it will always leave a distinct noticeable trail when the photo is integrated over many frames.

Image Thresholding

We implemented various thresholding methods to try to highlight the light in the light painting and light trail applications. Eventually, the simplest thresholding method, Ostu Thresholding, was chosen. Otsu Thresholding binarizes an image by choosing the threshold with minimal within class variance. In practice, this method perfectly segments the image between the illuminated scene and the background.

Since the Otsu Thresholding perfectly segments the image, we use it to mask the illuminated pixels in both Light Trail and Light Painting. With a mask identifying the pixels, we are able to replace the color of the light source and it's nature. FIGURE X (THE ONE IN THE ROOM WHERE YOU REPLACE THE LIGHT COLOR), shows an example of using the mask to replace the color. We can extend this concept to change the illumination source itself. For example, one can erode the wide light trail into a small path to make it seem light a laser pointer was used.

Noise Suppression

Low illumination coupled with high exposure results in noise issues for most imaging systems. The most serious noise source is photon noise, which is intrinsic to the camera. As discussed in lecture, this noise source is due to the discrete nature of photons. Treating light as a wave it may appear that a proper wavefront impacts the CCD causing a Gaussian distribution of intensity across the sensor. Using this approximation of reality, modern systems treat image noise as independent additive gaussian. However, reality dictates a more complex behavior. Light arrives in wave packets at nonuniform intervals and quantities across the sensor. This results in a noise signature directly related to scene brightness. Specifically, like most random processes involving arrival times, arriving light is Poisson distributed with respect to intensity of light. This means that the arriving single varies around it's mean value with a standard deviation of the square root of the mean value. This variation is called photon noise.

$SNR=\frac{\mu }{\sqrt{\mu }}=\sqrt{\mu }$

As shown above, a signal that varies with respect to the square root of it's mean has a SNR of the square root of the single. Since the square root function is monotonically increasing, as scene intensity increases so does the signal to noise ratio. This has serious consequences on our project because two of it's modes, Light Trail and Light Painting operate at low illuminations.

The noise, shown in FIGURE X, is apparent. Since this noise is fundamental to the nature of light, we must resort to post processing techniques to suppress it. After experimenting with many Digital Image Processing techniques, we settled for median filtering our captured images. A median filter works by passing a small window over an image replacing the pixel at the center of the window with the median value of the pixels inside the window. We used the Catalano Android Imaging toolkit to accomplish this. An image taken with this filtering mode on the smartphone is shown in FIGURE Y. Noise suppression is clear. It should be noted that Dark Current Noise is also an issue for our project as we deal with long exposures. However, our filtering technique also suppresses this.

References - Resources and related work

References

[1] "Slow Shutter Cam on the App Store on iTunes." itunes.apple.com. Apple Online Store, 18 Dec 2013. Web. 15 Mar 2014. <https://itunes.apple.com/us/app/slow-shutter-cam/id357404131?mt=8>.

[2] "LongExpo - slow shutter and long exposure camera on the App Store on iTunes." itunes.apple.com. Apple Online Store, 09 Jan 2014. Web. 15 Mar 2014. <https://itunes.apple.com/us/app/longexpo-slow-shutter-long/id594078421?mt=8>.

[3] "Digital SLR Camera Basics: Shutter Speed." imaging.nikon.com. Nikon Corporation. Web. 17 Mar 2014. <http://imaging.nikon.com/history/basics/04/03.htm>.

[4] Andrew Stanley, Ben Stabler, and Sean Chen. "Automatic Generation of Action Sequence Images from Burst Shots." EE 368: Digital Image Processing, Stanford University, 2013. Print.

[5] Rowse, Darren. "How to Capture Motion Blur in Photography - Digital Photography School." http://digital-photography-school.com. Digital Photography School. Web. 20 Mar 2014. <http://digital-photography-school.com/how-to-capture-motion-blur-in-photography>.

[6] Rowse, Darren, ed. "Mastering Panning - Photographing Moving Subjects - dPS." digital-photography-school.com. Digital Photography School. Web. 20 Mar 2014. <http://digital-photography-school.com/mastering-panning-to-photograph-moving-subjects/>.

[7] Simon, Joel. Personal Interview. 06 Mar 2014.

[8] Wandell, Brian. "Image Capture Sensor 2014 Slides." Psych 221 Class. Stanford University. Stanford. Feb 2014. Lecture.

[9] Lim, Jae S., Two-Dimensional Signal and Image Processing, Englewood Cliffs, NJ, Prentice Hall, 1990, pp. 469-476.

[10] Hildebrandt, Darlene. "Light Painting Part One - the Photography - Digital Photography School." digital-photography-school.com. Digital Photography School. Web. 20 Mar 2014. <http://digital-photography-school.com/light-painting-part-one-the-photography>.