IOS app for programmable camera

Introduction

Programmable cameras

A programmable camera is a portable digital camera which can be controlled using a smartphone or any Wi-Fi enabled remote device. These cameras have a fixed image sensor, but can have different lenses mounted on it. Today there are a lot of APIs being made available for the developers to integrate smartphones (Android and iOS) with these Wi-Fi enabled Digital cameras. The objective of this project was to use a prototype programmable camera and build an application on an iPhone which can make it easier for users to create cool effects on photos they have taken.

Smartphone Application

Users can take great pictures with these digital cameras by changing properties like ISO, Exposure, F-Number, white balance, etc. Unfortunately, the users need prior knowledge of photography to operate these cameras properly. Besides this, there is no preview available, when the user changes the properties of the camera and these cameras are low on processing memory.

Flowchart showing the phone acting as a controller to the Programmable camera

Our team decided to therefore, leverage the processing capability of the smartphone and create an application that would provide post-processing capabilities. We decided to implement real-time filters for the camera so that the users can see the effect of the filters applied on the screen before capturing the picture. The picture above shows how the smartphone acts as a controlling interface for the camera.

Project

Broadly, we decided to design an iOS application for the programmable camera that displayed image information as well as showcased filters and image processing techniques. By doing this, we hoped to learn more about the science behind these techniques. We also hoped to create the foundations for an application that users will eventually use to create better and more interesting photos.

We decided to come up with some simple solutions to the following questions.

What can we do to take better pictures?

Before starting work on this app, we thought about what to include in it that would help users take better pictures. One of the first ideas we had was to overlay a real-time image histogram over the live camera stream seen by the user. This is a great way to help users capture the sort of picture they want by adjusting angles and lighting to achieve a certain kind of histogram.

Ultimately, we implemented in our app two common histograms useful for photographers: the luminosity histogram and the RGB histogram. Both these histograms have relative pixel amount for a given color on the Y-axis. The luminosity histogram shows brightness values on the X-axis, i.e., the darkest shadow (pure black) is located at the origin and the brightest highlight (pure white) is at 255 (in terms of the 256-step color values scale). The RGB histogram also has the same X-axis going from black to white, but shows absolute color peaks (red, green, and blue) as opposed to a brightness distribution.

What can we do to take more interesting pictures?

We decided to implement a few real-time image filters in our app to add interesting effects to the live camera stream (and in turn, the pictures taken). Further down, we’ve discussed some of the filters we added to our application.

Methodology

User Interface of the Application

The User interface for the application is fairly simple to understand for someone who is not very well versed with the technical parameters of a digital camera. The top bar contains a menu of options which the user can use to select the property they wish to change. There is a ‘Filters Button’ which is a part of this menu and is described in greater detail in the section below. When any of the buttons are clicked, a table of options is launched in the Live Camera View.

Filters Button

Added to the top control panel with other adjustable options such as camera ISO, FNumber, Exposure, etc. When this button is tapped by the user, it gets highlighted in blue and triggers a table of filters which the user can choose from. The figure below shows the filters menu. Upon choosing a filter option, an appropriate filter is applied to the live camera stream. The user can pick and choose the filters or change the parameters and the live camera stream feedback changes accordingly. So, this makes it easy for the user to follow the changes and choose the best setting they like before capturing the image.

Histogram and histogram equalization

Image Histograms: Histograms are graphical representations of tonal and luminance distributions in an image. Inspecting histograms of live preview in a camera can help the photographers to adjust the exposure values of their cameras. Also, the RGB histograms can be used as a guide to selectively adjust exposure for different colors. You can make sure that you are not losing details by underexposing or overexposing certain colors. That's why we incorporated the live histogram feature into our app. The GPUImage library provides APIs by which histograms of images can be extracted. We used the APIs GPUImageHistogramFilter and GPUImageHistogramGenerator to extract a histogram and plot it. After extracting the histogram image, we blend it into the live preview image using GPU blend filter. The screenshots are attached.

Histogram equalization and histogram matching: Equalizing a histogram is a technique that is used to adjust pixel intensities to enhance contrast. It's a transformation by which the image pixel intensities (often ranging from 0 - 256) are normalized and the image is transformed to an image with a linear CDF of gray levels. Since CDF can be inverse transformed, the intensities can be got back from this linearized CDF.

In a grayscale image or a monotone image, this works very well and results in better exposure of underexposed parts. Both in live images and when applied to stored images, grayscale equalization is shown to result in improving the details of the image. We could not find histogram equalization APIs in GPUIMage library. So, we used the Accelerate framework from iOS to implement histogram equalization.

Equalizing the histogram on the live image preview is useful tool for the user to adjust exposure and lighting for the scene that he is shooting.

In a color image however, this equalization needs to be repeated for the three color channels R,G and B separately. However, applying the same levels in all bins for the three channels results in color distortion and the tonal distribution of the image is messed up. This is because rarely does any image have equal intensities in all the three color channels.

We show an example of a distorted color image when the R, G and B histogram is equalized.

The same images with their RGB histograms blended over the images are as shown.

There are many RGB equalization techniques in research and in practice. In future, we plan to incorporate one technique which is called the histogram matching technique in our camera app. This will allow the user to specify a reference photo and ask the camera to match the histogram of the current view to the specified image’s histogram. This will be useful for user’s to mimic the appearance of good photos.

Conclusions

We have developed an iOS application for a programmable camera. We explored various image processing libraries that are used in iOS development framework.

What we learnt

We had loads of fun in developing this application and learn a lot. The most important things that we learnt would be, 1. To develop an app for iOS, which is apple's OS for iPhones and iPad. 2. The structure of image processing pipeline in a digital camera in general and a programmable camera in particular. 3. To use the various amazing open source image processing libraries available for iOS. 4. The tradeoffs of applying complex operations on live previews Vs

References

[1] https://github.com/BradLarson/GPUImage

[2] https://developer.apple.com/library/ios/documentation/Accelerate/Reference/AccelerateFWRef/_index.html

Appendix 1

Due to the NDA we signed with the company that generously loaned us the programmable camera, we are not allowed to post or describe the code behind our application.