Psych221 Project Suggestions

These are project suggestions for Psych 221. We update this page regularly with ideas for projects.

• We describe how you should create the write-up on the Project Guidelines page.
• More than one person or group can work on the same project.
• Please turn in a short paragraph describing which project you will work on.
• If you see a project from a previous year that you would like to do - only better - you may do that.
• Some of the projects are marked special approval. We want to make sure you are the right person for that project. The unmarked projects we think are appropriate for anyone in the class.
• If you want to work on a project that is not listed, but perhaps it is helpful for your research, ask us.

There are links to earlier Psych 221 projects.

Projects 2015

This is a preliminary list of suggestions for now. These will become increasingly specific and be modified over the first few weeks of the class (BW).

Monitoring the environment

We have ideas about how to take calibrated underwater images captured by GoPro cameras to monitor the health of coral reefs. There are various components to the project (camera calibration, modeling of light transport through water, and automating image upload, storage and analysis).

Multispectral imaging for classification

Image classification is a very hot topic in computer vision. Most algorithms however operate on RGB camera channels, as if trying to mimic human visual system. In reality spectral information is much more abundant and can possibly be used to enhance classification algorithms. This project aims at investigating how much the accuracy of computer vision tasks can be improved if more spectrally sophisticated cameras were used . Specifically you will use a five band camera prototype to evaluate fruit and vegetable aging and perform flower classification, you will compare its performance to the performance of a classical RGB camera.

Skills Preferred: computer vision, machine learning

Mentor: Henryk Blasinski

An underwater, multispectral light source

Underwater imaging is quickly gaining importance not only due to its applicability in marine ecosystem monitoring, but also proliferation of inexpensive action cameras such as GoPro. Unfortunately, the colors in images acquired under water are severely distorted due to scatter and absorption phenomena. One approach to recover more spectral detail is to use active illumination techniques, this approach has proved to be very useful on the surface. In this project you will design and build an underwater, LED based multispectral light source that fits a standard GoPro size, underwater housing. With all the hardware in place you will have a chance to evaluate the accuracy and performance of active illumination spectral recovery in underwater scenarios. This is a hardware oriented project, you will be expected build and integrate the final system, which means that you should be familiar with soldering, PCB design and possibly even some CAD tools.

Skills Preferred: Hardware design experience

Mentor: Henryk Blasinski

A new approach to image processing pipeline (L3)

We have developed a new image processing pipeline (L3) for a digital camera based on machine learning and high speed processing with GPUs. L3 (Local, Linear, Learned) automates and customize image processing pipeline for a given design to speed camera development, leveraging advanced camera simulation and machine learning techniques.

Reference:

[1] Automating the design of image processing pipelines for novel color filter arrays: local, linear, learned (L3) method

[2] Automatically designing an image processing pipeline for a five-band camera prototype using the Local, Linear, Learned (L3) method

Accelerating L3 Processing Pipeline for Cameras with Novel CFAs on NVIDIA® Shield™ Tablets using GPUs

L3 classifies input image patches into categories that are local in space and response, and automatically learns linear operators that transform pixels to the calibrated output space using training data from camera simulation. The local and linear processing of individual pixels makes L3 ideal for parallelization.

This project aims to accelerate the L3 pipeline on NVIDIA® Shield™ Tablets using GPUs for real time rendering of videos. A tablet application that demonstrates the fast rendering feature of the L3 method is potentially to be accomplished. The learned linear operators and video data captured by a multispectral camera prototype will be provided. The CUDA / C++ (or CUDA / Matlab) code that works on a PC will be provided as a starting point.

Skills preferred: CUDA, Android Programming Mentor: Haomiao Jiang

High Dynamic Range Video Using the L3 Method

High dynamic range (HDR) imaging has advanced and translated to consumer products during the last decade. The majority of HDR techniques capture and combine multiple exposures to recover details and contrast simultaneously in dark and bright regions. However, this strategy requires the scene to be still during the multiple captures and is therefore inherently not suitable for HDR video acquisition. Altering the exposure settings in CFA is a promising approach for single-shot HDR image and HDR video acquisition, by trading-off spatial resolution. These novel HDR CFAs require time and effort to develop tuned image processing pipelines.

This project aims to explore the feasibility of L3 method on these novel HDR CFAs, particularly for HDR video application. Various HDR CFAs will be compared through the resultant images from the L3 processing pipeline in order to determine the optimal design.

References:

[3] Cheng, CH. et al., "High Dynamic Range image capturing by Spatial Varying Exposed Color Filter Array with specific Demosaicking Algorithm," IEEE Pacific Rim Conference on Communications, Computers and Signal Processing, 2009.

[4] F Yasuma, T Mitsunaga, D Iso, SK Nayar, Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum, Image Processing, IEEE Transactions on 19 (9), 2241-2253

Mentors: Qiyuan Tian and Steve Lansel

Designing L3 Processing Pipeline for a Camera Testkit with an RGB/W CFA

Clear pixels have been introduced to CFA to transmit much more light for low light photography (e.g. Aptina’s Clarity+ sensor, OmniVision’s Clear Pixel sensor inside Moto X and Sony’s Exmor RS RGB/W sensor). However, it is challenging to develop satisfying image processing pipelines that produce high image quality. In simulation, L3 has been demonstrated as an effective and efficient processing pipeline for RGB/W sensor (see the movie comparing L3 processing results for a conventional RGB sensor and a RGBW sensor at a series of light levels, link).

This project aims to design an L3 processing pipeline for a camera teskit with an RGB/W CFA following the procedures described in Reference [2]. The camera testkit will be first calibrated for camera simulation. L3 processing pipeline will then be created from the simulation and tested on the raw images captured by the testkit.

Mentor: Qiyuan Tian

Camera test kits

We have new camera test kits for a remote controlled camera (Olympus) and a high dynamic range sensor (secret). We would support projects that calibrate and use the capabilities of these test cameras.

Simulation projects describing visual perception (ISETBIO)

ISETBIO is an ISET based Matlab toolbox that can simulate human optics and photoreceptor sampling. With ISETBIO, we can accurately compute the optical irradiance image that impinges on the retina and the number of photons absorbed by human photoreceptors (cones) for a given scene. ISETBIO is capable of simulating human individuals with different optics (myopia, astigmatism, etc.) and cone mosaics (colorblind, density difference, etc.).

Simulate Fundamental Psychophysical Experiments

In this project, you will be expected to implement software simulations of various fundamental human psychophysical experiments that have been performed and validated. Specifically, these experiments could include, but not limited to, Vernier acuity, color discrimination contours, colorblind perceptions and modulation transfer function / contrast sensitivity simulations. You could also extend these basic experiments to seek answers for industrial problems: How large dpi of the display do we need, will different sub-pixel design make a difference. In this project, you will first generate noisy cone signals for human observers under different settings and use some classifiers to estimate the probability of observing the difference. Students who would like to participate in this project should choose at least one basic experiments and work with the mentor to investigate impact from different parameters.

Skills preferred: Matlab programming, machine learning

Mentor: Haomiao Jiang

Reproduce and Compare with Recent Papers

In this project, you are expected to reproduce the results from one recent paper with ISETBIO: Retina-V1 model of detectability across the visual field. The original code for the paper will be provided. You are expected to work with your mentor to rewrite it in ISETBIO and try to explain every difference (if any) from the original code.

Skills preferred: Matlab programming

Mentor: Haomiao Jiang

Visual simulation environment

We are designing a test environment in which we control a head-mounted display (from Oculus) to simulate the world as it would appear to someone with vision loss due to a retinal disease or condition. There are many components to this general project. Here are a few projects within this general one.

Image Capture In order to simulate degradations of the human visual system using images captured by a camera, it is necessary to know exactly how those images have been captured. Calibrate two cameras that are connected to a HMD (Oculus Rift).

Image Display Use the Oculus Rift to display images to human subjects that simulate (recreate) visual sensations that a person with a particular visual condition would see. This could be low vision, a type of color blindness, loss of central vision due to macular degeneration, or the effect of a retinal prosthesis in a blind person. We will help you use isetBio to create images to simulate one of these conditions. You will render the images on a calibrated Oculus Rift.

Information Display The goal of this project is to capture and display information so that people can track their movements and navigate in an environment with only visual input from the Oculus Rift. This will be accomplished by interfacing a Project Tango device with an Oculus Rift display. The Project Tango has sensors and software designed to track the 3D motion of the device and create a map of the environment using simultaneous localization and mapping (SLAM) algorithms. The output of the Project Tango is usually rendered on a laptop display. In this project, you will render the output on an Oculus Rift.

Projects 2014

Predicting human performance using ISETBIO

ISETBIO is an ISET based Matlab toolbox that can simulate human optics and photoreceptor sampling. With ISETBIO, we can compute the optical irradiance image that impinges on the retina and the number of photons absorbed by human photoreceptors (cones) for a given scene.

For this project, a tutorial script describing how to calculate cone absorptions will be provided and the students will be responsible for trying to answer one of following questions:

1. What's the maximum necessary display resolution (ppi) at certain viewing distance for Vernier acuity.
2. What's the maximum necessary display resolution (ppi) at certain viewing distance for contrast (CSF) resolution.

To answer these kind of questions, students are encouraged to build two scenes and use their preferred machine learning algorithm (e.g. SVM/Neural Network/Random Forest, etc.) to classify cone absorption sensor data for two same or two different scenes into "same" or "different" classes. When classification accuracy for cone absorption data is greater than a pre-determined value (say, 75%), we would predict that the observer can tell the difference between the two scenes. You can compare these predictions with published data from real human observers.

Preferred Knowledge: familiarity with at least one machine learning algorithm

Mentor: Haomiao Jiang

Hardware project: Build a Multispectral Imaging System

Build a multispectral imaging system based on a rotating color filter wheel and monochrome camera.

If you have experience in design and 3D printing, you can build several necessary parts.

If you have an interest in engineering applications for art history, there is an opportunity to use the system to capture images of paintings in the Cantor Arts museum.

Mentors: Henryk Blasinski and Joyce Farrell

Hardware project: Build an inexpensive spectrophotometer

In this project, you will build a simple spectrophotometer using a clean DVD-R, a USB webcam and stiff black card paper

Here's a website introducing how to do it: http://publiclab.org/wiki/spectrometer

After building the device, you need to compare it to the performance of a much more expensive (~$50K) spectrophotometer that we have in the lab

Mentor: Haomiao Jiang

Camera Image Quality Metrics

The International Standards Organization (ISO) is developing a set of camera image quality metrics to quantify the spatial resolution, noise and color accuracy of digital cameras. http://proceedings.spiedigitallibrary.org/data/Conferences/SPIEP/64097/829302_1.pdf

Many of these metrics have been implemented in ISET.

You can use ISET to calculate these metrics for simulated cameras that have different optical properties, numbers of pixels and image processing methods. You can also use ISET to simulate how each camera captures and processes natural scenes (e.g. faces and landscapes). You can then compare the metrics with the appearance of these images as they are rendered on a display.

In this project, you will use ISET and CPIQ to quantify and illustrate how the metrics and the images change when you decrease the size of camera pixels (and inversely increase the number of camera pixels). This method will allow you to analyze how resolution tradeoffs with sensitivity: Small pixels make it possible to increase the number of sensor pixels sampling the optical irradiance image, but it also decreases the amount of photons a small pixel can capture. What do you prefer, a high resolution noisy image or a low resolution clear image? How does this depend on the display, viewing distance, etc.?

Mentor: Joyce Farrell

ISET model for real camera

In this project, you will build an accurate ISET model for a physical camera we have. You will take pictures of known scenes, analyze the captured images, and try to build an ISET model.

The goal is for the ISET model of the camera to give approximately the same computational results as the RAW output from the real camera. The similarities could be measured by the noise, color, spatial resolution and etc. Analyzing the errors between the model and the real camera will determine the model's accuracy.

If time permits, you can try to implement an image processing pipeline for the camera and evaluate the performance of the processed images.

Mentors: Qiyuan Tian, Steve Lansel, Joyce Farrell

Analysis and Compression of L3 Filters

The L3 algorithm is a learned image processing pipeline for cameras. The algorithm learns optimal linear filters for a given camera based on training data, light level, illumination color, and optics. For a complete camera this may result in many (possibly hundreds) optimized filters. We believe the filters will be closely related for similar camera settings. The goal is to analyze the filters, store a compressed set of filters, and interpolate the needed filters from this compressed set. This way we only need to store a smaller set of filters and can extrapolate to lots of new camera settings. Here is a recent SPIE paper on L3: https://drive.google.com/file/d/0B0Gw85qGqJxhbXJlcmZjbmhOQ2s/edit?usp=sharing.

Mentors: Qiyuan Tian, Steve Lansel, Brian Wandell

ISET model for underwater imaging

With the proliferation of cameras such as GoPro more and more people have started taking underwater images. These usually have large amounts of distortion, both spectral and spatial, originating from the medium in which the image was taken. Rather than experiment in the real world, the impact of different light transport phenomena on RGB images can be understood via simulation environments. In this project you will implement, enhance and integrate with ISET the underwater image simulation system described in the paper below.

Color image simulation for underwater optics

Mentors: Joyce Farrell and Henryk Blasinski

App for Programmable Camera in iOS / Android

We have a prototype programmable camera to be used with iOS or Android devices. The project's goal is to make an app that will run on iOS or Android and uses the camera. Think of an interesting camera app, and we can work together to build it. Prior experience in iOS or Android is needed.

Mentors: Steve Lansel and Munenori Fukunishi

Image classification with a five band camera

Recently image classification and object recognition have become very popular topics. Large majority, if not all, algorithms use images acquired with traditional, three channel (RGB) cameras. The goal of the project is to evaluate the performance of the state of the art algorithms applied to images captured with a five band camera. Will the recognition/classification performance change, and if so by how much? To get the flavor of the project you can look at the following paper:

Multispectral SIFT for scene category recognition

Mentors: Henryk Blasinski, Steve Lansel

Analysis of a real camera lens

Can we characterize how a lens blurs a point of light (point spread functions or psfs) by analyzing camera images of test targets that are displayed on a color monitor? This project has many possible variations.

1. Illuminate red, green and blue pixels on a display and capture an image of the display with a camera placed on a tripod a far distance away. Vary the pattern of red, green and blue pixels (e.g. noise pattern).
2. Estimate the psfs of a real camera with a real lens by analyzing camera images of displayed targets. Use a prosumer digital camera and vary 1/f# and observer how the estimated psfs change.
3. Estimate the psfs for different field heights, wavelengths and depths.
4. Use the estimated psfs to predict camera images of other displayed "natural" images, such as a face. Compare the predicted camera images to actual camera images.

Here are links to papers that describe a method for empirically estimating the psf of a camera lens. The links include code that you can download

1. http://www.cs.ubc.ca/labs/imager/tr/2013/SimpleLensImaging/
2. http://www.ipol.im/pub/art/2012/admm-nppsf/

People: Brian Wandell, Andy Lin, Joyce Farrell

Psf analysis and image deblurring using a simulated camera lens

The point spread function(psf) of a lens is an extremely important lens property. One possible application of knowing the psf is image deconvolution (deblurring). Deconvolution can drastically improve image sharpness. The following paper provides a good technique for estimating a psf and deconvolving an image with that psf: http://www.cs.ubc.ca/labs/imager/tr/2013/SimpleLensImaging/

Tasks

1. Andy Lin will provide simulated camera images of several different types of spatial test targets. Your task will be to use the code from http://www.cs.ubc.ca/labs/imager/tr/2013/SimpleLensImaging/ to estimates psfs from the simulated camera images.
2. To evaluate how well the psf estimation code works, compare the estimated psfs to the known psfs that Andy used to generate the simulated camera images.
3. As another evaluation technique, use the estimated and known psfs to "deblur" a blurred image containing a secret message using the deconvolution code downloaded from the same site. The secret message will only be legible after proper deconvolution of the image. Andy will provide this blurred image.

Mentor: Andy Lin

Medical imaging: Super resolution microscopy

http://en.wikipedia.org/wiki/Super_resolution_microscopy

Super resolution microscopy refers to methods that build up a high resolution image of target by integrating many multiple images of the target illuminated such that only a small subset of the image points are captured in any one image. The camera image then samples a subset of the pixels in a high resolution image. The location of the pixels in many camera images are combined to construct a single full high resolution image of the target. By placing a point at the center of each sampled point, one can get very accurate spatial information about the location (phase) of illuminated points in the target. Because the center of a dot is smaller than the lens psf, some people assert that super-resolution methods beat the limit of lens diffraction. But you know better than that. Diffraction is a limit that no earthly being can beat. Nonetheless, by sampling with stochastic and sparse arrays of pixels, one can do a better job of locating the center of sampled points and hence build up a higher resolution image.

You can write an ISET simulation to test one of these super-resolution methods.

Alternatively, you can test methods for super-resolution imaging using real camera images. For example, take a camera image of a displayed image, (such as a face or a high resolution test chart) . Then take a capture a series of images of the display when only a subset of the pixels in the face (or chart) are illuminated. The illuminated pixels in each subset will be far away from each other such that the optical images of the pixels illuminated in each image do not overlap. You can further experiment by taking a blurry image of a face (say, by setting the caemra 1/f# to 12). Then, display subsets of pixels of the face that are widely separated. Find the location of the center of each illuminated pixel and combine the data to create a non-blurred camera image.

Mentor: Brian Wandell, Haomiao Jiang

Eulerian video processing (Bill Freeman thing)

Repeat one of the experiments from Bill Freeman. There published paper could be found from http://people.csail.mit.edu/mrub/vidmag/

Also, you need to compare the results for cameras with 3 color channels (rgb) and with 5 color channels (prototype in our lab).

Scene 3D System

The goal of the Scene3D project is to simulate the complete imaging pipeline for 3D scenes, from the scene to the lens , to the sensor and to the mage processing. Simulations of sensor and image processing are implemented in ISET. The novel part of Scene3D involves using a technique in 3D graphics called ray-tracing, which produces a physically accurate simulation of light rays that are refracted through lenses and towards the sensor. We modified the PBRT ray-tracer to simulate the important effects of diffraction and to be able to handle complex lenses and multispectral inputs and outputs. The end goal of the Scene3D project is to provide an infrastructure for rapid image systems prototyping.

[Scene3D project https://github.com/ydnality/Scene3D]

One important aspect in photography involves color balancing. Often times, photographs taken under different illuminant conditions will produce images that don't appear natural. For example, images taken under indoor tungsten lighting will exhibit an unnatural yellow/orange tint. These images must be corrected for in order to appear natural.

This class project involves applying the camera pipeline simulation provided by the Scene3D infrastructure for use in designing a color-balancing algorithm.

Tasks

1. Start with a 3D radiance scene generated by Andy Lin. Modify the parameters of the scene to make different renderings of the scene under different light conditions.
2. Design and implement an intelligent method for "correcting" (color-balancing) the illuminant.
3. (Challenge/Optional) Design a color balancing method that is able to correct for scenes with 2 or more different illuminants.

Mentor: Andy Lin

PBRT and Zemax optics modeling

Scene3D use a combination of PBRT and ISET to simulate the complete imaging processing pipeline of a digital camera. The unique contribution of Scene3D is that it applies a technique in 3D graphics called ray-tracing, to produce a physically accurate simulation of light rays as they are refracted through lenses and towards the sensor. We modified the PBRT ray-tracer to simulate the important effects of diffraction and to be able to handle complex lenses and multispectral inputs and outputs. However, we have yet to verify this pipeline empirically.

One way we plan to evaluate our modifications to PBRT is to compare the point spread functions we generate with point spread functions generated by Zemax, a well-established software package used by many optics professionals. We provide a Zemax macro that can be used to generate the PSFs that ISET needs. Although Zemax can produce physically accurate PSF's, it cannot produce rendered physically accurate 3D multispectral images like PBRT.

Tasks

1. This project would involve taking several PBRT multi-element lens models, and creating equivalent Zemax models.
2. Use the Zemax to ISET interface to generate the data necessary for the ISET simulations.
3. PSF's using the PBRT model will be provided as ISET optical images. We provide a Zemax macro that can be used to generate PSF for lenses that are modeled in Zemax.
4. Compare and analyze the PSF's produced by these two different methods under different aperture and distances as verification.

Experience with Zemax is preferred.

Mentor: Andy Lin

Gesturing in a Virtual 3D space

The Holografika multi-projector display system creates a 3D light field that people can view without the need for special googles. Leap Motion is a controller that can sense small finger movements using an infrared led and camera. We linked these two devices so that users can grasp and move virtual objects in the 3D light space created by the Holografika display. We also linked the Leap Motion to a conventional stereoscopic display that uses an LCD with shutter goggles. The goal of this project is to compare how well users can use the hand-gesture controller to move objects in the virtual 3D spaces created by the two different types of displays.

The project has possible variations. - You can find a suitable OpenGL app or game from the Leap Motion Airspace app store that measures agility to quantify the learning rates of new users. The objects floating in front of the Holografika display will be aligned to the users hands in that 3D space, but not so with the flat LCD display. Possibly include mouse mode in the tests. -Using a 3D top-down street view map of London, test users skills at finding a location by panning and zooming a holographic 3D map of London on both kinds of displays, using hand gestures. Does the user's self-reported confidence correlate to measured performance and how does display type affect that? Use the metrics to predict the actual benefit for different kinds of organizations to transition from mouse control to (hands in air) gesture devices with 2D and Holographic displays.

The equipment is calibrated and available in Packard 070.

Here is a link to the companies involved: www.holografika.com and www.leapmotion.com

You can watch a video of the talk by the inventor of Holografika (Tibor Balogh) at https://talks.stanford.edu/scien/scien-colloquium-series/

Mentors: Dave Singhal, Harlyn Baker, Peter Kovacs

Projects 2013

Camera Forensics

You are presented with a digital image and asked to determine if it has been manipulated and if so to localize the manipulation in the image. Color filter array (CFA) interpolation generates a tell-tale signature in a digital image that can be used in a forensic setting. CFA interpolation leads to strong correlations between a specific subset of pixels and their spatial and chromatic neighbors. Build a classifier that takes as input a digital image and automatically detects which parts of an image do and do not exhibit the expected CFA correlations. Begin by generating a synthetic set of test images that have undergone your choice of CFA interpolation. Test your forensic analysis on these uncompressed images and then quantify the efficacy of your approach on increasingly more JPEG compressed images. Disputes often erupt over the provenance of photos. Consider how you might use your new forensic technique to distinguish between images taken from different types of cameras (e.g., a Canon PowerShot vs. a Nikon D-series).

References

1. A Survey of Image Forgery Detection
2. Exposing Digital Forgeries in Color Filter Array Interpolated Images

Tasks

1. We provide you with training images
2. You develop the classifier based on the papers
3. We provide you with test images to see how you did

Image Forensics

You are presented with a JPEG image and asked to determine if it originated directly from a camera/mobile device, or if it was re-saved one or more times. Multiple compressions at different compression levels leave behind specific statistical artifacts in the distribution of DCT coefficients. These artifacts can be used to distinguish between singly and multiply compressed images. Build a classifier that can distinguish between singly and doubly compressed images (assume that the second compression level is different than the first). Validate your classifier on a large data set of images. Quantify the conditions under which the classifier is effective and not. Extend your classifier to distinguish between one, two, and three compressions. The expert forger becomes aware of your forensic technique and writes a special purpose encoder that will re-save a JPEG image with the same compression quality as the original. Consider how you might counter this by detecting multiple compressions made with the same compression setting.

References

1. A Survey of Image Forgery Detection
2. Statistical Tools for Digital Forensics

Tasks

1. We provide you with training images
2. You develop the classifier based on the papers
3. We provide you with test images to see how you did

Turbulence removal

X. Zhu and P. Milanfar, "Removing Atmospheric Turbulence via Space-Invariant Deconvolution" IEEE Trans. on Pattern Analysis and Machine Intelligence vol. 35, no. 1, pp. 157-170, Jan. 2013

Also see related talk and Project page

Options

1. You obtain by measurement or simulation example images and then use their methods.
2. You develop a variant of their method, exploring deconvolution, registration, or some other part of the algorithm more deeply than in the original paper.
3. You find another approach and compare that approach to this one.

Photon calculator utility (ISET)

Build a program, perhaps based on the ISET library, that calculates the spectral irradiance at the sensor from the scene radiance and a specification of the optics. Doing this for diffraction-limited optics, specifying only the f/#, is sufficient.

The utility should be backed by a wiki page that illustrates all of the steps in doing that calculation. This project should produce an educational and useful calculator.

• Doing an implementation that can run on a browser on the Internet is best.
• Doing a straight Matlab implementation with a nice GUI is also good.
• Implementing the ISET (Matlab) routines as a Python calculator has value, as well.

Updating Wikipedia

Help us make Wikipedia better. There are surprisingly many Wikipedia entries on imaging and human vision that are just a few sentences long. Look-up for example: 'Troland', 'Stiles-Crawford effect', 'Photopic vision', 'Human PSF' or 'Active Pixel Sensor' to see how poor these entries are. Your mission, should you choose to accept it, is to improve these (or other) entries. Think of your work as of a paper, which is published online, rather than in a .pdf format. Of course, just as with writing any research paper, your work should start with a thorough literature review, select the relevant pieces of information and write them up in a way approachable to a non-expert in the field.

Neuroimaging (special approval)

With the opening of Stanford's Center for Cognitive and Neurobiological Imaging (CNI), we now have access to a large number of MR scans of the human brain. We are also closely connected to the MR hardware and image processing algorithms.

While this course is not specifically about neuroimaging, some of the methods in the course might be usefully applied to the data collected at the CNI. For students already working in MR and interested in such signal processing, we might be able to develop some projects that build on your interest.

Two possible projects are algorithms to:

• Identify when two MR images are of the same brain (brainprint), even if they were acquired using different contrasts.

• Evaluate image quality and MR artifacts

Scene database for computer vision testing (special approval)

Build scenes, say using Blender and PBRT, that we can run through the ISET simulation to produce images. Then analyze these calibrated scenes using computer vision algorithms to derive the depth, illumination, and shading. See this example page for folks who created a database from real, rather than simulated, scenes.

Color balancing (special approval)

Color balancing refers to the process of converting camera rgb data into display rgb values. If one simply copies the sensor pixel values into the display values, the resulting image will not generally be a good color representation of the original scene. An important step in the image processing pipeline is to transform camera rgb values to display values such that the display image appears to match the original scene that was captured.

A simple and common approach to color balancing is to make an educated guess about the scene illumination based on an analysis of the camera rgb values. The estimated illuminant is used to select a color transform (typically a 3x3 transform or a look-up table) that maps camera rgb values into human sensor (xyz) values for an ideal illuminant, such as daylight. The goal of this transform is to render the scene that the camera captured as if the scene were illuminated by daylight.

Most camera processing pipelines use a standard illuminant called D65 as the ideal rendering illuminant. As far as we know, no one has tested the assumption that people prefer to view objects illuminated by D65. The preferred rendering illuminant may also depend on the objects that are being rendered..

The project will use hyperspectral data of faces, fruit and vegetables and outdoor scenes, and spectral power distributions of different illuminants to generate images that people will view on calibrated displays. People will be asked to indicate which color renderings they prefer. In this way, we will collect preference data about preferred rendering illuminants. The preference data will provide a useful guide for engineers who are designing color balancing methods.

Hyperspectral video (special approval)

Help us build and evaluate a hyperspectral video system based on led lights synced with a monochrome video camera. Capture hyperspectral video images of human faces and estimate pulse rate by the change in color sensor values over time (see http://people.csail.mit.edu/mrub/papers/vidmag.pdf)

Biology of the mouse eye image formation (special approval)

There is a huge amount of biology done in mouse. There is a movement to study the mouse retina in particular. To study the retina, we would like to be able to understand how the cornea lens in the mouse blur the retinal image.

Adaptive optics to the rescue: Williams and his colleagues analyzed the optical quality of the mouse eye. Specifically, they measured the wavefront aberrations from 20 wild type mice. They provide the data in their article.

Optical properties of the mouse eye

Brainard, Hofer and I have written a wavefront toolbox in Matlab that enables us to specify the wavefront aberration and calculate retinal images in ISET. This project is to use our software to reproduce Figures 10 and 11 from the paper.

You can do this! If you do, many people will cite your project because there are many people who work on mouse.

Active LED-based illumination (special approval)

These days LEDs can produce high intensity light with well defined spectral properties. We are interested in a hardware system that allows to control both the on/off times of a set of LEDs, as well as their intensity using a simple Arduino microcontroller. One way you can do this, and we have a working prototype (refer to this project), is to use pulse width modulated signals to control the duty cycle of an LED. If you operate at high enough frequency, then you will perceive the rapidly flickering LEDs as having lower or higher intensity. In this project, however, we are interested in controlling the LED intensity more directly, so that even at the micro-time scale you control the LED intensity directly, rather than switch it on/off.

Projects 2012

Image processing

Hyperspectral Imaging

Analysis of hyperspectral images of paintings by famous artists

Consumer digital cameras capture electromagnetic energy in three different spectral bands. Multispectral and hyperspectral cameras capture electromagnetic energy in many more spectral bands. We used two different hyperspectral cameras to capture images of several paintings in the Cantor Arts Museum. One of the cameras captures images in 160 different spectral bands ranging between 400 and 1000 nm (visible and near-infrared or VNIR). The other camera captures images in 256 different spectral bands ranging between 1000 and 2500 nm (short-wave infrared or SWIR). There is a very large literature on hyperspectral imaging of paintings that we will use to guide our analysis of the data we have already collected. (http://www.springerlink.com/content/80342384844k0r21/fulltext.pdf) In particular, we should be able determine if there is a drawing beneath the painting (an “underpainting) and to characterize the paint pigment. This analysis will allow us to determine the history of the painting and assess its originality. We hope that this project will serve as the groundwork for an exhibit at the Cantor Arts Museum. (JEF and TS) Here is a nice website that describes methods used in art forensics (http://www.webexhibits.org/pigments/intro/look.html)

Analysis of hyperspectral images of live organs during surgery

Several research labs are investigating the advantages of hyperspectral imaging in robotic surgery. This is because hyperspectral cameras can capture a wider range of spectral data, including electromagnetic energy that the human eye cannot see. One of the challenges is how to map information that is normally invisible to surgeons onto visible images that enhance the ability to discriminate between different tissue types in a meaningful way. We have collected hyperspectral images of organs in a live pig during surgery. This project will analyze this data to determine if information in the invisible regions of the electromagnetic spectrum (> 700nm) can be used to enhance the information that surgeons see during an operation. (see http://www.intechopen.com/source/pdfs/9221/InTech-Hyperspectral_imaging_a_new_modality_in_surgery.pdf ) (JEF and TS)

Colorimetric reproduction of human faces

We collected VNIR (160 narrowband spectral images ranging between 400-100 nm) hyperspectral images of human faces, outdoor scenes, still life (fruit) and paintings. The hyperspectral image data can be used to generate a representation of spectral reflectance of the objects in a scene and the spectral power of the scene illumination. These representations can, in turn, be used as input to the ISET digital camera simulation software. ISET can then be used to predict the output of digital cameras with different color channels. For example, one can simulate a digital camera with three or more color channels, and vary the spectral sensitivities of each of the color channels. One can also vary the spatial distribution of those channels. Finally, one can vary both the demosaicking and color balancing algorithms in the digital camera. This project provides an excellent tutorial on how a digital camera works and gives you the opportunity to develop your own color imaging processing algorithms. (JEF and TS)

References and Web Links

Novel detectors for RGB and NIR

Applications, devices and algorithms for the separation of visible and near infrared signals in monolithic Si sensors

ANALYSIS OF FOVEON MULTI-SPECTRAL SENSOR FOR COUNTER-CAMOUFLAGE, CONCEALMENT AND DECEPTION APPLICATION by Devon Courtney Nugent

Using NIR to enhance visible data

Several Susstrunk lab papers. Some others.

http://infoscience.epfl.ch/record/148419/files/81_susstrunk_v5.pdf http://infoscience.epfl.ch/record/153994/files/de24567-susstrunk.pdf

http://www.comp.nus.edu.sg/~dfanbo/papers/VisualEnhanceHSI_Kim_PR2011July.pdf

http://gitl.sysu.edu.cn/papers/cvpr-2008-zhang.pdf

http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5652900&tag=1

SIFT on visible and NIR

http://www.intechopen.com/source/pdfs/15840/InTech-An_exploration_of_color_fusion_with_multispectral_images_for_night_vision_enhancement.pdf

NIR Flash

- NIR flash

http://www.comp.nus.edu.sg/~zhuoshao/NIRFlash/nirflash_icip2010_high.pdf

Photo retouching metric (Kee and Farid)

A perceptual metric for photo retouching Eric Kee and Hany Farid

Department of Computer Science, Dartmouth College, Hanover, NH 03755 October 19, 2011 (received for review July 5, 2011)

In recent years, advertisers and magazine editors have been widely criticized for taking digital photo retouching to an extreme. Impossibly thin, tall, and wrinkle- and blemish-free models are routinely splashed onto billboards, advertisements, and magazine covers. The ubiquity of these unrealistic and highly idealized images has been linked to eating disorders and body image dissatisfaction in men, women, and children. In response, several countries have considered legislating the labeling of retouched photos. We describe a quantitative and perceptually meaningful metric of photo retouching. Photographs are rated on the degree to which they have been digitally altered by explicitly modeling and estimating geometric and photometric changes. This metric correlates well with perceptual judgments of photo retouching and can be used to objectively judge by how much a retouched photo has strayed from reality.

• Watch a video of the SCIEN talk by Farid at Stanford, Jan. 31.
• Implement and analyze the algorithm.
• Perform experiments with existing online pictures
• Suggest improvements

Visibility of movie subtitles

A persistent problem in watching foreign movies is that sometimes the subtitles are illegible. Why? Because the contrast of the default background that is assumed is wrong and you have white characters on a light background. I assume this is done automatically because it is too expensive to have people judge frame by frame whether the script is visible. Need I say more. Some automated system that could assess the brightness of the standard background space where subtitles are printed and then adjust the contrast to be legible would be a huge improvement for the industry.

E. Markman, a committed viewer of subtitled films.

Image Quality

3D Image Quality Metrics

Develop algorithms for Shooting in 3D and Displaying in 2D. Explore ways in which to improve 2D rendering of 3D content in order to enhance “immersive video”.

Optics

Wavefront Toolbox

(BW)

Advances in adaptive optics now make it possible to measure the wavefront aberrations of the living human eye. Many groups are making these measurements in both control subjects and subjects with different types of optical dysfunctions.

These aberrations are usually specified in a way that is difficult to apply to image processing: The aberrations are specified as the weights on a set of Zernike polynomials. It is a simple matter of programming to convert these polynomial weights to a point spread function that can be applied in image processing algorithms.

We have received software from experts on this topic that implements the conversion. We can probably obtain a large number of samples of measurements from different categories of human eyes. In this project, we would create a web-site to convert the Zernike polynomials to point spread functions and illustrate how those pointspread functions would influence the quality of the optical image falling on the retina.

As we accumulate additional summaries of the human measurements, we might look for statistical patterns that might be explained in terms of the biological properties of the human cornea and lens.

See: Chromatic and wavefront aberrations: L-, M- and S-cone stimulation with typical and extreme retinal image quality
Florent Autrusseau, Larry Thibos, Steven K. Shevell
Vision Research 51 (2011) 2282–2294

Integrating 3D Distributed Ray Tracing and Image Quality

(BW), (AL), (JEF)

PBRT

Radiance

RenderToolbox

Neuroimaging

(AT), (AM), (RFD), (GS)

With the opening of Stanford's Center for Cognitive and Neurobiological Imaging (CNI), we now have access to a large number of MR scans of the human brain. We are also closely connected to the MR hardware and image processing algorithms.

While this course is not specifically about neuroimaging, some of the methods in the course might be usefully applied to the data collected at the CNI. For students already working in MR and interested in such signal processing, we might be able to develop some projects that build on your interest.

• Intelligent compression algorithm for multi-channel image data stored in frequency space (p-file compression)

• Algorithm to classify volumes that contain brains in a database of MR images that includes phantoms, squash, fruits, etc. (brain detector)

• Algorithm to identify when two MR images are of the same brain (brainprint), even if they were acquired using different contrasts.

• We can also do another one on MR artifact detection (so many artifacts, so few projects...)

Suggestions and projects from previous years

Web page of Project Ideas for 2011

Web page of Project Ideas for 2010

PDF of Project Ideas in 2008

To see projectsfrom previous years, visit SCIEN Class Projects Page.