Psych221 Project Suggestions
These are project suggestions for Psych 221. We update this page regularly with ideas for projects.
- We describe how you should create the write-up on the Project Guidelines page.
- More than one person or group can work on the same project.
- Please turn in a short paragraph describing which project you will work on.
- If you see a project from a previous year that you would like to do - only better - you may do that.
- Some of the projects are marked special approval. We want to make sure you are the right person for that project. The unmarked projects we think are appropriate for anyone in the class.
- If you want to work on a project that is not listed, but perhaps it is helpful for your research, ask us.
There are links to earlier Psych 221 projects.
Projects Fall 2016
The projects this year are centered around opportunities for producing new image systems simulations that incorporate graphics, camera simulation, and image processing.
Autonomous vehicle sensors
Project 1 ideas
Forensic analysis of the fatal Tesla car crash.
On May 7, 2016, a 40-year old man was killed when his Tesla crashed in Florida. There are many articles describing the accident and speculating about the cause. For example, Telsa reported that “Neither Autopilot nor the driver noticed the white side of the tractor trailer against a brightly lit sky, so the brake was not applied.”
The Tesla car had a Mobileye system that includes several cameras and an image processing module. There is enough known about the imaging sensors in the Mobileye system to predict the images the sensors would have captured for different types of scenes.
This class project will use the ISET digital camera simulation software to model different scenes and image sensor parameters (e.g. exposure duration and video rate). Extra bonus points if you use machine learning (svm) to determine whether a system can detect the difference between different types of scenes. For example, is can a system detect the difference between a “white side of a tractor trailer” and “a brightly lit sky”?
References:
Inside the Self-Driving Tesla Fatal Accident, by Anjali Singhvi and Karl Russell, NYTimes, July 12, 2016
Tesla faults brakes, but not autopilot, in fatal crash. By Neal Boudette, Business Day, July 29, 2016
Mobileye EMP evaluation platform
Fatal crash prompts federal investigation of Tesla self-driving cars, by Sam Thielman, The Guardian, July 13, 2016
Autopilot 2.0 adds more sensors to be better than ever, report says, by Chris Mills, BGR, Aug 11, 2016
Tesla Autopilot 2.0: retrofit to next gen sensors likely to be available for some owners, Fred Lambert, electrek, August 6, 2016
Tesla Autopilot 2.0: next gen Autopilot powered by more radar, new triple camera, some equipment already in production, Fred Lambert, electrek, August 11, 2016
Researchers trick Tesla Model S. Autopilot, Brandon Turkus, Autoblog, Aug 4, 2016
Another crash on Telsa autopilog, another driver admits to not paying attention, was cleaning his dash, by Fred Lambert, electrek, August 19, 2016
Tesla Model S, Wikipedia
Understanding the fatal Tesla accident on Autopilot and the NHTSA probe, Fred Lambert, July 1 2016
“WTF is the deal with driverless car guru George Hotz’s Comma Points?”, by Joe Carmichael, July 7, 2016
Uber and Volvo partner up, robot ride-sharing starts this summer, by Jonathan Gitlin, ARS Technica, Aug 18, 2016
Comma.ai startup in SF
Drive.ai startup in SF
Nauto – startup in Palo Alto
Learning a driving simulator, by Eder Santana and George Hotz
Object Classification
Henryk
Light field cameras
Henryk Algorithms that derive depth estimates from the light field camera data
Underwater simulations
Henryk, Joyce
Fluorescence simulations
Henryk
Optics extensions
Brian
Model RealSense camera data
Achin
Virtual Reality Facebook camera design simulation
Trisha?
Projects Fall 2015
A new approach to image processing (L3)
We have developed a new image processing pipeline (L3) for a digital camera based on machine learning and high speed processing with GPUs. L3 (Local, Linear, Learned) automates and customize image processing pipeline for a given design to speed camera development, leveraging advanced camera simulation and machine learning techniques.
Reference:
[2] Automatically designing an image processing pipeline for a five-band camera prototype using the Local, Linear, Learned (L3) method
Accelerating L3 Processing Pipeline for Cameras with Novel CFAs on NVIDIA® Shield™ Tablets using GPUs
L3 classifies input image patches into categories that are local in space and response, and automatically learns linear operators that transform pixels to the calibrated output space using training data from camera simulation. The local and linear processing of individual pixels makes L3 ideal for parallelization.
This project aims to accelerate the L3 pipeline on NVIDIA® Shield™ Tablets using GPUs for real time rendering of videos. A tablet application that demonstrates the fast rendering feature of the L3 method is potentially to be accomplished. The learned linear operators and video data captured by a multispectral camera prototype will be provided. The CUDA / C++ (or CUDA / Matlab) code that works on a PC will be provided as a starting point.
Skills preferred: CUDA, Android Programming
Mentor: Haomiao Jiang
High Dynamic Range Video Using the L3 Method
High dynamic range (HDR) imaging has advanced and translated to consumer products during the last decade. The majority of HDR techniques capture and combine multiple exposures to recover details and contrast simultaneously in dark and bright regions. However, this strategy requires the scene to be still during the multiple captures and is therefore inherently not suitable for HDR video acquisition. Altering the exposure settings in CFA is a promising approach for single-shot HDR image and HDR video acquisition, by trading-off spatial resolution. These novel HDR CFAs require time and effort to develop tuned image processing pipelines.
This project aims to explore the feasibility of L3 method on these novel HDR CFAs, particularly for HDR video application. Various HDR CFAs will be compared through the resultant images from the L3 processing pipeline in order to determine the optimal design.
References:
[3] Cheng, CH. et al., "High Dynamic Range image capturing by Spatial Varying Exposed Color Filter Array with specific Demosaicking Algorithm," IEEE Pacific Rim Conference on Communications, Computers and Signal Processing, 2009.
[4] F Yasuma, T Mitsunaga, D Iso, SK Nayar, Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum, Image Processing, IEEE Transactions on 19 (9), 2241-2253
Mentors: Qiyuan Tian and Steve Lansel
Designing L3 Processing Pipeline for a Camera Testkit with an RGB/W CFA Clear pixels have been introduced to CFA to transmit much more light for low light photography (e.g. Aptina’s Clarity+ sensor, OmniVision’s Clear Pixel sensor inside Moto X and Sony’s Exmor RS RGB/W sensor). However, it is challenging to develop satisfying image processing pipelines that produce high image quality. In simulation, L3 has been demonstrated as an effective and efficient processing pipeline for RGB/W sensor (see the movie comparing L3 processing results for a conventional RGB sensor and a RGBW sensor at a series of light levels, link).
This project aims to design an L3 processing pipeline for a camera teskit with an RGB/W CFA following the procedures described in Reference [2]. The camera testkit will be first calibrated for camera simulation. L3 processing pipeline will then be created from the simulation and tested on the raw images captured by the testkit.
Mentor: Qiyuan Tian, Haomiao Jiang
Color Matching in Dentistry
When dentists fill a cavity, they must select a composite material. When they replace a tooth or place a crown or veneer on an existing tooth, they design or order a porcelain implant. These decisions require the dentist to compare the color of teeth with the color of the composite or porcelain material. Dentists try to select the color or shade of the material that provides the best color match to the surrounding teeth, but they also complain that this is a difficult task.
By now you have learned how to use the CIELAB color difference metric to predict whether two colors will appear to match under a fixed illumination. You have also learned that these predictions are not invariant with changes in illumination. In other words, if you change the lighting, the colors of two different materials may no longer appear to match. Therefore, the smile that looks so perfect in the dentist’s office under fluorescent lighting might have imperfections under daylight.
This project has three components. First, we will make spectrophotometric measurements of 1) the reflectance of teeth in-situ in different individuals, 2) the reflectance of different composite and porcelain material, and 3) the spectral power of the light that falls on teeth in-situ under different lighting conditions. Second, we will use this data and the CIELAB color difference metric to predict whether people will be able to detect the difference between teeth and composite and porcelain material under different lighting conditions. Third, we will use the data in ISET simulations in order to determine the tradeoffs in color matching accuracy, cost and convenience. More specifically, we will simulate an imaging system based on a cell phone camera with flash/no-flash mode that has the potential of providing dentists with an alternative to the more expensive spectrophotometric devices that are currently on the market.
Mentor: Joyce Farrell (joyce_farrell@stanford.edu) and Henryk Blasinski (hblasins@stanford.edu)
Simulation projects dusing ISETBIO
ISETBIO is an ISET based Matlab toolbox that can simulate human optics and photoreceptor sampling. With ISETBIO, we can accurately compute the optical irradiance image that impinges on the retina and the number of photons absorbed by human photoreceptors (cones) for a given scene. ISETBIO is capable of simulating human individuals with different optics (myopia, astigmatism, etc.) and cone mosaics (colorblind, density difference, etc.).
Reproduce and Compare with Recent Papers In this project, you are expected to reproduce the results from one recent paper with ISETBIO. You are expected to work with your mentor to rewrite it in ISETBIO and try to explain every difference (if any) from the original code.
Here is a set of papers by Watson that are computational, in Mathematica, and related to Optics and Retina
Modulation Transfer Function and pupil size Pupil size and light level
Here a paper related to the human point spread function
Computing human optical point spread functions
Retinal ganglion cell modeling
A formula for human retinal ganglion cell receptive field density as a function of visual field location
Or ganglion cells and behavior
Retina-V1 model of detectability across the visual field. The original code for the paper will be provided.
Skills preferred: Matlab programming
Mentor: Haomiao Jiang, Brian Wandell
Simulate an eccentric camera
Write a simulation of the Foveon sensor.
Or,
Write a simulation of the Light.co camera.
Mentor: Brian Wandell
Monitoring the environment
We have ideas about how to take calibrated underwater images captured by GoPro cameras to monitor the health of coral reefs. There are various components to the project (camera calibration, modeling of light transport through water, and automating image upload, storage and analysis).
Mentor: Henryk Blasinski
An underwater, multispectral light source
Underwater imaging is quickly gaining importance not only due to its applicability in marine ecosystem monitoring, but also proliferation of inexpensive action cameras such as GoPro. Unfortunately, the colors in images acquired under water are severely distorted due to scatter and absorption phenomena. One approach to recover more spectral detail is to use active illumination techniques, this approach has proved to be very useful on the surface. In this project you will design and build an underwater, LED based multispectral light source that fits a standard GoPro size, underwater housing. With all the hardware in place you will have a chance to evaluate the accuracy and performance of active illumination spectral recovery in underwater scenarios. This is a hardware oriented project, you will be expected build and integrate the final system, which means that you should be familiar with soldering, PCB design and possibly even some CAD tools.
Skills Preferred: Hardware design experience, OR good with web-site programming.
Mentor: Henryk Blasinski
Oculus
Geometric Camera Calibration In order to simulate degradations of the human visual system using images captured by a camera, it is necessary to know exactly how those images have been captured. This project uses simple camera models that use efficient and flexible calibration procedures to derive geometric parameters such as focal length, radial distortion and the position and rotation of two cameras. There are well-established techniques that estimate these parameters using a specific calibration target like a checkerboard. The goal of this project is to become familiar with those techniques and use them on real images (OpenCV provides many building blocks which can be used) with an image undistortion procedure and a stereo image rectification procedure.
Skills preferred: Knowledge of C++
Mentor:
Streaming and Augmenting Stereo Camera Images One of the long term goals is the simulation of certain degradations of the human visual system and the evaluation of computer-aided visual enhancements to counteract those degradations. A crucial ingredient in achieving this is a software pipeline which can stream images from a stereo camera to an augmented or virtual reality device in real-time. Hence, the goal of this project is to build such a pipeline to capture, stream, and feed images from cameras in real-time to an Oculus Rift device.
Skills preferred: Knowledge of C++, Willing to learn Oculus Rift SDK, optionally also OpenGL SL or CUDA
Mentor:
Image Display Use the Oculus Rift to display images to human subjects that simulate (recreate) visual sensations that a person with a particular visual condition would see. This could be low vision, a type of color blindness, loss of central vision due to macular degeneration, or the effect of a retinal prosthesis in a blind person. We will help you use isetBio to create images to simulate one of these conditions. You will render the images on a calibrated Oculus Rift.
Information Display The goal of this project is to capture and display information so that people can track their movements and navigate in an environment with only visual input from the Oculus Rift. This will be accomplished by interfacing a Project Tango device with an Oculus Rift display. The Project Tango has sensors and software designed to track the 3D motion of the device and create a map of the environment using simultaneous localization and mapping (SLAM) algorithms. The output of the Project Tango is usually rendered on a laptop display. In this project, you will render the output on an Oculus Rift.
3D Projects
Almost anything with Real Sense
Depth Sensing With an Endoscope Using Flashes
Depth sensing has been a recent industry trend for many imaging applications. One less explored route is the use of depth sensing for endoscopes, to help identify tumors or other problems. For this project, initially use simulated Scene3D endoscope images to prototype a depth sensing algorithm involving 2 flashes and 2 captures (other capture procedures could be used as well). Prototyping using simulation is a nice, structured way to try out new algorithms quickly. Next, apply this algorithm using a real endoscope and tackle the real-world challenges involved.
Mentor: Steve Lansel
Curved Sensor Simulation
Sony and other imaging companies have recently unveiled curved sensors to improve image quality. Curved sensors bring imaging improvements because of the physics of geometric optics. For simple lenses, usually the focal area is in the shape of the surface of a sphere. However, most sensors are planar, so are only able to capture a small portion of the focal area. Lens engineers usually try to account for this problem using many lens elements and aspheric lenses. However, a curved sensor could potentially be a far simpler, and less expensive solution to obtain high quality images, in a smaller form factor. Instead of using a complex lens to obtain high resolution, imaging engineers could simply use a simple lens and a curved sensor to obtain the same, or even better results.
This projects involves using Scene3D, a full pipeline camera simulator to compare the resolution and chromatic aberration benefits of curved sensors and a simple lens, versus a planar sensor and a complex lens.
Mentor: Brian Wandell
Integration with OpenCV
Do we want to create scenes of some sort (stereo? different illumination? different noise? Different optics?) and test openCV algorithms for robustness against the range of simulated images.
Integration with Caffe
Simulation environments can be used to produce millions of images with a purpose in mind. We can then use these images to train machine learning algorithms. Is there something we want to ask people to do with, say, RenderToolbox to generate many examples and train on with Caffe?
Multispectral imaging for classification
Image classification is a very hot topic in computer vision. Most algorithms however operate on RGB camera channels, as if trying to mimic human visual system. In reality spectral information is much more abundant and can possibly be used to enhance classification algorithms. This project aims at investigating how much the accuracy of computer vision tasks can be improved if more spectrally sophisticated cameras were used . Specifically you will use a five band camera prototype to evaluate fruit and vegetable aging and perform flower classification, you will compare its performance to the performance of a classical RGB camera.
Skills Preferred: computer vision, machine learning
Gullstrand Eye and ray tracing of human optics
We are building a tool for modeling eyes, including the human, from ray tracing fundamentals. There is a famous model eye that we would like to implement.
Gullstrand eye search
We would like you to implement and test the Gullstrand eye with the ray tracing software in the ciset package (a close relative of ISET).
Mentor: Brian Wandell
Projects 2014
Predicting human performance using ISETBIO
ISETBIO is an ISET based Matlab toolbox that can simulate human optics and photoreceptor sampling. With ISETBIO, we can compute the optical irradiance image that impinges on the retina and the number of photons absorbed by human photoreceptors (cones) for a given scene.
For this project, a tutorial script describing how to calculate cone absorptions will be provided and the students will be responsible for trying to answer one of following questions:
- What's the maximum necessary display resolution (ppi) at certain viewing distance for Vernier acuity.
- What's the maximum necessary display resolution (ppi) at certain viewing distance for contrast (CSF) resolution.
To answer these kind of questions, students are encouraged to build two scenes and use their preferred machine learning algorithm (e.g. SVM/Neural Network/Random Forest, etc.) to classify cone absorption sensor data for two same or two different scenes into "same" or "different" classes. When classification accuracy for cone absorption data is greater than a pre-determined value (say, 75%), we would predict that the observer can tell the difference between the two scenes. You can compare these predictions with published data from real human observers.
Preferred Knowledge: familiarity with at least one machine learning algorithm
Mentor: Haomiao Jiang
Hardware project: Build a Multispectral Imaging System
Build a multispectral imaging system based on a rotating color filter wheel and monochrome camera.
If you have experience in design and 3D printing, you can build several necessary parts.
If you have an interest in engineering applications for art history, there is an opportunity to use the system to capture images of paintings in the Cantor Arts museum.
Mentors: Henryk Blasinski and Joyce Farrell
Hardware project: Build an inexpensive spectrophotometer
In this project, you will build a simple spectrophotometer using a clean DVD-R, a USB webcam and stiff black card paper
Here's a website introducing how to do it: http://publiclab.org/wiki/spectrometer
After building the device, you need to compare it to the performance of a much more expensive (~$50K) spectrophotometer that we have in the lab
Mentor: Haomiao Jiang
Camera Image Quality Metrics
The International Standards Organization (ISO) is developing a set of camera image quality metrics to quantify the spatial resolution, noise and color accuracy of digital cameras. http://proceedings.spiedigitallibrary.org/data/Conferences/SPIEP/64097/829302_1.pdf
Many of these metrics have been implemented in ISET.
You can use ISET to calculate these metrics for simulated cameras that have different optical properties, numbers of pixels and image processing methods. You can also use ISET to simulate how each camera captures and processes natural scenes (e.g. faces and landscapes). You can then compare the metrics with the appearance of these images as they are rendered on a display.
In this project, you will use ISET and CPIQ to quantify and illustrate how the metrics and the images change when you decrease the size of camera pixels (and inversely increase the number of camera pixels). This method will allow you to analyze how resolution tradeoffs with sensitivity: Small pixels make it possible to increase the number of sensor pixels sampling the optical irradiance image, but it also decreases the amount of photons a small pixel can capture. What do you prefer, a high resolution noisy image or a low resolution clear image? How does this depend on the display, viewing distance, etc.?
Mentor: Joyce Farrell
ISET model for real camera
In this project, you will build an accurate ISET model for a physical camera we have. You will take pictures of known scenes, analyze the captured images, and try to build an ISET model.
The goal is for the ISET model of the camera to give approximately the same computational results as the RAW output from the real camera. The similarities could be measured by the noise, color, spatial resolution and etc. Analyzing the errors between the model and the real camera will determine the model's accuracy.
If time permits, you can try to implement an image processing pipeline for the camera and evaluate the performance of the processed images.
Mentors: Qiyuan Tian, Steve Lansel, Joyce Farrell
Analysis and Compression of L3 Filters
The L3 algorithm is a learned image processing pipeline for cameras. The algorithm learns optimal linear filters for a given camera based on training data, light level, illumination color, and optics. For a complete camera this may result in many (possibly hundreds) optimized filters. We believe the filters will be closely related for similar camera settings. The goal is to analyze the filters, store a compressed set of filters, and interpolate the needed filters from this compressed set. This way we only need to store a smaller set of filters and can extrapolate to lots of new camera settings. Here is a recent SPIE paper on L3: https://drive.google.com/file/d/0B0Gw85qGqJxhbXJlcmZjbmhOQ2s/edit?usp=sharing.
Mentors: Qiyuan Tian, Steve Lansel, Brian Wandell
ISET model for underwater imaging
With the proliferation of cameras such as GoPro more and more people have started taking underwater images. These usually have large amounts of distortion, both spectral and spatial, originating from the medium in which the image was taken. Rather than experiment in the real world, the impact of different light transport phenomena on RGB images can be understood via simulation environments. In this project you will implement, enhance and integrate with ISET the underwater image simulation system described in the paper below.
Color image simulation for underwater optics
Mentors: Joyce Farrell and Henryk Blasinski
App for Programmable Camera in iOS / Android
We have a prototype programmable camera to be used with iOS or Android devices. The project's goal is to make an app that will run on iOS or Android and uses the camera. Think of an interesting camera app, and we can work together to build it. Prior experience in iOS or Android is needed.
Mentors: Steve Lansel and Munenori Fukunishi
Image classification with a five band camera
Recently image classification and object recognition have become very popular topics. Large majority, if not all, algorithms use images acquired with traditional, three channel (RGB) cameras. The goal of the project is to evaluate the performance of the state of the art algorithms applied to images captured with a five band camera. Will the recognition/classification performance change, and if so by how much? To get the flavor of the project you can look at the following paper:
Multispectral SIFT for scene category recognition
Mentors: Henryk Blasinski, Steve Lansel
Analysis of a real camera lens
Can we characterize how a lens blurs a point of light (point spread functions or psfs) by analyzing camera images of test targets that are displayed on a color monitor? This project has many possible variations.
- Illuminate red, green and blue pixels on a display and capture an image of the display with a camera placed on a tripod a far distance away. Vary the pattern of red, green and blue pixels (e.g. noise pattern).
- Estimate the psfs of a real camera with a real lens by analyzing camera images of displayed targets. Use a prosumer digital camera and vary 1/f# and observer how the estimated psfs change.
- Estimate the psfs for different field heights, wavelengths and depths.
- Use the estimated psfs to predict camera images of other displayed "natural" images, such as a face. Compare the predicted camera images to actual camera images.
Here are links to papers that describe a method for empirically estimating the psf of a camera lens. The links include code that you can download
- http://www.cs.ubc.ca/labs/imager/tr/2013/SimpleLensImaging/
- http://www.ipol.im/pub/art/2012/admm-nppsf/
People: Brian Wandell, Andy Lin, Joyce Farrell
Psf analysis and image deblurring using a simulated camera lens
The point spread function(psf) of a lens is an extremely important lens property. One possible application of knowing the psf is image deconvolution (deblurring). Deconvolution can drastically improve image sharpness. The following paper provides a good technique for estimating a psf and deconvolving an image with that psf: http://www.cs.ubc.ca/labs/imager/tr/2013/SimpleLensImaging/
Tasks
- Andy Lin will provide simulated camera images of several different types of spatial test targets. Your task will be to use the code from http://www.cs.ubc.ca/labs/imager/tr/2013/SimpleLensImaging/ to estimates psfs from the simulated camera images.
- To evaluate how well the psf estimation code works, compare the estimated psfs to the known psfs that Andy used to generate the simulated camera images.
- As another evaluation technique, use the estimated and known psfs to "deblur" a blurred image containing a secret message using the deconvolution code downloaded from the same site. The secret message will only be legible after proper deconvolution of the image. Andy will provide this blurred image.
Mentor: Andy Lin
Medical imaging: Super resolution microscopy
http://en.wikipedia.org/wiki/Super_resolution_microscopy
Super resolution microscopy refers to methods that build up a high resolution image of target by integrating many multiple images of the target illuminated such that only a small subset of the image points are captured in any one image. The camera image then samples a subset of the pixels in a high resolution image. The location of the pixels in many camera images are combined to construct a single full high resolution image of the target. By placing a point at the center of each sampled point, one can get very accurate spatial information about the location (phase) of illuminated points in the target. Because the center of a dot is smaller than the lens psf, some people assert that super-resolution methods beat the limit of lens diffraction. But you know better than that. Diffraction is a limit that no earthly being can beat. Nonetheless, by sampling with stochastic and sparse arrays of pixels, one can do a better job of locating the center of sampled points and hence build up a higher resolution image.
You can write an ISET simulation to test one of these super-resolution methods.
Alternatively, you can test methods for super-resolution imaging using real camera images. For example, take a camera image of a displayed image, (such as a face or a high resolution test chart) . Then take a capture a series of images of the display when only a subset of the pixels in the face (or chart) are illuminated. The illuminated pixels in each subset will be far away from each other such that the optical images of the pixels illuminated in each image do not overlap. You can further experiment by taking a blurry image of a face (say, by setting the caemra 1/f# to 12). Then, display subsets of pixels of the face that are widely separated. Find the location of the center of each illuminated pixel and combine the data to create a non-blurred camera image.
Mentor: Brian Wandell, Haomiao Jiang
Eulerian video processing (Bill Freeman thing)
Repeat one of the experiments from Bill Freeman. There published paper could be found from http://people.csail.mit.edu/mrub/vidmag/
Also, you need to compare the results for cameras with 3 color channels (rgb) and with 5 color channels (prototype in our lab).
Scene 3D System
The goal of the Scene3D project is to simulate the complete imaging pipeline for 3D scenes, from the scene to the lens , to the sensor and to the mage processing. Simulations of sensor and image processing are implemented in ISET. The novel part of Scene3D involves using a technique in 3D graphics called ray-tracing, which produces a physically accurate simulation of light rays that are refracted through lenses and towards the sensor. We modified the PBRT ray-tracer to simulate the important effects of diffraction and to be able to handle complex lenses and multispectral inputs and outputs. The end goal of the Scene3D project is to provide an infrastructure for rapid image systems prototyping.
[Scene3D project https://github.com/ydnality/Scene3D]
One important aspect in photography involves color balancing. Often times, photographs taken under different illuminant conditions will produce images that don't appear natural. For example, images taken under indoor tungsten lighting will exhibit an unnatural yellow/orange tint. These images must be corrected for in order to appear natural.
This class project involves applying the camera pipeline simulation provided by the Scene3D infrastructure for use in designing a color-balancing algorithm.
Tasks
- Start with a 3D radiance scene generated by Andy Lin. Modify the parameters of the scene to make different renderings of the scene under different light conditions.
- Design and implement an intelligent method for "correcting" (color-balancing) the illuminant.
- (Challenge/Optional) Design a color balancing method that is able to correct for scenes with 2 or more different illuminants.
Mentor: Andy Lin
PBRT and Zemax optics modeling
Scene3D use a combination of PBRT and ISET to simulate the complete imaging processing pipeline of a digital camera. The unique contribution of Scene3D is that it applies a technique in 3D graphics called ray-tracing, to produce a physically accurate simulation of light rays as they are refracted through lenses and towards the sensor. We modified the PBRT ray-tracer to simulate the important effects of diffraction and to be able to handle complex lenses and multispectral inputs and outputs. However, we have yet to verify this pipeline empirically.
One way we plan to evaluate our modifications to PBRT is to compare the point spread functions we generate with point spread functions generated by Zemax, a well-established software package used by many optics professionals. We provide a Zemax macro that can be used to generate the PSFs that ISET needs. Although Zemax can produce physically accurate PSF's, it cannot produce rendered physically accurate 3D multispectral images like PBRT.
Tasks
- This project would involve taking several PBRT multi-element lens models, and creating equivalent Zemax models.
- Use the Zemax to ISET interface to generate the data necessary for the ISET simulations.
- PSF's using the PBRT model will be provided as ISET optical images. We provide a Zemax macro that can be used to generate PSF for lenses that are modeled in Zemax.
- Compare and analyze the PSF's produced by these two different methods under different aperture and distances as verification.
Experience with Zemax is preferred.
Mentor: Andy Lin
Gesturing in a Virtual 3D space
The Holografika multi-projector display system creates a 3D light field that people can view without the need for special googles. Leap Motion is a controller that can sense small finger movements using an infrared led and camera. We linked these two devices so that users can grasp and move virtual objects in the 3D light space created by the Holografika display. We also linked the Leap Motion to a conventional stereoscopic display that uses an LCD with shutter goggles. The goal of this project is to compare how well users can use the hand-gesture controller to move objects in the virtual 3D spaces created by the two different types of displays.
The project has possible variations. - You can find a suitable OpenGL app or game from the Leap Motion Airspace app store that measures agility to quantify the learning rates of new users. The objects floating in front of the Holografika display will be aligned to the users hands in that 3D space, but not so with the flat LCD display. Possibly include mouse mode in the tests. -Using a 3D top-down street view map of London, test users skills at finding a location by panning and zooming a holographic 3D map of London on both kinds of displays, using hand gestures. Does the user's self-reported confidence correlate to measured performance and how does display type affect that? Use the metrics to predict the actual benefit for different kinds of organizations to transition from mouse control to (hands in air) gesture devices with 2D and Holographic displays.
The equipment is calibrated and available in Packard 070.
Here is a link to the companies involved: www.holografika.com and www.leapmotion.com
You can watch a video of the talk by the inventor of Holografika (Tibor Balogh) at https://talks.stanford.edu/scien/scien-colloquium-series/
Mentors: Dave Singhal, Harlyn Baker, Peter Kovacs
Projects 2013
Camera Forensics
You are presented with a digital image and asked to determine if it has been manipulated and if so to localize the manipulation in the image. Color filter array (CFA) interpolation generates a tell-tale signature in a digital image that can be used in a forensic setting. CFA interpolation leads to strong correlations between a specific subset of pixels and their spatial and chromatic neighbors. Build a classifier that takes as input a digital image and automatically detects which parts of an image do and do not exhibit the expected CFA correlations. Begin by generating a synthetic set of test images that have undergone your choice of CFA interpolation. Test your forensic analysis on these uncompressed images and then quantify the efficacy of your approach on increasingly more JPEG compressed images. Disputes often erupt over the provenance of photos. Consider how you might use your new forensic technique to distinguish between images taken from different types of cameras (e.g., a Canon PowerShot vs. a Nikon D-series).
References
- A Survey of Image Forgery Detection
- Exposing Digital Forgeries in Color Filter Array Interpolated Images
Tasks
- We provide you with training images
- You develop the classifier based on the papers
- We provide you with test images to see how you did
Image Forensics
You are presented with a JPEG image and asked to determine if it originated directly from a camera/mobile device, or if it was re-saved one or more times. Multiple compressions at different compression levels leave behind specific statistical artifacts in the distribution of DCT coefficients. These artifacts can be used to distinguish between singly and multiply compressed images. Build a classifier that can distinguish between singly and doubly compressed images (assume that the second compression level is different than the first). Validate your classifier on a large data set of images. Quantify the conditions under which the classifier is effective and not. Extend your classifier to distinguish between one, two, and three compressions. The expert forger becomes aware of your forensic technique and writes a special purpose encoder that will re-save a JPEG image with the same compression quality as the original. Consider how you might counter this by detecting multiple compressions made with the same compression setting.
References
Tasks
- We provide you with training images
- You develop the classifier based on the papers
- We provide you with test images to see how you did
Turbulence removal
X. Zhu and P. Milanfar, "Removing Atmospheric Turbulence via Space-Invariant Deconvolution" IEEE Trans. on Pattern Analysis and Machine Intelligence vol. 35, no. 1, pp. 157-170, Jan. 2013
Also see related talk and Project page
Options
- You obtain by measurement or simulation example images and then use their methods.
- You develop a variant of their method, exploring deconvolution, registration, or some other part of the algorithm more deeply than in the original paper.
- You find another approach and compare that approach to this one.
Photon calculator utility (ISET)
Build a program, perhaps based on the ISET library, that calculates the spectral irradiance at the sensor from the scene radiance and a specification of the optics. Doing this for diffraction-limited optics, specifying only the f/#, is sufficient.
The utility should be backed by a wiki page that illustrates all of the steps in doing that calculation. This project should produce an educational and useful calculator.
- Doing an implementation that can run on a browser on the Internet is best.
- Doing a straight Matlab implementation with a nice GUI is also good.
- Implementing the ISET (Matlab) routines as a Python calculator has value, as well.
Updating Wikipedia
Help us make Wikipedia better. There are surprisingly many Wikipedia entries on imaging and human vision that are just a few sentences long. Look-up for example: 'Troland', 'Stiles-Crawford effect', 'Photopic vision', 'Human PSF' or 'Active Pixel Sensor' to see how poor these entries are. Your mission, should you choose to accept it, is to improve these (or other) entries. Think of your work as of a paper, which is published online, rather than in a .pdf format. Of course, just as with writing any research paper, your work should start with a thorough literature review, select the relevant pieces of information and write them up in a way approachable to a non-expert in the field.
Neuroimaging (special approval)
With the opening of Stanford's Center for Cognitive and Neurobiological Imaging (CNI), we now have access to a large number of MR scans of the human brain. We are also closely connected to the MR hardware and image processing algorithms.
While this course is not specifically about neuroimaging, some of the methods in the course might be usefully applied to the data collected at the CNI. For students already working in MR and interested in such signal processing, we might be able to develop some projects that build on your interest.
Two possible projects are algorithms to:
- Identify when two MR images are of the same brain (brainprint), even if they were acquired using different contrasts.
- Evaluate image quality and MR artifacts
Scene database for computer vision testing (special approval)
Build scenes, say using Blender and PBRT, that we can run through the ISET simulation to produce images. Then analyze these calibrated scenes using computer vision algorithms to derive the depth, illumination, and shading. See this example page for folks who created a database from real, rather than simulated, scenes.
Color balancing (special approval)
Color balancing refers to the process of converting camera rgb data into display rgb values. If one simply copies the sensor pixel values into the display values, the resulting image will not generally be a good color representation of the original scene. An important step in the image processing pipeline is to transform camera rgb values to display values such that the display image appears to match the original scene that was captured.
A simple and common approach to color balancing is to make an educated guess about the scene illumination based on an analysis of the camera rgb values. The estimated illuminant is used to select a color transform (typically a 3x3 transform or a look-up table) that maps camera rgb values into human sensor (xyz) values for an ideal illuminant, such as daylight. The goal of this transform is to render the scene that the camera captured as if the scene were illuminated by daylight.
Most camera processing pipelines use a standard illuminant called D65 as the ideal rendering illuminant. As far as we know, no one has tested the assumption that people prefer to view objects illuminated by D65. The preferred rendering illuminant may also depend on the objects that are being rendered..
The project will use hyperspectral data of faces, fruit and vegetables and outdoor scenes, and spectral power distributions of different illuminants to generate images that people will view on calibrated displays. People will be asked to indicate which color renderings they prefer. In this way, we will collect preference data about preferred rendering illuminants. The preference data will provide a useful guide for engineers who are designing color balancing methods.
Hyperspectral video (special approval)
Help us build and evaluate a hyperspectral video system based on led lights synced with a monochrome video camera. Capture hyperspectral video images of human faces and estimate pulse rate by the change in color sensor values over time (see http://people.csail.mit.edu/mrub/papers/vidmag.pdf)
Biology of the mouse eye image formation (special approval)
There is a huge amount of biology done in mouse. There is a movement to study the mouse retina in particular. To study the retina, we would like to be able to understand how the cornea lens in the mouse blur the retinal image.
Adaptive optics to the rescue: Williams and his colleagues analyzed the optical quality of the mouse eye. Specifically, they measured the wavefront aberrations from 20 wild type mice. They provide the data in their article.
Optical properties of the mouse eye
Brainard, Hofer and I have written a wavefront toolbox in Matlab that enables us to specify the wavefront aberration and calculate retinal images in ISET. This project is to use our software to reproduce Figures 10 and 11 from the paper.
You can do this! If you do, many people will cite your project because there are many people who work on mouse.
Active LED-based illumination (special approval)
These days LEDs can produce high intensity light with well defined spectral properties. We are interested in a hardware system that allows to control both the on/off times of a set of LEDs, as well as their intensity using a simple Arduino microcontroller. One way you can do this, and we have a working prototype (refer to this project), is to use pulse width modulated signals to control the duty cycle of an LED. If you operate at high enough frequency, then you will perceive the rapidly flickering LEDs as having lower or higher intensity. In this project, however, we are interested in controlling the LED intensity more directly, so that even at the micro-time scale you control the LED intensity directly, rather than switch it on/off.
Projects 2012
Image processing
Hyperspectral Imaging
Analysis of hyperspectral images of paintings by famous artists
Consumer digital cameras capture electromagnetic energy in three different spectral bands. Multispectral and hyperspectral cameras capture electromagnetic energy in many more spectral bands. We used two different hyperspectral cameras to capture images of several paintings in the Cantor Arts Museum. One of the cameras captures images in 160 different spectral bands ranging between 400 and 1000 nm (visible and near-infrared or VNIR). The other camera captures images in 256 different spectral bands ranging between 1000 and 2500 nm (short-wave infrared or SWIR). There is a very large literature on hyperspectral imaging of paintings that we will use to guide our analysis of the data we have already collected. (http://www.springerlink.com/content/80342384844k0r21/fulltext.pdf) In particular, we should be able determine if there is a drawing beneath the painting (an “underpainting) and to characterize the paint pigment. This analysis will allow us to determine the history of the painting and assess its originality. We hope that this project will serve as the groundwork for an exhibit at the Cantor Arts Museum. (JEF and TS) Here is a nice website that describes methods used in art forensics (http://www.webexhibits.org/pigments/intro/look.html)
Analysis of hyperspectral images of live organs during surgery
Several research labs are investigating the advantages of hyperspectral imaging in robotic surgery. This is because hyperspectral cameras can capture a wider range of spectral data, including electromagnetic energy that the human eye cannot see. One of the challenges is how to map information that is normally invisible to surgeons onto visible images that enhance the ability to discriminate between different tissue types in a meaningful way. We have collected hyperspectral images of organs in a live pig during surgery. This project will analyze this data to determine if information in the invisible regions of the electromagnetic spectrum (> 700nm) can be used to enhance the information that surgeons see during an operation. (see http://www.intechopen.com/source/pdfs/9221/InTech-Hyperspectral_imaging_a_new_modality_in_surgery.pdf ) (JEF and TS)
Colorimetric reproduction of human faces
We collected VNIR (160 narrowband spectral images ranging between 400-100 nm) hyperspectral images of human faces, outdoor scenes, still life (fruit) and paintings. The hyperspectral image data can be used to generate a representation of spectral reflectance of the objects in a scene and the spectral power of the scene illumination. These representations can, in turn, be used as input to the ISET digital camera simulation software. ISET can then be used to predict the output of digital cameras with different color channels. For example, one can simulate a digital camera with three or more color channels, and vary the spectral sensitivities of each of the color channels. One can also vary the spatial distribution of those channels. Finally, one can vary both the demosaicking and color balancing algorithms in the digital camera. This project provides an excellent tutorial on how a digital camera works and gives you the opportunity to develop your own color imaging processing algorithms. (JEF and TS)
References and Web Links
Novel detectors for RGB and NIR
Using NIR to enhance visible data
Several Susstrunk lab papers. Some others.
http://infoscience.epfl.ch/record/148419/files/81_susstrunk_v5.pdf http://infoscience.epfl.ch/record/153994/files/de24567-susstrunk.pdf
http://www.comp.nus.edu.sg/~dfanbo/papers/VisualEnhanceHSI_Kim_PR2011July.pdf
http://gitl.sysu.edu.cn/papers/cvpr-2008-zhang.pdf
http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5652900&tag=1
NIR Flash
http://www.comp.nus.edu.sg/~zhuoshao/NIRFlash/nirflash_icip2010_high.pdf
Photo retouching metric (Kee and Farid)
A perceptual metric for photo retouching Eric Kee and Hany Farid
Department of Computer Science, Dartmouth College, Hanover, NH 03755 October 19, 2011 (received for review July 5, 2011)
In recent years, advertisers and magazine editors have been widely criticized for taking digital photo retouching to an extreme. Impossibly thin, tall, and wrinkle- and blemish-free models are routinely splashed onto billboards, advertisements, and magazine covers. The ubiquity of these unrealistic and highly idealized images has been linked to eating disorders and body image dissatisfaction in men, women, and children. In response, several countries have considered legislating the labeling of retouched photos. We describe a quantitative and perceptually meaningful metric of photo retouching. Photographs are rated on the degree to which they have been digitally altered by explicitly modeling and estimating geometric and photometric changes. This metric correlates well with perceptual judgments of photo retouching and can be used to objectively judge by how much a retouched photo has strayed from reality.
- Watch a video of the SCIEN talk by Farid at Stanford, Jan. 31.
- Implement and analyze the algorithm.
- Perform experiments with existing online pictures
- Suggest improvements
Visibility of movie subtitles
A persistent problem in watching foreign movies is that sometimes the subtitles are illegible. Why? Because the contrast of the default background that is assumed is wrong and you have white characters on a light background. I assume this is done automatically because it is too expensive to have people judge frame by frame whether the script is visible. Need I say more. Some automated system that could assess the brightness of the standard background space where subtitles are printed and then adjust the contrast to be legible would be a huge improvement for the industry.
E. Markman, a committed viewer of subtitled films.
Image Quality
3D Image Quality Metrics
Develop algorithms for Shooting in 3D and Displaying in 2D. Explore ways in which to improve 2D rendering of 3D content in order to enhance “immersive video”.
Optics
Wavefront Toolbox
(BW)
Advances in adaptive optics now make it possible to measure the wavefront aberrations of the living human eye. Many groups are making these measurements in both control subjects and subjects with different types of optical dysfunctions.
These aberrations are usually specified in a way that is difficult to apply to image processing: The aberrations are specified as the weights on a set of Zernike polynomials. It is a simple matter of programming to convert these polynomial weights to a point spread function that can be applied in image processing algorithms.
We have received software from experts on this topic that implements the conversion. We can probably obtain a large number of samples of measurements from different categories of human eyes. In this project, we would create a web-site to convert the Zernike polynomials to point spread functions and illustrate how those pointspread functions would influence the quality of the optical image falling on the retina.
As we accumulate additional summaries of the human measurements, we might look for statistical patterns that might be explained in terms of the biological properties of the human cornea and lens.
See:
Chromatic and wavefront aberrations: L-, M- and S-cone stimulation with typical
and extreme retinal image quality
Florent Autrusseau, Larry Thibos, Steven K. Shevell
Vision Research 51 (2011) 2282–2294
Integrating 3D Distributed Ray Tracing and Image Quality
(BW), (AL), (JEF)
PBRT
Radiance
RenderToolbox
Implement and test Nayar Generalized Patent
Read the patent and implement tests of the idea.
Reference:
Neuroimaging
(AT), (AM), (RFD), (GS)
With the opening of Stanford's Center for Cognitive and Neurobiological Imaging (CNI), we now have access to a large number of MR scans of the human brain. We are also closely connected to the MR hardware and image processing algorithms.
While this course is not specifically about neuroimaging, some of the methods in the course might be usefully applied to the data collected at the CNI. For students already working in MR and interested in such signal processing, we might be able to develop some projects that build on your interest.
- Intelligent compression algorithm for multi-channel image data stored in frequency space (p-file compression)
- Algorithm to classify volumes that contain brains in a database of MR images that includes phantoms, squash, fruits, etc. (brain detector)
- Algorithm to identify when two MR images are of the same brain (brainprint), even if they were acquired using different contrasts.
- We can also do another one on MR artifact detection (so many artifacts, so few projects...)
Suggestions and projects from previous years
To see projectsfrom previous years, visit SCIEN Class Projects Page.