WenkeZhangChien-YiChang

Methods

Overview

Figure 1: Cornell box lighting estimation simulation pipeline

Camera Modeling

Lense Shading

Lens shading, also known as lens vignetting, refers to the phenomenon where the light falls off towards the edges of an image when captured by digital camera systems. Lens shading occurs because in the imaging system with an optical lens, the off-axis pixels often receive a smaller bundle of incident rays compared to on-axis pixels, resulting in relatively lower illumination towards the edges. This effect is ubiquitous and not surprisingly can be observed in the data we collected for the Cornell box lighting estimation.

In this project, our goal is to accurately estimate the lighting condition for the Cornell box. In order to do this, we first need to deconfound the lens shading effect from the empirical data. One way to achieve this is to preprocess the captured images to remove the lens shading effect and then compared the preprocessed images with the simulated ones. Equivalently, we can apply the artificial lens shading effect to simulated images and then compare it with real images. Here we opted for the latter method.

We first assume that the lens shading effect can be represented as element-wise matrix multiplications

$X_{LS}=X\circ M$, 

where $M$ is the lens shading matrix, $X$ is the simulated image, and $X_{LS}$ is the simulated image with lens shading effect. We also assume that the lens shading matrix $M$ is of the same dimension as the simulated image $X$, and $M_{ij}\in [0,1]$. The problem is equivalent to finding the lens shading matrix $M$. Once we have found the lens shading matrix, we can apply the lens shading effect to any simulated image by simply multiplying it elementwisly against $M$.

In order to estimate the values of the lens shading matrix $M$, let us consider the case where $X=cJ$, where $J$ is the matrix of ones where every element is equal to one and $c$ is a constant. We have

$J_{LS}=cJ\circ M=cM$,

and equivalently

$M={\frac {J_{LS}}{c}}$.

Here the left-hand side is the lens shading matrix and the right-hand side is the normalized image with the lens shading effect.

Results

Lens Shading

TODO @chien-yi

Lighting Estimation

Real Scene

TODO @wenke add colorbar to grayscale colormap

3D Scene

TODO @wenke add model figures

Reference

[1] A system for generating complex physically accurate sensor images for automotive applications. Z. Liu, M. Shen, J. Zhang, S. Liu, H. Blasinski, T. Lian, and B. Wandell,Electron. Imag., vol. 2019, no. 15, pp. 53-1–53-6, Jan. 2019.

[2] A simulation tool for evaluating digital camera image quality (2004). J. E. Farrell, F. Xiao, P. Catrysse, B. Wandell . Proc. SPIE vol. 5294, p. 124-131, Image Quality and System Performance, Miyake and Rasmussen (Eds). January 2004

TODO @chien-yi

Appendix

Psych221 Cornell Box Lighting Estimation project page

Psych221 Cornell Box Lighting Estimation source code