AlexOliviaAudrey
Calibration of Headlight Brightness in ISET Simulations
Authors: Lee, Audrey; Loh, Olivia; Sun, Alex
Introduction
In the world of autonomous driving, the simulation of car headlamps serves practical purposes. It involves the application of principles from optics to study how light interacts with the components of vehicle sensors and sight. This process is integral to ensuring that headlights meet safety regulations, provide optimal visibility, and contribute to an efficient driving experience. Typically, projects involved with autonomous driving or assisted driving often use simulation to perform testing before bringing it into the real-world. For example, lane keeping and pedestrian detection rely heavily on cameras and headlights which are first simulated to ensure feasibility and test capability before being implemented in real world situations. Therefore, ensuring that a simulation can accurately represent the real world and get accurate 3D spectral radiance scenes is important, and is also challenging.
One motivation for our project was that there was a recent tragedy where Google Maps led a father to drive off the end of a broken bridge. This tragedy, although not caused by autonomous driving, could possibly have been avoided if the father could see with brighter or more properly aligned headlights. Thus, we want to further improve headlights on cars through simulation. And the first step is calibrating the simulation to ensure that simulated headlights are more representative of real-world situations. And, maybe in the future, this step can help ensure that headlights will adhere to a regulatory compliance standard, are more energy efficient, and we could ensure that headlights are more standardized.
For our project, we wanted to calibrate the ISET simulation to have more accurate beam patterns for different headlight patterns. With our project, we used computer graphics (PBRT) and MATLAB to evaluate camera and headlight design options when used with industry standard benchmark scenarios.
Background
In this project, we utilize a couple radiometric and photometric equations we learned both in PSYCH 221 and from our mentors for this project to analyze the data we collected from the PR-670 spectroradiometer. These equations were to calculate irradiance from the radiance spectrum and luminance from the radiance spectrum.
The PR-670 spectroradiometer (shown below) was designed by JADAK to take spectral based photometric and colorimetric light measurements.
It was designed to capture precise light measurements from different possible light sources ranging from display monitors and projectors to reflective surfaces to industrial testing applications. It is configured with 256 detector elements and has a range of detection from 0.2 fL to 2,500,000 fL and a spectral resolution of 1.56 nm per pixel. We were able to use the PR-670 camera to take accurate measurements while using MATLAB to save the data and analyze it.
We also used a couple of the ISET repositories such as ISET3d and ISETauto to develop our calibration methodology. ISET3d interfaces with PBRTv4 to render realistic, ray-traced 3D scenes. In order to better understand night-time driving scenes, we must port our measurements from the real world into simulation. Our project focuses on this process for a specific case: the calibration of headlight brightness.
With the help of Dave Cardinal, we chose two headlamps to simulate: a halogen bulb headlight and a LED headlight. The LED headlight has a low and a high beam setting, however, we decided to calibrate the LED headlight to the low beam setting.
Methods
Data Collection
In order to fully characterize a light source for simulation, we would like to know the spectral radiance of the light, i.e. how much power per square meter per steradian is emitted, and which wavelengths compose the light.
To take such measurements, one uses a spectroradiometer, which records exactly the spectral radiance of a source over some small angular portion of its field of view. The PSYCH221 teaching team owns a PR-670 spectroradiometer and has a MATLAB script that runs the spectroradiometer, which we used to take our data.
The data from the PR-670 comes in the form of a .mat file for each measurement, containing a dictionary with two fields – “wavelength” and “data” – encoding the wavelengths at which the spectral radiance is measured, and the corresponding spectral radiance at each wavelength, respectively.
Caption: Spectral radiances of the halogen beam (left) and LED beam (right) on a white color card. Note these spectra make sense; A halogen bulb is a black body radiator, so we expect a broad black body curve across many wavelengths, peaking somewhere in the infrared, exactly as shown in the data. White LEDs are made from blue LEDs coated in phosphors that fluoresce yellow when radiated with blue light. Hence we expect a spectral peak in blue and yellow, as we see from the spectrum.
In deciding how to take measurements which would characterize the headlight, we quickly realized that it would be ineffective to point the spectroradiometer directly at a bare headlight, as the sensor would quickly be saturated. We considered using a neutral density filter in front of the lamp and a short integration time on the spectroradiometer. In the end, however, we realized that in ISET ray-tracing simulations, the most easily measurable metric of “brightness” was the luminance of different points in the scene.
With this in mind, we decided to take an indirect measurement of brightness by shining the headlamp at a surface of characterized reflectance and then measuring the reflected spectral radiance at various sample points in that scene, and then map those observed luminances to an ISET scene simulating the same scenario. This can be seen in the two figures below where the first figure contains the Halogen light’s beam pattern and the second figure contains the LED light’s beam pattern.
We collected 24 data points for each beam, assembled in a 3x8 matrix as shown above. We did this manually, labeling each file name by its corresponding matrix indices (as is shown, although hard to make out, in the photos). Each measurement was taken at the center of the sticky note, and we would manually adjust the spectroradiometer’s focus between measurements. Sticky notes were used only for alignment, not for measurements. Once the spectroradiometer was focused on the center of the sticky note, one of us would hold the grey or white card against the wall for the actual measurement as shown in the figure below:
With this data taken, we moved on to try different methods of synthesizing and analyzing this data. We first tried to calculate irradiance from the spectral radiance to see if our collected data matches our expectations using the equations in the below figure:
With M being the emittance, ρ being the reflectance, and R being the radiance, we can calculate E which is the irradiance. From this measurement, we were able to plot the irradiance maps for both the Halogen and the LED headlight beam patterns:
With these maps, we confirmed that the data we captured were representative of our sampled beam patterns, but we realized that this was still a radiometric quantity. So, we decided to calculate a photometric quantity instead: luminance.
As we know, spectral radiance is a radiometric quantity: one where spectra are measured as they physically exist in the real world. Luminance, on the other hand, is a photometric quantity: one that describes how humans perceive spectra. In converting from a radiometric quantity to a photometric one, we must always take into account the photopic luminosity function, which describes how bright the human eye perceives different wavelengths. This calculation for spectral radiance is summarized in the below figure:
Caption: When working with real data, it is crucial that the units match. Rendered ISET scenes yield luminance in candelas per square meter, whereas our data was spectral radiance in watts per square meter per steradian per nanometer. Thus to convert our data into a form comparable to ISET simulations, we performed the above transform from spectral radiance to luminance.
With this equation, we then transform every data point in the same way, yielding the following luminance maps for the halogen and LED beams (incident on grey cards and white cards)
Calibration
PBRT lights in ISET have several properties that we can modify to change the appearance of light
1) Type
2) Scale
3) FOV
4) Power
In our simulations, the light “type” was set to be a [projection light], and the “FOV” was set to 40 degrees. The attributes “scale” and “power” both change the intensity of the light, but “power” changes it nonlinearly. We thus chose to scale the intensity of our simulated headlamp using the “scale” parameter.
In addition to these basic parameters, we can also give the headlamp a property “obj.lightMask”, where the light mask is a 2D matrix that matches the size of the resolution of the headlamp, thus specifying a pixel-by-pixel attenuation. This is what allows us to port the specific beam profiles into the simulation.
With this in mind, we established a calibration method consisting of five steps, summarized in the image below:
1) We calculate a normalized version of our luminance profile (divide all values by the peak value). This gives us a dimensionless “attenuation map” that specifies the beam profile shape by specifying relative intensity without setting absolute intensity.
2) We then render the scene with this uncalibrated attenuation map and some default settings for the light, and measure the luminance from the scene.
3) We then calculate the ratio of luminance in real life to luminance in simulation, which tells us by how much we need to scale the power in the simulated light to match the absolute intensity we saw in real life.
4) We then multiply the “scale” parameter in ISET by this calculated ratio.
5) We then re-render the scene with this scaled intensity and ensure that it matches our real life data.
We experimented with two different ways of calculating the scale factor:
We found these two ways of calculating the scale factor to be quite similar, but we surmise that the “peak luminance” method is more accurate because the mean luminance from ISET includes points outside the main beam pattern.
Interpolation
We coarsely sampled the beam patterns using 3x8 sticky note grids with a total of 24 spectroradiometer samples. Our sampling does not fully capture the details of the beam patterns, particularly the halogen pattern with its distinct geometry. To upscale, we tried multiple interpolation functions in Matlab to test which interpolation method would accurately match the beam patterns we saw.
After interpolation, we increased our resolution from 3x8 to 150x400 samples. We then plot the interpolated samples onto a luminance 3d grid, where we begin to see the beam pattern anti-alias and form a better representation of the beam patterns we captured on camera. Below is a comparison of all interpolation methods, with halogen on the left and on LED on the right.
Nearest Neighbor
Bilinear
Cubic
Spline
Configuring and Experimenting with the ISETauto and ISET3d Pipelines
Our process with understanding and experimenting with the ISETauto and ISET3d repositories is outlined in the figure below:
We first had to gain access to the servers, in which we want to thank the teaching team and our mentors for helping us with the permissions and accessing the servers. Next was SSH-ing into the servers. The initial SSH worked, but there was difficulty with generating and copying the SSH keys, so we didn’t have to enter a password. To run the ISET code in MATLAB, it required access to the servers without needing to manually input a password. However, we ran into several issues with this and there were some inconsistencies as well where sometimes it would work one day, but not the next.
Thankfully, we were able to somewhat resolve these issues and were able to run the code in MATLAB. However, we ran into some issues with the code where some parts of the code did not work. We deduced that the script depended on a certain MATLAB toolkit, but as a fix for our purposes, we changed the code to a similar function. We propose that the ISET repositories update and list their dependencies in their documentation.
After running the code, we then needed to understand the code. This section was the biggest learning curve as we were unsure where to begin. We needed to understand what parameters affected the simulation and how our calculations could make an impact on the simulation so we can test and experiment with our data. After toiling with the different parameters and functions within the code, we were able to understand it more and we recognize that it is a solid pipeline where we can adapt the code and input our night time driving headlight simulation masks into. Moreover, these masks were organized in saveable presets, where we contribute a preset per beam pattern, 'cheap-halogen-beam' and 'cheap-led-beam'.
Results
Calibration Renders
Below are the results of peak calibration and mean calibration renders for both beam patterns. The mean calibration, visually, yields little to no difference compared to the peak calibration render.
Halogen
LED
Interpolation Renders
The nearest neighbor interpolation algorithm yields pixelated results and does not effectively anti-alias. It does not capture the beam pattern. The bilinear interpolation method captures both beam patterns very well, including details of the halogen beam pattern. Cubic interpolation completely smoothes out beam patterns, still fully capturing the gradient of the led beam pattern, but smoothing over the details of the halogen beam pattern. Spline interpolation yielded nearly identical results as cubic interpolation.
We chose bilinear interpolation for our final render since it captures the details of the halogen beam pattern very well. There is no significant difference in using the bilinear, cubic, and spline beam methods for capturing the LED beam pattern.
Halogen
LED
Final Render Comparison
In our final renders, we have selected the peak calibrated, and linearly-interpolated render. Below is a comparison of the beam pattern on a grey card we measure in lab with the final render in ISET simulation. Upon visual inspection, the render looks pretty accurate. However, ideally in future iterations of this project, we would want to determine quantitative metrics to validate the visual accuracy of the beam pattern we rendered.
Halogen
LED
Conclusions
Overall, we were able to calculate the brightness of two headlamps for ISET simulations, which we have added to ISETauto as headlamp presets. The code creating the headlamp presets as well as the scene we rendered to calibrate them can be found on our forked versions of the ISET3d and ISETauto repositories.
Although we have successfully calibrated the brightness of these headlamps, there is more that can be done to improve their realism in simulation:
- 1) Implementation of the lamp spectrum in ISET as a light preset
Our current brightness calibration is done with the projection light spectrum as D65, which is not accurate for halogen or LED beams. Creation of custom light profiles for the halogen beam and LED beam would be straightforward, given that we have spectral radiance data from both lights, and is a viable next step.
- 2) Higher resolution and more efficient mapping of the beam profile
If one desired to calibrate a variety of different headlamps, with wide angular spreads, then doing this manually as we did would be very time consuming. Taking even 24 data points for each headlamp took us over three hours due to the spectroradiometer’s data collection time and reset time as well as the need to recenter the focus on the center of a specific location with each measurement.
In reality, this method is probably quite redundant, given that the beam profile should be largely spectrally uniform. This means that we should be able to take the spectrum once, and then from thereafter just take in a measure of radiance or brightness. In the future, this should be straightforward with the Google Pixel 4a sensor that the Wandell Lab has calibrated and can use to take RAW photos. One could use exposure bracketing to get an accurate picture of the beam profile, and then scale intensity as we did with the luminance data. In this case, there would be no need to interpolate, as the beam profile would already be high resolution (resolution of the phone camera).
- 3) Adding more headlight presets
Indeed, if one hoped to do realistic night time driving simulations, one would want to characterize more headlights. These cheap headlights we characterized are much smaller and dimmer than those used on most modern cars today. The halogen headlight has no high beam / low beam switching, and although the LED light does, the low-beam setting light is not particularly well defined to a low area. That being said, the actual light source is not as important as the process by which one would calibrate it, which is what we have shown in our project.
References
Appendix
Appendix 1
We created new presets to test for the different setups. This script was used to test different parameters and masks.
We changed and tested different parameters to reflect the new presets such as distance from the wall and mask presets. We also fixed a bug in the code to accommodate MATLAB's libraries. This various testing and adaptations to the source code can be found at the forked version of the ISETauto repository.
Appendix 2
Alex Sun: Communicated with mentors asking technical questions, collaborated with groupmates to set up beam measurement experiment and collect spectroradiometer data, formulated solutions for MATLAB simulations, ran simulations to generate irradiance maps, calibration, and final renders, understanding the ISET simulation and writing MATLAB code for simulations, organized code on forked repositories, and contributed to documenting our work in the report page.
Olivia Loh: Scheduled meetings and communicated with project mentors, collaborated with groupmates to set up beam measurement experiments and collect spectroradiometer data, helped brainstorm ideas for MATLAB simulations, set up and understand the ISET pipeline and simulation, wrote mean calibration and interpolation scripts, ran simulations to generate calibration and interpolation renders, contributed to forked repositories, and contributed to report documentation.
Audrey Lee: Communicated with the mentors and instructors, worked in person and collaborated with groupmates to collect spectroradiometer data, helped and brainstormed solutions for the MATLAB simulation, contributed to understanding the ISET simulation and writing MATLAB code for the simulation, reviewed existing documentation for solution problem solving and understanding, assisted with evaluating results, and contributed to organizing and writing the final report paper.