Pytlarz
Introduction/Background
Over the past year, the use of video conferences has skyrocketed and with it has come the prevalence of virtual backgrounds. The video conference systems implement body/head detection, attempting to replace the background of the user [1]. This study will explore the believability of these virtual backgrounds, targeting specifically color temperature differences between the user and the background. A relationship will be shown between the color temperature of the original image, the color temperature of the background, and the detectability of the virtual background.
Methods
The experiment was conducted using a 2-alternate forced choice experiment design where a reference (real background) image was always present. The observers were asked to select the image with the fake background. There were a total of 16 unique trials which were repeated twice for each observer.
Display Characteristics
The experiment was conducted on a 2020 13" Macbook Pro. To ensure accurate processing and color transformation, measurements were taken of the target display and are shown below. These measurements were taken in a dark lab using a PhotoResearch PR-740 spectroradiometer.
Red, green, blue and white patches were displayed full screen through the Preview app. Since these images are not tagged as DCI-P3 or Rec2020, it is not surprising that we see the display attempting a Rec709/sRGB recreation. Notice the green stimulus produces power in what appears to be the red primary. Hence we are not getting the true native capabilities. However, this is the mode that the experiment will be conducted in, so we can use these results as our "native" assumed display.
The chromaticity diagram shows us the color gamut. The primaries are quite close to Rec709/sRGB shown in white. The white point however is far away from the typical D65 (0.3127, 0.3290). We will assume that the images have been rendered properly on the display and will decode accordingly based on the measured display primaries/white point.
The gamma/power response was measured in steps of 10 8-bit code words, including 0 and 255. It is nearly spot-on a power of 2.1 after scaling to the peak and minimum measured luminance values. The peak luminance is 256.5cd/m^2 and the minimum luminance is 0.28cd/m^2 (with the backlight full on). From these measurements, I was then able to derive a transform to/from display space to XYZ or LMS for color transformation.
Stimuli
The stimuli were created by capturing a video of the subject in a scene, then leaving that scene to get identical framing on an iPhone11Pro with auto exposure and auto focus locked in during the video capture. Then the proper frames were extracted in Matlab. A mask of the subject was created Photoshop to be as realistic as possible. In addition, the frames were initially color graded in Photoshop to best match the hue/saturation/contrast of each image. This helps to ensure the direction I was probing was clear of interference from other image attributes. In addition, the framing of the two shots was taken to best match lighting angles so that the lighting direction on the face was as close as possible. An example of the renderings is given below. You may notice from the top row that the color tone of the two scenes match very well. The two renderings showcase the extremes of testing (4500 and 20000 Kelvin). When blending with the mask, the blend was done in linear space to best emulate the physics of a true background.
Once I had the base renderings and the color transforms defined from the display measurements, I created a database of virtual background pairings while adjusting the color temperature of the background. It was assumed that the desired white point was D65 and the chromatic adaptation was derived from here. The Von Kries LMS space was used for chromatic adaptation.
Experiment Design
Results
Conclusions
Appendix
You can write math equations as follows:
References
[1] G. Finlayson, C. Fredembach and M. S. Drew, "Detecting Illumination in Images," 2007 IEEE 11th International Conference on Computer Vision, Rio de Janeiro, 2007, pp. 1-8, doi: 10.1109/ICCV.2007.4409089.
[2] Xiaomao Ding, Ana Radonjić, Nicolas P. Cottaris, Haomiao Jiang, Brian A. Wandell, David H. Brainard; Computational-observer analysis of illumination discrimination. Journal of Vision 2019;19(7):11. doi: https://doi.org/10.1167/19.7.11.
[3] CCT LUT: https://www.waveformlighting.com/tech/black-body-and-reconstituted-daylight-locus-coordinates-by-cct-csv-excel-format