Pytlarz

Introduction

Over the past year, the use of video conferences has skyrocketed and with it has come the prevalence of virtual backgrounds. The video conference systems implement body/head detection, attempting to replace the background of the user [1]. This study will explore the believability of these virtual backgrounds, targeting specifically color temperature differences between the user and the background. A relationship will be shown between the color temperature of the original image, the color temperature of the background, and the detectability of the virtual background.

Background

The experiment will be conducted on a 2020 13" Macbook Pro. To ensure accurate processing and color transformation, measurements were taken of the target display and are show below. These measurements were taken in a dark lab using a PhotoResearch PR-740 spectroradiometer.

Red, green, blue and white patches were displayed full screen through the preview app. Since these images are not tagged as DCI-P3 or Rec2020, it is not surprising that we see the display attempting a Rec709/sRGB recreation. Notice the green stimulus produces power in what appears to be the red primary. However, this is the mode that the experiment will be conducted in, so we can use these results as our "native" assumed display.

The chromaticity diagram shows us the color gamut. The primaries are quite close to Rec709/sRGB shown in white. The white point however is far away from the typical D65 (0.3127, 0.3290). We will assume that the images have been rendered properly on the display and will decode accordingly.

The gamma/power response is nearly spot-on a power of 2.1. The peak luminance is 256.5cd/m^2 and the minimum luminance is 0.28cd/m^2 (with the backlight full on).

Methods

The experiment was conducted using a 2-alternate forced choice experiment design where a reference (real background) image was always present. The observers were asked to select the image with the fake background. There were a total of 16 unique trials which were repeated twice for each observer. The stimuli were created by capturing a video of the subject in a scene, then leaving that scene to get identical framing on an iPhone11Pro with auto exposure and auto focus locked in during the video capture. Then the proper frames were extracted in Matlab. A mask of the subject was created Photoshop to be as realistic as possible. In addition, the frames were initially color graded to best match hue/saturation/contrast as the best match baseline to ensure the direction I was probing was clear of interference. In addition, the framing of the two shots was taken to best match lighting angles. An example of the renderings is given below. You may notice from the top row that the color tone of the two scenes match very well. The two renderings showcase the extremes of testing (4500 and 20000 Kelvin).

Once I had the renderings, I created a database of virtual background pairings while adjusting the color temperature of the background. It was assumed that the desired white point was D65 and the chromatic adaptation was derived from here. The Von Kries LMS space was used for chromatic adaptation.

Results

Appendix

You can write math equations as follows: $y=x+5$

References

[1] G. Finlayson, C. Fredembach and M. S. Drew, "Detecting Illumination in Images," 2007 IEEE 11th International Conference on Computer Vision, Rio de Janeiro, 2007, pp. 1-8, doi: 10.1109/ICCV.2007.4409089.

[2] Xiaomao Ding, Ana Radonjić, Nicolas P. Cottaris, Haomiao Jiang, Brian A. Wandell, David H. Brainard; Computational-observer analysis of illumination discrimination. Journal of Vision 2019;19(7):11. doi: https://doi.org/10.1167/19.7.11.

[3] CCT LUT: https://www.waveformlighting.com/tech/black-body-and-reconstituted-daylight-locus-coordinates-by-cct-csv-excel-format