Effects of inter-ocular gamma discrepancy on subjective viewer discomfort device while using a head-mounted display
Introduction
The sudden resurgence of virtual reality (VR) technology has brought about renewed interest in some of the unique challenges these technologies. These devices use head mounted displays, which are worn by the users who are ultimately expected to use them for extended periods. When viewing distance is so short, and with each eye getting an essentially independent image, it is of extreme importance to test for possible discomfort when the images sent to each eye mismatch.
As in some HMD devices the left and right screens are independently manufactured and calibrated, parameters that are likely to vary between screens of the same make and model are prime for testing. One such parameter is gamma, a non-linear correction function that maps intensity values from stored values to actual intensity values displayed on the monitor.
Background
Past research into disparity has traditionally used stereoscopic displays (utilizing polarized light) or cruder setups. The most recent study from Kooi and Toet (2004) has utilized polarized displays to obtain viewer rankings of subjective discomfort to inter-ocular disparities along multiple parameters, though not explicitly gamma. Under this setup, they found that almost any disparity could cause some noticeable discomfort, but the most sensitive factors were vertical disparity and blur. Luminance differences (not gamma) were only found to cause discomfort when over 25% difference in global luminance was found. Recent research into viewing discomfort when using HMDs (Carnegie and Rhee, 2015) studies discomfort in relation of depth of field, accommodation, and convergence. They did not focus on inter-ocular differences which might affect viewer discomfort. Finally, I would be remiss if I did not mention the wide and branching literature on binocular rivalry (Blake, 2001). Though not directly related to this research, this paradigm has been used extensively in both vision and psychological research as a tool for assessing attention, preferences, biases, and even consciousness.
Methods
Setup
A Windows 7 Professional SP1 64-bit host system was setup with Matlab 2012b. The then latest PsychToolbox (PTB) 3-beta November 2015 with recent support for Oculus VR was installed. The PTB-beta required a specific oculus runtime, 0.5.0.1-beta and specific configuration based on host OS and other dependencies (see PTB documentation) for operation. The head-mounted display used was an Oculus Rift SDK2.
Stimuli
The stimulus set chosen was a freely available set published by Moreno-Martinez and Montoro (2012). The stimulus set contains 22 ecologically relevant categories from Living and Non-living superordinate categories. Living: Animals, birds, body parts, flowers, fruits, insects, marine creatures, nuts, trees, and vegetables. Non-living: Buildings, clothing, office supplies, food, furniture, jewelry, kitchen utensils, musical instruments, sports, tools, vehicles, and weapons. All images were 720x540 pixels in size and in full color. The final set included 344 individual images used for this experiment. Sample stimulus is shown in Fig. 1.
Experimental design
The experiment consisted of a serial presentation of images, each requiring a participant’s response. Description of required responses and participant instructions are included in the next subsection. For each image presentation, gamma was manipulated for each of the eyes in the head-mounted display individually. Random values between 0.7-1.3 were chosen at random, and gamma adjustment was applied. The function used to apply gamma was Matlab’s imadjust function. This rather narrow range of values was chosen in order to restrict the experiment to a relatively plausible range of gamma differences between individual displays. Below in Fig. 2, sample manipulations are shown.
Procedure
Three participants were recruited for this experiment from the Stanford psychology department. Participants all had normal or corrected-to-normal vision. In the case of near-sightedness, an additional set of lenses provided by oculus was used, specifically fitted for near-sighted individuals.
Participants were fitted with the oculus set to their comfort, and once adjusted a calibration process from the oculus control panel was started. The calibration process measures viewing angles of the user and calculates based on that their inter-pupillary distance to correctly display images within the head mounted display set. A demo scene was then attempted until participants confirmed they were comfortable and that the oculus display appeared correctly.
Prior to the start of the experiment participants were run through a brief calibration to prepare them for making subsequent replies in the experiment. Calibration images were chosen based on prior attempts to test fusion (left-right visual field integration) and color perception. The first calibration image was an unaltered presentation of the same image to both eyes. They were told that they should be seeing the object correctly, and normal presentations should be respond to by button press ‘z’ – all OK, no discomfort. The second calibration image contained the same image displayed to both eyes, only one of the eyes received a color image and the other a grayscale image. Though this image features fusion (no participant reported trouble integrating left/right images), the colors are “off”. Participants were instructed to respond to pictures that are “off” as having some discomfort by button press ‘x’. Finally, another calibration image was presented showing the same image, where one eye received a regular color image, and the other an inverted color image. For a 100% of participants, this image failed to fuse from both visual fields. Participants were instructed to this type of stimuli presentation with button press ‘c’ – completely off, discomfort noticeable.
Once participants confirmed that they understood the instructions the experiment was started. Participants viewed all 344 stimuli images at their own pace, responding by button press to arrive at the next stimulus. Responses were recorded for every trial as well as reaction times. Images were displayed in a random fashion without repetition until all images were exhausted.
Participants were then tested for ocular dominance by using a standard method. Participants held both arms straight forward and aligned a distance object. They then closed each eye and noted any alignment shift. The commonly accepted interpretation is that when alignment shifts, the eye which was just closed is the dominant eye.
Pre-processing
Our primary variables of interest in this experiment was the gamma differences between the eyes and participants subjective assessment of the viewing experience. Thus, we coded gamma differences as the difference between the left and right eye, and participnts responses was coded as: All OK – 0, something off – 1, and completely off – 2.
Gamma difference analysis
We used both a linear regression model and a binomial logit regression model to test for the effect of gamma difference on response. In order to make the binomial fit, values were converted to a binary model where OK responses were kept intact and results where participants indicated anything was off (ones and twos) were re-coded as ones.
Categorical differences (i.e. image category and superordinate category) and eye dominance were included in the model to test for possible interaction effects on perception.
Results
Increasing disparity in gamma adjustment increases discomfort
Results of the linear regression show a significant positive effect of gamma binocular gamma differences on participants’ subjective perceptual discomfort, F(1, 1030) = 100, p < .001, Adjusted R2 = 0.087. No significant effects were found for either category, superordinate category, or eye dominance. Fig. 3 shows the regression line overlaid on the scatterplot of the data.
As can be seen, though a linear trend is certainly evident, there’s also positive (‘off’) responses at zero gamma difference, as well as negative responses at maximal gamma difference tested. Binning the results based on low and high gamma differences (respectively, below or equal to .3 and above .3) shows a far higher occurrence of positive responses at higher gamma differences, Fig 4A and 4B.
Since it’s possible there’s a critical point somewhere in the data, as suggested by the above distribution, a binomial logistic regression was run on the data as well. Not surprisingly, results again confirm gamma differences as a meaningful predictor of participants’ subjective viewing discomfort (Log Odds = 3.62, p < .001). Plotting the probability curve of the transformed coefficient shows the results to be linear for all intents and purposes, Fig 5.
Conclusions
Our use of digital display devices in the past 20 years has become a veritable tornado. We now stare at digital display devices for extended periods of time, far removed from the natural condition under which our visual system evolved. Head mounted displays and VR tech take a step further by moving the displays extremely close to our eyes, and generating both intentional and unintentional disparities between the eyes.
The effects of such a drastic change in our processing of visual information and the introduction of orthogonal manipulation of that information to each eye calls for a methodical examination of such disparities – how are they reconciled (if at all), what are the short and long term problems, what kind of discomfort are viewers likely to encounter and is there a way to avoid them, and so on.
As gamma is often miscalibrated across displays, we chose to focus on it first and foremost, and determine the practical limits over which differences in gamma are noticeably causing discomfort.
Our findings show that perceived subjective discomfort is linearly related to the difference in gamma between both eyes. We found this relationship to exist across all image categories tested and with no relation to a participants’ dominant eye.
Despite that, it’s undeniable that participants still rated images with zero difference in gamma as having some measure of subjective discomfort. As subjective assessment is a problematic metric exposed to wildly varying individual differences in disposition, as well as interpretation of instructions, such artifacts are likely unavoidable.
One significant problem with this design was the lack of proper image statistics before and after the gamma manipulations. As noted in the background material, global brightness changes might lead to discomfort, but the total change in brightness following our manipulations are still unknown. This is but one example of important image statistics that went by unnoticed. More robust measurement can, and should be implemented in future iterations of this experiment.
Nevertheless this experiment has provided much insight into subsequent investigations of this phenomena: (a) greatly clarifying instructions and introduce an extended calibration and training period. (b) couple (or subsume) the subjective assessment task with a pure change detection task such as a forced choice paradigm. (c) generate before and after image statistics to correctly assess the effects of gamma adjustments, and utilize in analysis (d) obtaining a significantly larger participant pool to provide a robust measurement of the phenomena and mitigate individual differences.
References
Blake, R. (2001). A primer on binocular rivalry, including current controversies. Brain and mind, 2(1), 5-38.
Carnegie, K., & Rhee, T. (2015). Reducing Visual Discomfort with HMDs Using Dynamic Depth of Field. Computer Graphics and Applications, IEEE, 35(5), 34-41.
Kooi, F. L., & Toet, A. (2004). Visual comfort of binocular and 3D displays. Displays, 25(2), 99-108.
Moreno-Martínez, F. J., & Montoro, P. R. (2012). An ecological alternative to Snodgrass & Vanderwart: 360 high quality colour images with norms for seven psycholinguistic variables. PloS one, 7(5), e37527.
Appendix
APPENDIX A: MATLAB CODE
This matlab file controls the presentation of all stimuli as well as data collection. A string variable needs to be passed containing participant initials, or a debug string will be used. Data will be collected into a csv file outputted to the working directory starting with the supplied string and a time stamp.
Requirements
• Matlab version 7.0 and above • PsychToolBox version 3-November 2015 beta (link: https://github.com/Psychtoolbox-3/Psychtoolbox-3/tree/PTB_Beta-2015-10-12_V3.0.12) • Oculus Runtime 0.5.01-beta setup as extended mode • Oculus “monitor” need to be set to portrait mode (and on some systems flipped as well)
Code
function binocularGammaAdjustment(sid)
%
%
% History:
% 30-Nov-2015 First version
if isempty(sid)
sid = 'dbg';
end
% Initialize
PsychDefaultSetup(2);
stereoscopic = 1;
% Select screen with highest id as Oculus output display:
screenid = max(Screen('Screens'));
% Open our fullscreen onscreen window with black background clear color:
PsychImaging('PrepareConfiguration');
% Setup for stereoscopic presentation:
PsychVRHMD('AutoSetupHMD', 'Stereoscopic', 'LowPersistence FastResponse');
% Start display
[win, rect] = PsychImaging('OpenWindow', screenid);
% Set up directories
baseDir = 'C:\toolbox-ptb3-beta11112015\Experiment';
stimDir = [baseDir '\Images'];
% Load test images
images = rdir([stimDir '*\**\*.jpg']);
% Generate a random presentation of the images on each run
imagePresentationOrder = randperm(size(images,1));
% Initialize output file
outFile = fopen([sid '_' datestr(now,'yyyymmddTHHMMSS') '.csv'], 'w');
fprintf(outFile, 'Image, Category, Superordinate, GammaL, GammaR, GammaDiff, Response, RT\n');
%Start calibration
for tt = 1:3
Screen('Flip', win);
% Load calibration images
imagePath = [baseDir '\calibration\cal_' num2str((tt-1)*2) '.jpg']
imageTextureL = imread(imagePath);
imagePath = [baseDir '\calibration\cal_' num2str((tt-1)*2 + 1) '.jpg']
imageTextureR = imread(imagePath);
% Render one view for each eye in stereoscopic mode:
for eye = 0:stereoscopic
% Prepare each display
Screen('SelectStereoDrawBuffer', win, eye);
Screen('FillRect', win, [0 0 1]);
% Grab image
switch eye
case 0
image = Screen('MakeTexture', win, imageTextureL);
Screen('DrawTexture', win, image);
Screen('Flip', win, [], 1);
case 1
image = Screen('MakeTexture', win, imageTextureR);
Screen('DrawTexture', win, image);
Screen('Flip', win, [], 1);
end
end
Screen('Flip', win, [], 1);
% Wait for response and get reaction time
time_start = GetSecs;
response = GetKbChar;
time_end = GetSecs;
response_time = floor((time_end - time_start)*1000);
% Write to file
fprintf(outFile, '%s, %s, %s, %f, %f, %f, %s, %f\n', 'Calibration', 'Calibration', 'Calibration', 0, 0, 0, response, response_time);
end
% Start presenting images
for tt = 1:size(images,1)
clear('imageTexture', 'imageTextureL', 'imageTextureR');
% Load image and parse
imagePath = images(imagePresentationOrder(tt)).name;
imageFileParts = regexp(imagePath,'\','split');
imageTexture = imread(imagePath);
% Randomize gamma values on each run
% gammaL = randi(19) * .1;
% gammaR = randi(19) * .1;
gammaL = 0.6 + randi(7) * .1;
gammaR = 0.6 + randi(7) * .1;
gammaDiff = gammaR - gammaL;
% Adjust gamma values for each eye presentation
imageTextureL = imadjust(imageTexture,[],[],gammaL);
imageTextureR = imadjust(imageTexture,[],[],gammaR);
% Render one view for each eye in stereoscopic mode:
for eye = 0:stereoscopic
clear image;
% Prepare each display
Screen('SelectStereoDrawBuffer', win, eye);
Screen('FillRect', win, [0 0 1]);
switch eye
case 0
image = Screen('MakeTexture', win, imageTextureL);
Screen('DrawTexture', win, image);
Screen('Flip', win, [], 1);
case 1
image = Screen('MakeTexture', win, imageTextureR);
Screen('DrawTexture', win, image);
Screen('Flip', win, [], 1);
end
% Display!
Screen('Flip', win, [], 1);
end
% Wait for response and get reaction time
time_start = GetSecs;
response = GetKbChar;
time_end = GetSecs;
response_time = floor((time_end - time_start)*1000);
% Write to file
fprintf(outFile, '%s, %s, %s, %f, %f, %f, %s, %f\n', imageFileParts{end}, imageFileParts{end-1}, imageFileParts{end-2}, gammaL, gammaR, gammaDiff, response, response_time);
end
% Close file and exit
fclose(outFile) ;
sca;
end
Acknowledgements
Michael Barnett
Trisha Lian
Haomiao Jiang