There are many virtual environments found in the serious game community that simulate real world scenarios. There is a broad range of fidelity and experimental controls among these serious games. An important component to most evaluations is the extent to which level of fidelity impacts the persons immersed in the serious game. While a great deal of virtual environment and serious game research has assessed the subjective state or feeling of the participant (e.g., the participant’s sense of presence) through the use of questionnaires, the current study examines participant experience by examining psychophysiological responses of participants to their surroundings. The primary goal in this study was evaluative: will a virtual environment with arousing contents result in increased sensory arousal if it is presented in a highly immersive configuration? A secondary goal of this study was to investigate the utility of our environment to offer varying levels of stimulus threat to impact the user’s experience of the virtual environment. Increased simulation fidelity in an arousing environment resulted in faster heart rates and increased startle eyeblink amplitudes, suggesting that higher fidelity scenarios had great efficacy related to sensory arousal.
Virtual environments (VEs) and serious games offer the potential to stimulate and measure changes in the users’ emotion, neurocognition, and motivation processes. The value in using simulation technology to produce serious games targeting such processes has been acknowledged by an encouraging body of research. Some of the work in this area has addressed affective processes: anxiety disorders, pain distraction, and posttraumatic stress disorder [
The incorporation of simulation technology into neuroergonomic and psychophysiological research is advancing at a steady rate [
In this paper, we attempt to build on earlier work that used psychophysiology to assess the propensity of users to respond to virtually generated sensory data as if they were real [
Discussions of the level of fidelity and experimental control needed for a VE often go beyond simple discussions of the “immersive” qualities of the environment to discussions of the impact upon the perceived feeling of “presence” of the individual while immersed in the environment [
Knowledge of the user-state during exposure to the VE is imperative for development and assessment of VE design. A number of presence studies have researched such issues using questionnaires [
Up until this point, VE studies have typically relied on self-report and behavioral measures to assess levels of fear and arousal [
Meehan et al. [
Psychophysiological metrics proffer the advantage of an objective measure of response that can be recorded in real-time as the environment is experienced, providing a continuous measure of presence. Indeed, highly immersive presentations are thought to not only increase subjective ratings, but also result in increased physiological responses [
In the current study, we aimed to look at the psychophysiological responses of participants experiencing “high” versus “low” levels of immersion into a virtual Iraqi scenario that had varying levels of stimulus intensity. Further, these psychophysiological responses may aid researchers in their development of VEs that balance fidelity and experimental control.
The primary goal in this study was evaluative: will a virtual environment with arousing contents result in increased sensory arousal if it is presented in a highly immersive configuration? To assess this, we looked at subjects immersed in a VE on two separate experimental runs consisting of both a “high” immersion condition and a “low” immersion condition. A secondary goal of this study was to investigate the utility of our environment to offer varying levels of stimulus threat to impact the user’s experience of the VE. Within each of the immersion conditions (high and low), arousal was manipulated by presenting participants with differing “safe” and “ambush” zones. Safe (low threat) zones consisted of little activity aside from driving down a desert road, while the more stressful ambush (high threat) zones included gunfire, explosions, and shouting amongst other stressors.
In the current study, startle eyeblink and heart rate were measured to assess psychophysiological differences in response to varying degrees of immersion and levels of arousal in a virtual Iraqi environment. Participants encountered a highly immersive VE while wearing a head mounted display (HMD) that used two OLED microdisplays with on-board 3D frame sequential video processors to deliver flicker-free motion. Together with the integrated
We hypothesized that the highly immersive condition would elicit a more intense physiological response to the stressful high threat zones due to the higher-fidelity environment. It is important to note that the volume levels in both the high and low immersion conditions were held constant in order to increase experimental control of the arousal manipulation in this study and to limit changes in arousal to changes brought on by increased levels of fidelity, rather than changes in volume level. It is our hope that this research will (1) proffer a greater understanding of the psychophysiological correlates of immersion in an arousing VE and (2) act as an initial validation (ecological validation) of the intended impact of varying degrees of stimulus intensity programmed into our virtual Iraqi city.
Serious game researchers are increasingly interested in working with clinicians to better understand a military service member’s ability to return to active duty. Recent conflicts have increased the prevalence of blast injuries to the head. Many of these brain injuries may have no external marker of injury. As a result, there is need for the serious games community to research innovative assessment methods. Currently, clinicians make “Return-to-Duty” assessments that are based upon the “Return-to-Play” guidelines found in Sports Medicine. Both have incorporated two dimensional cognitive assessments to aid in decisions related to resuming activities following a concussion. Unfortunately, these two dimensional computerized assessments were not developed with the intention of tapping into everyday behaviors like driving through a Middle Eastern city.
Serious gaming environments can increase the ecological validity of neurocognitive batteries through the use of simulation technologies for assessment and treatment planning. The success of such serious games may lead to a psychophysiological computing approach, in which such data gleaned from persons interacting within a military relevant simulation may be used to develop adaptive virtual environments for training and rehabilitation. A beginning step is the identification of the level of immersion needed for a serious game to proffer the appropriate level of arousal. This is the overarching goal of this study.
A total of 50 healthy college aged students (males:
The apparatus used for the virtual humvee (i.e., a high mobility multipurpose wheeled vehicle) included a Pentium 4 desktop computer with a 3 GHz Processor; 3 GB of RAM; and an nVidia GeForce 6800. Two monitors were used: (1) one for displaying the Launcher application which is used by the Examiner and (2) another for displaying the participant’s view of the VE in the HMD. Participants wore an eMagin Z800 head mounted display, and an InterSense InteriaCube 2+ attached for enhanced tracking. A Logitech Driving Force steering wheel was clamped on to the edge of a table in front of the monitors. A separate module consisting of the gas and brake pedals was positioned under the table. To increase the potential for sensory immersion, we built a tactile transducer using a three foot square platform with six Aura bass shaker speakers (AST-2B-04, 4 Ω 50 W Bass Shaker) attached. The tactile transducer was powered by a Sherwood RX-4105 amplifier with 100 Watts per Channel ×2 in Stereo Mode.
The software was designed using Virtual Battle Space 2 (VBS2). The VBS2 engine was used due to its robust fidelity simulation, ease of modification, and the fact that many military forces have adopted it. The VBS2 engine offers enhanced capability for interoperability and compatibility with existing standards for simulation. We designed the scenarios using a visual scenario editor and VBS2’s own scripting language. To implement the scenario we used VBS2-engine specific script language and the built-in Finite State Machine (FSM) functionality.
The application uses the Neuroscience and Simulation Interface (NSI) developed in the Neuroscience and Simulation Laboratory (NeuroSim) at the University of Southern California [
The University of Southern California’s Institutional Review Board approved the study. After informed consent was obtained, basic demographic information was obtained. Next, participants were immersed in a VE on two separate experimental runs consisting of both a “high” immersion condition and a “low” immersion condition. In the high immersion condition, participants wore a head mounted display (HMD) with full tracking capabilities and were free to explore their environment visually. The high immersion condition also made use of headphones and a tactile transducer floor to simulate the experience of a large vehicle. The low immersion condition consisted of the same virtual Iraqi scenario presented on a 17 inch laptop screen while wearing headphones. Stimuli within the virtual environment experienced in both immersion conditions were identical. The only differences between conditions were due to the inclusion of the enhanced presentation quality of the high immersion condition. The presentation order of high and low immersion conditions was counterbalanced across subjects.
The VE used in both immersion conditions was comprised of a series of low threat and high threat zones in a virtual Iraqi city. In both the high immersion and low immersion conditions, participants experienced the VE from the perspective of the driver of a Humvee. The speed of the vehicle was kept constant as it followed a predefined trajectory to control for time spent in each zone of the VE and to keep that time consistent across participants. Participants were given a basic 10° steering wheel to limit the trajectory, though they were instructed to stay on the road. This allowed for some level of control of the environment without sacrificing experimental control of the stimuli experienced. Low threat zones consisted mainly of a road surrounded by a desert landscape and were free of gunfire and other loud noises (see Figure
Serious gaming environment: low threat zone.
Serious gaming environment: high threat zone.
An acoustic startle stimulus was used to elicit startle eyeblink responses. Following accepted guidelines for human startle eyeblink electromyographic studies [
Psychophysiological assessment included: startle eyeblink amplitude and heart rate, which were recorded simultaneously throughout the experiment using Contact Precision Instruments equipment and a computer running SAM1 software.
One psychophysiological measure employed in the current study, and that is widely used as an index of valence (e.g., emotional positive or negative reactions), is electromyographic (EMG) recording of the startle eyeblink reflex. This reflex is often elicited by a burst of loud white noise with a nearly immediate rise and fall time presented at very high decibel levels (e.g., 110 dB) for a brief duration (e.g., 50 ms). Vrana et al. [
Startle eyeblink responses were recorded as electromyographic activity using two small (4 mm in diameter) silver-silver chloride electrodes placed over the orbicularis oculi muscle of the left eye and an 8 mm silver-silver chloride electrode placed behind the left ear to serve as a ground. One 4 mm electrode was placed directly below the pupil in forward gaze while the other was placed about 1 cm lateral to the first. The electrodes were placed as close to the eye as possible while still allowing the participant to open and close his or her eyes comfortably. Impedance between the two electrodes was measured and deemed acceptable if below 10 kΩ.
A second psychophysiological measure employed in the current study was the electrocardiographic (ECG) recording. Heart rate is a psychophysiological measure that is useful in differentiating between orienting and defensive responses. A person’s heart rate will accelerate during a defensive response and decelerate when orienting occurs [
ECG was recorded with use of a Lead 1 electrode placement, with one 8 mm silver-silver chloride electrode placed on the right inner forearm about 2 cm below the elbow and another placed in the same position on the left inner forearm. Electrode sites were cleaned with rubbing alcohol in order to improve contact.
The raw EMG signal was recorded at a rate of 1000 Hz throughout the experiment using a 10 Hz high pass and 200 Hz low pass filter. Raw signals were stored and exported for analysis in microvolt (
Due to the high levels of variability between participants in EMG responses, all blink amplitude values were standardized by taking the difference between each participant’s raw EMG amplitude value on each trial and that participant’s mean value across all trials and dividing by the standard deviation of all values. Scores were then subjected to a linear transformation resulting in a mean of 50 and a standard deviation of 10 for display purposes. This helped to ensure that all participants contributed to group means equally, minimizing the influence that one participant could have on the outcome of the subsequent analyses.
Interbeat intervals (IBIs) were scored as the time difference in milliseconds between successive R waves in the ECG signal. IBIs across a period of 5 seconds during each high threat and low threat zone were analyzed. The 5 second period occurred at least 10 seconds following any startle stimulus or large explosion, and no startle stimuli or explosions occurred during the period. A mean IBI score was recorded for each 5 second period and analyzed.
For each dependent variable, a 2 (immersion level) by 2 (zone type) repeated measures analysis of variance (ANOVA) was utilized to determine whether the high immersion setting was effective in increasing psychophysiological responding in general and whether it affected participants differently in low threat versus high threat zones.
All significant main effects and interactions were followed with paired samples
A significant immersion level main effect was uncovered, and was the result of increased blink amplitudes when participants were in the high immersion setting,
Distribution statistics for EMG eyeblink results.
Immersion level | Median safe zone | Median ambush zone | 25% quartile | 75% quartile | Minimum | Maximum |
---|---|---|---|---|---|---|
High | 50.6 | 52.3 | 40.7 | 58.5 | 36.3 | 65.4 |
Low | 47.8 | 47.1 | 38.2 | 55.7 | 32.4 | 62.4 |
Quartile and range data are given for the entire sample, while separate median values are given for both the safe and ambush zones.
EMG eyeblink response amplitudes for high and low immersion conditions. All amplitudes are reported as T-scores.
In general, ECG results were in agreement with EMG results. Again, a significant main effect of immersion level was found,
Distribution statistics for heart rate results.
Immersion level | Median safe zone | Median ambush zone | 25% quartile | 75% quartile | Minimum | Maximum |
---|---|---|---|---|---|---|
High | 67.5 | 68.2 | 60.2 | 83.5 | 52.3 | 95.4 |
Low | 66.3 | 66.4 | 59.2 | 80.8 | 49.9 | 90.2 |
Quartile and range data are given for the entire sample, while separate median values are given for both the safe and ambush zones.
Heart rate responses (in beats per minute) for high and low immersion conditions.
For our primary analysis in this study we sought to evaluate whether a highly immersive environment results in increased sensory arousal as measured by psychophysiological responses. Immersion effects were consistent with each measure. Participants consistently had faster heart rates when in the high immersion setting, suggesting that highly immersive VEs are more arousing than experiencing the same presentation on a computer screen. Participants also had larger startle eyeblinks when highly immersed, especially during the high threat zones, which suggests that the high immersion format facilitated startle eyeblinks.
Although on first reading these results appear to reflect the possibility that highly immersive VEs are more effective for eliciting increased arousal and producing fear responses than are low immersion VEs, this conclusion cannot be generalized given that there are restorative virtual environments that decrease arousal [
Another area that may put our results at odds with those reported by others is the issue that our study was for neuroscientific assessment of varying levels of fidelity and threat in a nonclinical sample of healthy college age students. Clinical populations tend to have significantly greater responding to threat stimuli presented in VEs when compared to nonclinical populations. For example, virtual stimuli that are relevant to a given phobia (e.g., phobics respond with more anxiety to phobogenic stimuli) will have more robust reactions to threatening stimuli. Further, it also seems intuitively clear that participants in the current study would react less to the threatening zones than would persons sensitive to the content of the virtual Iraq (e.g., soldiers returning from a rotation in Iraq, suffering from PTSD, or having been in a war zone) [
A secondary goal of this study was to investigate the utility of our environment to offer varying levels of stimulus threat to impact the user’s experience of the VE. Our analysis revealed that high threat zones were ineffective in creating statistically significant increases in arousal levels compared to the low threat zones, according to eyeblink and heart rate responses. However, participants appeared to show the appropriate directional trend toward increased heart rate and eyeblink responding in the high immersion setting, lending credence to the notion that the high immersion setting may be more effective in creating differential responding between the two zone types. However, these trends in response did not lead to significant interactions between immersion level and zone type.
The lack of differential responding in the high threat and low threat zones may have been due to the fixed order of presentation. While the presentation of the low and high immersion settings was counterbalanced across participants, the order of the zones was not. This meant that in each pair of low threat and high threat zones, the low threat zone was experienced first. While it is impossible to know what the exact effects of a counterbalanced presentation order would have on psychophysiological response, one possible explanation for the lack of differential responding may have been caused by habituation that led to a general decrease in responding during the high threat zones in comparison to the low threat zones that always preceded them. Had the high threat zones occurred prior to the low threat zones, a greater difference between the different types of zones may have been revealed, especially in the high immersion setting.
Additionally, the low threat zones were generally longer in duration than the high threat zones. This may have led to greater habituation taking place during the low threat zones, and created an additional confound that is difficult to account for in participant responses. Moreover, the low threat zones would transition into the high threat zones unpredictably and without warning, making the low threat zones potentially threatening.
The presentation of startle stimuli may also have added to the lack of differential responding in the low threat and high threat zones. In order to make the startle stimuli stand out from the background noise in the environment enough to elicit a startle response, the maximum capacity of the environmental noises were reduced to ten percent of the startle stimulus volume, greatly lowering the potentially arousing effects of gunshots and explosions experienced in the high threat zones.
It is important to note that there is parallel research on the restorative effects of nature that has explored the relationship between presence/immersion, psychophysiological measurements, and virtual reality. Previous research examining whether immersion in a VE simulated nature setting could produce restorative effects found that immersion in virtual nature settings has similar beneficial effects as exposure to surrogate nature. These results also suggest that VR can be used as a tool to study and understand restorative effects [
Future studies using this VE may be enhanced through counterbalancing of the order of zones experienced in the VE. Counterbalancing across participants to allow for half to experience low threat zones first and half to experience high threat zones first should help to alleviate the possible order effects that occurred in the present study. In order to better understand which particular zone is the most effective in increasing arousal, it is important that the high and low immersion conditions can begin with any zone. We can then counterbalance whether a low threat zone or a high threat zone is experienced first, and which particular low threat or high threat zone is experienced first. A uniform amount of time spent in each zone will also help to control the effects of habituation from zone to zone. Furthermore, in order to make the low threat zones more clearly perceived as being safe, a cue could be given to warn the user of the impending high threat zone. This way, the low threat zones are clearly separated from the high threat zones.
The removal of startle stimuli to allow background environmental noises to be played at one hundred percent capacity may also be beneficial in creating more arousing high threat zones. Eyeblink responses will no longer be an option as a psychophysiological measure of valence in a noisy background environment, but facial corrugator EMG recording can be used as an index of perceived valence in its stead. Other metrics such as electrodermal activity, respiration, and blood pressure may also be useful measures of arousal, and responses would most likely be enhanced by the increased volume levels.
A further enhancement for future studies would be the addition of subjective evaluations. Having both subjective and objective information would strengthen the validity of the results and allow combining them for the conclusions [
One of the main goals of the present research was to assess whether a VE with arousing contents would result in increased sensory arousal if it is presented in a highly immersive configuration. A secondary goal of this study was to investigate the utility of our environment to offer varying levels of stimulus threat to impact the user’s experience of the VE. Increased simulation fidelity in an arousing VE resulted in faster heart rates and increased startle eyeblink amplitudes, suggesting that higher fidelity scenarios with threatening contents were related to sensory arousal. Hence, highly immersive VEs appear to be more effective for eliciting increased arousal and producing fear responses than are low immersion VEs.
No financial Conflict of interests exist for any of the authors of this paper.
This research is partially supported by the US Army Research Laboratory, Human Research & Engineering Directorate, Translational Neuroscience Branch (Aberdeen Proving Ground, MD).