Figure 1. Perception of the exterior world relies on the brain’s building a mental model based on sensory input.
Affiliations:
1ITT
Technical Institute, Fort Myers, FL, USA.
2C.G. Masi
Technology Communications, Naples, FL, USA.
3Light
Reach Media Group, LLC, 15525 Cedarwood Dr. 104, Naples, Fl 34110,
USA.
*Correspondence to: cgmasi@cgmasi.com.
†Unit
2706, 2086 Rookery Bay Drive, Naples, FL 34114, USA.
Immersive Virtual Reality (IVR) is a communications medium, just like television, film, and even print. While being the newest, most sophisticated and ultimately the most effective communications medium, it is, by itself, devoid of content. It is, as it were, an empty vessel. Generally the value of an empty vessel consists of what it can contain and deliver. This presentation discusses how IVR differs from other media, and the characteristics of the messages it can best be used to carry. Examples from education, research, and entertainment are used to enumerate characteristics that prove diagnostic of what content is best, or can only be, delivered through IVR channels, and what content can – and should – be left for other less difficult- and expensive-to-produce media.
This paper discusses how IVR differs from other media, and the characteristics of the messages it can best be used to carry vs. those that should be left for other less difficult- and expensive-to-produce media.
Immersion 2016 contains many papers and presentations describing marvelous new technologies for presenting information in new and compelling ways. What sets these technologies apart is the extent to which they engage the user/viewer/audience. While using conventional presentation technologies, the user is constantly aware of the reality of their immediate surroundings. Immersive Virtual Reality (IVR) technologies, however, so fully engage the user, that the user's perception can be overwhelmed. The basic purpose of these technologies is to substitute the IVR experience for perception of the user's immediate environment.
Scope
This paper seeks to discuss in general terms the advantages and disadvantages IVR technologies have vis-a-vis conventional presentation technologies for conveying various types of content. Especially, it looks at the advantages and disadvantages of IVR in an educational setting.
Information
Presentation technologies, in general, are methods of conveying information to the user. So we start off by defining exactly what we mean by “information.” For the purposes of this paper, we take information to be a representation of reality. Reality is what exists in the external world. Information is data available to the user describing that reality. The user then creates a mental model of reality based on that information. Perception is the user's action of interpreting that model as an experience. Figure 1 illustrates this process. Actual objects existing in the ambient reality impinge on the user's senses in various ways: sight, sound, touch, etc. The brain then automatically constructs a model of its surroundings based on those sensations, informed by and based on memories of previous experiences. In a very real sense the mental model is the brain's “best guess” at what exists in external reality. The naive assumption is that the mental model corresponds exactly to the external reality.
Figure
1. Perception of the exterior world relies on the brain’s
building a mental model based on sensory input.
Information is thus a set of sensory inputs the brain can use to construct its model of reality.
Presentation technologies are an artificial means of conveying such information into the brain. That implies a sender (the author) who arranges the sensory information in a way that the receiver (the user) can most easily use to create the mental model the author wishes him or her to perceive. The more complete and consistent the information, the more compelling the mental model.
For example, graphic artists know that the human brain generally assumes that objects with lighter-valued, warmer colors are in the foreground. The author can thus use this tendency to create a three-dimensional illusion in a two-dimensional picture, as demonstrated in Figure 2. There are a large number of such perceptual tendencies common in human psyches, and the artist's skill level largely depends on the range of such tendencies he or she is competent to use to create illusions.
Figure
2. Surrealist Landscape with White Thing by C.G.
Masi
Ultimately, the purpose of this process is to effect a change in the user's perception of reality. That is, to create a memory in the user based on what the author wants the user to perceive.
Medium
Over time, humans have devised a number of strategies for conveying information to users, mainly through the senses of sound and sight. These break down into audio, textual, and visual categories.
Auditory methods, such as music, have proven especially useful in conveying emotion.
“The blues ain't nothing but a good man feelin' bad.”
–
Lee Roy "Lasses" White [1]
The above quote points out the power of auditory stimulation (music) to express emotional information.
Other examples include the strong emotional content of Pyotr Ilyich Tchaikovsky's final completed symphony (Symphony No. 6 in B minor, Op. 74) [2] in Media 1, often called Pathétique; the gay lightness of Opus 314, By the Beautiful Blue Danube by Johann Strauss [3] in Media 2; the loneliness and remorse expressed in Behind Blue Eyes by The Who [4] in Media 3.
Media 2. Johann Strauss’ By the Beautiful Blue Danube evokes joy. BlueDanubeClip.wav
Media 3. The Who’s Behind Blue Eyes evokes sadness and loss. BehindBlueEyesClip.wav
Textual methods are ideal for conveying quantitative information. Examples, of course, abound. Most notably, the five Ws (who, what, when, where, and why) have been recognized since the time of the Roman Empire as the basis for journalistic writing. They point out the importance of quantitative facts in telling a complete story. The portion of a spreadsheet presented in Figure 3 illustrates the power of textual methods for conveying quantitative information [5].
Figure
3. Spreadsheets use textual methods to convey quantitative
information concisely and clearly.
Visual methods best convey qualitative information, such as color, shape, relationships, etc. Especially, the transitoriness of motion pictures makes it difficult to convey quantitative information, but is ideal for expressing relationships and interconnections. Again, Figure 2 demonstrates the human minds' ability to reconstruct a three dimensional scene from two dimensional sensory information.
While presentation technologies, such as print, music, film, and television, provide media for conveying information, what is most important is the message. In other words, the information content invented by the author and perceived by the user is of paramount importance. The landscape images of twentieth-century photographer Ansel Adams (Figure 4) derive their power from the artist's ability to overcome the limitations of his small-scale, two-dimensional, black-and-white photographic medium, and cause the user to perceive a powerful vision of an immense sweep of landscape [6]. It is the user's experience of that “vision of an immense sweep of landscape” that is the whole point of the exercise!
Figure
4. Following Ansel Adams, photographer Peter Essick overcame
the limitations of black and white photography to create this
powerful image with exceptional depth illusion.
There are two kinds of information a work can convey: factual and entertainment. Factual information is presented as verified to be “true” and the user is supposed to retain and act upon it. Entertainment is presented as fantasy information provided to hold the user's attention and interest, but then to be forgotten immediately, or at least not acted upon.
Educators traditionally concentrate on factual information and suppress entertainment information, which is why educational material is often thought of as boring. This tradition, however, makes it relatively easy for the user to separate the factual from the purely entertaining information, since the latter is largely absent.
Of course, in order to successfully transfer information, it is necessary to engage the user's attention in the first place. So, it is necessary to provide entertaining information as well as factual information. The best educators provide both, and are experts in providing cues to the user about what is factual and what is purely entertainment.
In fact, the above statement applies equally to literature provided ostensibly for entertainment purposes. Herman Melville, for example, added numerous essays to his novel Moby Dick that purport to provide factual information about life aboard a whaling ship interspersed with the narrative of his story [7]. Generations of schoolchildren have found the combination fascinating, and have had no difficulty separating the factual from the entertainment elements.
Another example is the work of the early Greek poet Homer, who taught religious and social values illustrated by an entertaining story. His Iliad is a treasure trove of factual information about Bronze Age Greek religious practices and social conventions [8]. It has also been generally acknowleged as the first example of the War Story genre of literature.
We have recently completed a short film (Bonnie Pureheart and the Golden Ghost) that provides factual information about the culture of the Calusa Indians of Southern Florida and their historical relationship with the Spanish conquistadores embedded in a purely fictional modern treasure-hunt story. We separate fact from fiction by, for example, having the factual information presented by an archeology professor, whose pronouncements are treated as true information.
The purpose of this paper is to look at how IVR technologies differ from conventional presentation technologies, and what that means to the types of messages they should be called upon to deliver in an educational setting. As hinted above, different technologies are best used to convey different kinds of messages.
Conventional Technologies
Conventional presentation technologies do not provide a complete all-encompassing sensory field. That is, for example, a motion picture covers most, but not all, of a viewer's visual field. Especially, the viewer can come back to reality immediately by simply looking away from the direction of the screen. Music, of course, typically does not include a visual field, so the audience is constantly reminded of reality by their visual perception of their immediate surroundings.
Immersive Technologies
Immersive presentation technologies have the power to overwhelm the user's sensory fields. The headphones audiophiles have used for over a century [9] to muffle ambient environmental sounds, allow the audio information of the presentation to overwhelm the user's sense of hearing.
Virtual reality goggles, now available from Samsung, Sony, and other manufacturers [10] perform the same function for the user's visual field as audio headphones perform for the audio field. They block ambient signals from the user's immediate environment, effectively completing the user's isolation from direct perception of his or her surroundings.
Figure
5. Virtual reality goggles, or headsets, block ambient
visual and auditory sensory input.
Of course, this isolation creates a serious problem for the user and for anyone and anything in the user's immediate surroundings: the user is unable to perceive potential threats to safety and security in the environment, and may perform physical actions based on the IVR experience that are inappropriate in their ambient environment.
This issue is clearly and humorously illustrated in a 30 second commercial entitled Hunger Can't Kill My Game (Media 4). Several young men are shown playing a violent video game using IVR technology. Alternate shots show the IVR world and the ambient environment, with one of the young men breaking into the game to prepare a snack without leaving the IVR experience. He has to use his kinesthetic memory of spatial locations in the real world along with his remaining unblocked sensory inputs (mainly touch) to find the microwave oven containing the food he wants. It ends humorously with his taking physical action based on the IVR experience that causes damage in the real world.
Media 4. TV commercial featuring IVR game play. https://vimeo.com/128896460
Another issue with IVR technology is cognitive dissonance between the users unblocked kinesthetic (mainly balance) sense and IVR-blocked visual sense. This effect leads to motion sickness (simulator sickness) [11] when the IVR message involves a lot of rapid acceleration over a sufficient period of time.
Note that the IVR experience does not overwhelm the user's tactile, olfactory or kinesthetic senses. To our knowledge, no fully successful technology yet exists to replace those senses.
Our 2013 novel Silver Rivers [12] presents illustrations of how IVR technology may be used in research and educational environments. The novel posits an historically significant underground archeological site containing toxic amounts of mercury and an anoxic atmosphere. Human investigators cannot be allowed into the space because of the toxic atmosphere, and the toxic elements cannot be removed without damaging the site.
In the novel, researchers use autonomous robots to explore the site and create a three-dimensional IVR representation of the space and its contents. They are thus able to extract meaningful information from the site without damaging it. They do it in two ways: via a non-IVR projected-3D theater system and via IVR. In the non-IVR theater, the characters are able to successfully interact with each other in reality while viewing the 3D presentation. In the IVR version, the single character has to place himself in a safe, secure environment while extracting information from the IVR presentation. In fact, he has to break the IVR experience (by removing the IVR headset) to record observations on a computer in ambient reality.
The novel discusses use of the IVR presentation for educational purposes by positing an IVR presentation where visitors could virtually explore the site on their own. They would be provided with a safe and secure physical environment in which they could interact with the IVR experience. They would be provided with a virtual point-of-view, which they could move throughout the virtual space via a Wii-style baton, without needing to take unsafe physical actions.
Of special note is the demonstration that multiple individuals can explore the same IVR space simultaneously without interacting with each other. For example, the user with the IVR technology is able to enter the space virtually at the same time as the users of the non-IVR theater without knowing or caring that there are other users. This is true because the virtual reality actually exists only in the user's mind.
Considerable work is being done on augmented virtual reality (AVR). AVR seems like a logical and technically simple solution – especially to the rousal and motion-sickness issues – plaguing IVR technology. It is, however, actually more difficult to achieve.
Recent experiments with Google Glass technology, where a small display is added to a clear lens and overlaid on part of the user’s visual field.[13] illustrate one technical solution. The user percieves two realities: the ambient environment seen through the clear lenses, and the virtual environment seen in the VR display.
Such an AVR solution promises to solve the rousal problem by not overwhelming the user’s senses, and the motion-sickness problem by providing a clear view of the user’s ambient environment to match his or her kinesthetic sense. Unfortunately, it raises two other, related problems, both caused by mixing the virtual and ambient realities.
The first problem, related to the rousal issue, is inattention to the ambient environment. When presented with both VR and real worlds simultaneously, how well can a user percieve both simultaneously? (See discussion below.)
The second problem is matching the VR presentation to the ambient environment. The hope is that it will be possible to add VR objects into the ambient scene in such a way that they will appear as part of the scene. The difficulty is, of course, that the ambient environment is chaotic. It is impossible to predict with accuracy where the virtual objects should be placed in the visual field to match the positions of actual objects in the chaotic ambient scene. The VR system, therefore, will need to sense and understand the ambient scene from the user’s actual point of view to calculate the positions and orientations of the VR objects it projects to create AVR.
Humans, and many other higher animals, are quite familiar with a natural form of IVR – dreaming. They instinctively seek out a safe, secure physical location where they can break off perception of reality to sleep. In the sleep state, they are largely immobilized. While they can generally imagine a full range of movement, motor signals imagined in the dream state are largely prevented from reaching the muscles. This prevents the dreamer from taking action based on the dream content that would be hazardous in reality. Natural mechanisms are available (i.e., tactile senses, loud noises) to rouse the sleeper in the event of a threat arising in reality.
Similarly, humans daydream. In a real sense, daydreaming is the logical equivalent of AVR. The daydreamer is more-or-less aware of the ambient environment while fantasizing a virtual environment. It is important to note that daydreamers are notoriously prone to inattention to their ambient environments.
The good news, of course, is that the correspondence between dreaming and daydreaming to IVR and AVR respectively shows that the human brain is capable of operating in both of these environments. Our task is mainly to use presentation technology to invoke these capabilities.
Immersive Virtual Reality differs from other presentation technologies in the compelling nature of the experience, which, by its nature, isolates the individual from perception of his or her immediate environment. This presents three issues:
Security – The user is less able to perceive threats arising in reality;
Safety – The user may take action or make movements based on the IVR experience that prove hazardous in reality;
Rousal – Some mechanism is needed to signal to the user that reality requires his or her immediate attention, and allows breaking the IVR experience immediately to respond in a timely fashion.
To be fully effective, any IVR presentation technology must address these three issues in a reliable and effective way.
IVR presentations are best reserved for content that invites the user to actively participate in an alternate reality. Examples include exploring dangerous or inaccessible places. This could be because the space includes elements that are physically toxic, inconveniently remote, or don't exist in reality at all. IVR technology also allows more visitors to the space than could be physically accommodated in reality because the IVR space actually exists in the user's mind, not in real space.
Lastly, some mechanism for the user to manipulate his or her IVR experience without making large, potentially hazardous muscular movements is highly advisable.
Examples of content where IVR would be of great value include:
Exploring other planetary bodies via robotic systems;
Visiting remote locations, such as cities in foreign countries;
Visiting museums where space for viewing exhibits is limited;
Exploring collections of objects that have never actually been assembled in one location;
Providing immersive experience of historical events or fictional events.
Examples of content that should not be presented via IVR include:
Experiences that do not include all senses, such as music, or graphic arts;
Experiences that require the user to make large movements, such as physically violent video games;
At least until further technologies are developed, experiences that rely on tactile, olfactory or kinesthetic senses for complete appreciation.
1. L. White, Nigger Blues (sheet music) ( Bush & Gerts, Dallas, TX, 1913).
2. P. Tchaikovsky, Symphony No.6, Op.74 (P. Jurgenson, Moscow,1893).
3. J. Strauss, On the beautiful blue Danube, op. 314 (John Church Company, Cincinnati, OH, 1868).
4. P. Townshend, Behind Blue Eyes (Decca Music Group, London, UK, 1971).
5. C.G. Masi, CF Business Plan, Cruizin’ Florida LLC (2015).
6. P. Essick, THE ANSEL ADAMS WILDERNESS (National Geographic Books, Washington, DC, 2012).
7. H. Melville, Moby Dick (Harper & Brothers, New York, 1851).
8. Homer, Illiad, (Penguin Books, London UK, 1950) [E.V. Rieu translator]
9. M. Newman, The History of Headphones Cool Material, 24 Dec 2013.
10. P. Lamkin, The best VR headsets: The virtual reality race is on Wareable, 12 Apr 2016.
11. Introduction to Best Practices, https://developer.oculus.com/documentation/intro-vr/latest/concepts/bp_intro/
12. C.G. Masi, Silver Rivers (C.G. Masi, Naples FL, 2012).
13. M. Honan, I, Glasshole: My Year With Google Glass Wired, 30 Dec 2013.