While proponents of new media or computer sciences regard current technologies of virtual realities such as the Oculus Rift (Oculus VR 2016) as constituting the pinnacle of immersive media, representations and illusions of three dimensional spaces start with early art forms. Coupled to the creation of illusionary spaces are the attempts of physically or mentally entering such designed, augmented or artificial environments. Hence, the concept of immersion can not be regarded solely as an effect of digital media technology or modern interfaces. Some immersive techniques can be connected to earlier forms of artistic media and cultural practices and thereby situated within general acts of perception. Other forms of immersion, especially those based on computing systems, derive from current innovations and technologies, and are therefore just being formed and stabilized as new cultural programs. Taking a closer look at some milestones of the art of illusionary spaces might help to liberate the concept of immersion from the technical or solely digitally-oriented rubrics under which it is often thought of. And instead of concentrating on technology, I suggest focusing on the interfaces of immersive media. To argue for this approach, I would like to elaborate how various interfaces of spatial media create effects of immersion by addressing the body in different ways.

EARLY FORMS OF IMMERSION IN ILLUSIONARY SPACES

Oliver Grau’s extensive research has proven that the genre of immersive aesthetic spaces has been actively pursued since pre-modern times:

“The idea of virtual reality only appears to be without a history; in fact, it rests firmly on historical art traditions, which belong to a discontinuous movement of seeking illusionary image spaces.”¹

The next leap forward in the art of immersive spaces arose when techniques of optical illusions could be combined with motion. Fred Waller’s Cinerama of 1952 did not just unite the words “Cinema” and “Panorama”. In Cinerama, three cameras and a circular screen were used to offer panoramic viewing in correct perspective. But to transcend still images, this procedure was combined with motion pictures – much to the delight and sometimes to frightening effects for the audience.

“The shrill screams of the ladies and the pop-eyed amazement of the men when the huge screen was opened to its full size and a thrillingly realistic ride on a roller-coaster was pictured upon it, attested to the shock of the surprise.”⁶

Very shortly after the Cinerama, in 1956, Morton Heilig wanted to create more than just optical illusions of movement in three dimensional space in his “reality machines”⁷. Hence, he continued the “-orama” sort of naming tradition and turned it into a complete multi-sensorial experience in Sensorama:

“The Revolutionary Motion Picture System that takes you into another world with 3-D, wide vision, motion, color, stereo-sound, aromas, wind, vibration.” (Sensorama Advertising, 1962) 

Besides vehicle simulation like in Sensorama, Morton Heilig also patented the first movable and Head Mounted Display (HMD) with 3D graphics, stereo sound, and an “Odor Generator”, ⁸ but was never able to build one. But already in 1968, the first completely functional Head Mounted Display could be implemented by Ivan Sutherland at MIT. It consisted of stereoscopic displays for each eye and a mechanical tracking system for adapting the visual output to the current view point.⁹

IMMERSION - SIMPLY AN EFFECT OF DEPTH CUES?

Immersive practices and techniques of spatial illusion make use of certain characteristics of human perception. The visual system uses various depth cues to extract spatial information out of its environment. For example, oculomotor cues (oculus is the Greek word for eye) mean spatial information that is derived from the motor function of the eye. In physical spaces, the stretching and relaxing of the muscles of the eye lens and the rotating of the eyes give information about the distance of objects. Of course, these cues do not work with illusionary spaces of paintings, walls, screens and so on as all these objects are placed in the same distance to the eyes. Nevertheless, two-dimensional images may give the impression of spatial depth by using monocular depth cues such as occlusion, relative size, texture gradient, and linear and aerial perspective (i.e. changes in contrast and color). With these techniques, an illusion of spatial depth can be perceived by just one eye. The most advanced technique to create illusionary image spaces is binocular disparity and stereopsis, as used in stereoscopic apparatuses since the 1900s.

Another technique for the creation of illusionary spaces is motion. The so-called kinetic depth effect¹⁰ describes the optical illusion of three dimensions by motion, for example, in images that change from flat into a three dimensional figure just by rotating.¹¹ Furthermore, the moving of objects within the field of view can be used to create spatial effects in a whole scene. In real environments, motion parallax is a depth cue that results from our own motion. As we move, objects that are closer to us move further across our field of view than do objects that are in the distance. This effect can be used to create a kinetic illusion of depth, for example in films, if closer objects seem to move faster than those further away. A uniform motion of objects in the field of view may even be perceived as one’s own body movements. That is why images of a street with objects passing by on the roadside are the simplest ways of simulating a ride, just like in the vehicle simulation of Sensorama.

What can be gathered from pre-digital examples of immersive media is that the ideas and techniques used in deceiving perception and to create illusions of artificial spaces in three dimensions are not new as such. Of course, the devices and technologies change with technological progress from control of lighting conditions to complex computer hardware and software, whereas the biological basis of the perception of space quite obviously remains the same. So why not simply define immersion as an objectively measurable effect of certain parameters of media technology depending on human perception and presented depth cues? Following such a technological approach, the definition of three distinct degrees of immersion within computer science derives from devices only:

1) Virtual environments are regarded as “non-immersive”, when the device only enables a viewpoint from outside the environment and the user only looks at the artificial world.
2) Environments are called “semi-immersive”, when the viewpoint is inside the environment like in a cave, ¹² but there are still other stimuli available.
3) Virtual environments are defined as “fully-immersive”, when they work with devices like a head mounted display that shows a viewpoint inside the environment and at the same time blocks out other sensory information.

And yet, the broad range of examples from antiquity to current media shows that immersion in artificial space, while certainly influenced by technology, is not dependent on technology alone. Immersion did not automatically increase with technological progress. Therefore, the concept of immersion can not be defined solely by physical models, be they based on technical specifications or on ‘human factors’ and biological capacities. What does change, besides technological progress, are the regimes of viewing, the staging, the ways of what is set into scene and to what end, and the cultural contexts of its perception or usage – from the staging of saints, religion and religious power in circular frescoes, to political expositions of battles as in the famous Sedan Panorama,¹³ to mass entertainment like in Cinerama. Still, immersive practices, though subject to ongoing transformation, sometimes crystallize into cultural figures, “topoi”, or even “moulds for experience”.¹⁴ And with that, the cultural programs how to read such images, how to decipher illusionary spaces, and how to handle perception and apperception of such “reality machines” become the central point of attention. Instead of concentrating on technology, I suggest focusing on the interplay between recipient and media, the interaction between user and technology, in short on the interfaces of media practice. It is only in the interfaces that it becomes evident how cultural practices, media techniques, and technological devices are intertwined in the creation and usage of immersive environments. And in shifting the focus towards the interfaces of immersive media – including devices, practices, and cultural programs – substantial insight on the concept of immersion in virtual environments can be gained without limiting it to technical definitions on the one side or opening the term to arbitrariness on the other side. I will try to prove my point by picking out the topos of what has been called surrogate or “armchair travelling”¹⁵ as an example, considering how various interfaces of immersive media address the body in different ways and to what effect.

INTERFACE TECHNIQUES AND PRACTICES OF IMMERSIVE MEDIA

In computer science, an interface is defined as the boundary or contact surface for human-computer interaction. The interface includes both sides of data exchange, via input devices such as keyboard and mouse as well as output devices such as the screen or loudspeaker. Even more important than these hardware and software components, the interface also ‘translates’ and mediates between the two unlike partners, for instance, by providing interaction techniques and metaphors based on cultural programs instead of digital code for the representations, signs, icons, and images that are used to communicate and interact via the screen. As it evolved, the concept of the interface has come to encompass the functions to be performed and cognitive, emotional, and cultural aspects of the user’s experience as well¹⁶. Nowadays, interfaces enable all kinds of human-computer communication and interaction¹⁷. Nevertheless, because of the need for input and output devices, there has always been a request for the vanishing of the interface in the fully immersive “ultimate display”¹⁸ or in “interfaceless interface(s)” of the future¹⁹. “The ultimate display would, of course, be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such room would be fatal. With appropriate programming, such a display could literally be the Wonderland into which Alice walked.”²⁰ So instead of just looking at the screen, interfaces of immersive media are described with metaphors such as “through the looking glass“,²¹ or as a “doorway to other worlds”,²² where users could literally be inside a virtual environment and act as they would in the real world. Though meant for seamless futuristic devices, the term “interfaceless interface(s)”²³ could also be used for pre-digital immersive media and art forms. Common to all of these early examples of illusionary spaces are that they are media, where the ‘interface’ – for lack of a better word – only allows the representation of the output side of the communication. There is no interacting with these media in the sense of mutual adaption, nor any kind of input from the user’s side. That is why Lev Manovich uniformly uses the term “screen” for any “flat rectangular surface, existing in the space of our body and acting as a window into another space”,²⁴ including anything from renaissance paintings to photography and film. Although he divides his archaeology of screens after the temporality of what they show,²⁵ he points out that the relation of the body and the screen constantly remains that of an immobilised body in front of increasingly realistic images. So, what exactly constitutes immersive effects of being drawn into such “interfaceless” image spaces?

According to Oliver Grau, earlier illusionary spaces have a frame or a marked difference between the representation – the illusionary space – and the ‘real’ space. He argues that it is exactly this vanishing difference or border to reality that marks later concepts of “immersive” or what he calls “interactive image spaces”.²⁶ If we do not have input devices for interaction, pre-digital immersion must rely purely on visual output and on optical illusions created by the aforementioned depth cues. But the border to reality does not only vanish in ever more realistic images, it literally becomes ‘out of sight’ by manipulating the limiting frame of the image and the field of view of the spectators. Basically, immersive strategies work along two main lines: On the one hand, there are illusionary spaces based on given spatial conditions such as circular frescos or wall paintings, and panoramas. These illusions actually surround the observer or many observers, if not always in 360°, at least partially. Consequently, the point of view of the spectator is always one from the inside – exactly as computer scientists requested for digital environments. And if the painting or the screen is just big enough, it fills out the entire field of view of the spectators, so that all they see is the surrounding image space. On the other hand, there are illusionary techniques and devices that work with so called peep-throughs. Here, just one observer looks into an artificial space through a small device that blocks out any other visual input. Although the viewer is not really inside the peep box, he is drawn into the image space by the immersive strategy of restricting his field of view to the confined space of the peep-through. In both ways of manipulating the field of view of the spectators, the border to reality disappears from sight. In surroundings, the observer physically enters the media space, where he can turn his head, move around to a certain degree, and experience different views of the artificial environment. Though body movements may be restricted, viewers are certainly not immobilized. In contrast, early peep boxes could not enable movements or changes in the field of view and always presented the same image space. And yet, the disappearance of the (visual) presence of one’s own body together with the depth cues and the infinity of details, for example contained in a stereograph, inspired euphoric descriptions about leaving one’s body behind and traveling in spirit.²⁷ When Sutherland finally built and programmed the first digital Head Mounted Display, one distinctive innovation was that he added the tracking of head positions to earlier concepts of a movable display close to the head.²⁸ By position tracking of the head, images of the HMD could be adapted to the actual viewpoint, and for the first time, also the viewers of such smaller devices could change what they saw just by turning the head.

In fact, the tracking of head or body positions and using this kind of information in various feedback devices and ‘machines’ marks a turning point in the interfaces of immersive media. The former “interfaceless” media could only work with visual immersive strategies, where the body, if at all, could only be used for a change of view. When interfaces with feedback or input from the user side were developed – no matter how basic in terms of technology and not necessarily digital – image spaces started to become not only passive output, but responsive to the viewer, immersive strategies leapt to a next level, and last but not least, the body returned.

Myron Krueger was the first artist to shift emphasis from optical illusions to full body interaction in his Responsive Environments:

“It is the composition of these relationships between action and response that is important. The beauty of the visual and aural response is secondary. Response is the medium!”²⁹

Starting in the late 1960s, Krueger developed numerous artistic projects such as Videoplace, where projectors, video cameras, and onscreen silhouettes were used to place users – respectively their images – within a surrounding environment that responded to their movements and actions.

Audience members could playfully interact with the computer or each other, for example by finger painting or touching each other’s silhouettes, and see the response on huge screens.³⁰ It is because of this motion tracking in his Responsive Environments that Myron Krueger has been called the ‘father’ of artificial reality,³¹ although his earliest installations did not even use computers.

Also in the field of digital technology, from the manipulation of viewpoints in Head Mounted Displays, the idea of changing the positions by movements of viewers was not far away. And with manipulating body positions, the motif of traveling through artificial space returned as an immersive practice. The starting point for the idea of surrogate traveling within computer science was a student project of Peter Clay at the Massachusets Intstitute of Technology (MIT), who suggested ‘mapping’ the floors of MIT and videotaped his paths with the help of Bob Mohl und Michael Naimark .³² As Michael Naimark states,

“Peter and Bob made a simple computer program that allowed control of speed and direction moving up and down the hallways. Voila! ‘Virtual travel’”³³

By following the principle of movie mapping – that is “the process of rigorously filming path and turn sequences to simulate interactive travel and to use as a spatial interface for a multimedia database”³⁴ – the team of Andy Lippman from the MIT Architecture Machine Group created a simulated ride through Aspen in Colorado and called it the “Aspen Movie Map”.

Earlier examples of the cultural topos of surrogate traveling rose with motion pictures, for example, in the rollercoaster-scene of Cinerama’s first show or in Heilig’s Sensorama rides. Instead of being passively moved through an environment like in these surrogate travels, the Aspen Movie Map of 1978 was the first travel application to enable active control of the ride by providing an interface for navigation, for example via arrows for changes in direction or by choosing a destination or a path in the map. By using a spatial interface, the virtual environment was made accessible through locomotion – if not by physical movements, at least by a sensory illusion of movement.

In the following years, a broad variety of interfaces with input devices for locomotion was developed, both in computer science and in artistic contexts, such as the Legible City by Jeffrey Shaw that could be explored on a bicycle.³⁵

All of them used body movements as a means to actively explore virtual environments and to increase the sense of immersion and ‘being there’. Besides locomotion devices, researchers also explored interfaces for the manipulation of objects. The first input device for the manipulation of virtual objects by hand was the Sayre Glove of 1977.³⁶ The VPL DataGlove of Thomas Zimmermann and Jaron Lanier³⁷ was the first commercially used device that used the hand for glove-based input and integrated an image of the hand into the virtual environment. So technically speaking, in recent virtual environments, we usually have a “goggles and gloves”³⁸ interface constellation to enable input and output between digital image spaces and viewers.

Nowadays, 3D images can be seen with a conventional computer monitor for monocular cues only, or using a monitor in stereo mode with stereo glasses and head tracker like with a so called Fishtank Virtual Reality System.³⁹ The main principle of stereoscopy with up-to-date stereo glasses is, still, to separate one image into two pictures, one for each eye. But today we have a variety of glasses for 3D image effects: Passive stereo glasses use either polarization or spectral filters. While spectral displays present overlaid images in different colors and use corresponding glasses with red/blue, red/green, or red/cyan films, polarized glasses create the illusion of three-dimensional images by restricting the light that reaches each eye. To present a stereoscopic motion picture, two images are projected superimposed onto the same screen through orthogonal polarizing filters. In contrast to such passive glasses, active shutter glasses are synchronized to open and close their shutters very fast so that the two images are perceived as one.⁴⁰ Most interface output systems use additional glasses, for example, the CAVE,⁴¹ a system that works like a digital panorama with input devices. Furthermore, we have hemispheric displays and of course, advanced head mounted displays like the Oculus Rift now with included earphones (Oculus VR 2015). So far, there are not so many differences in technological settings – surrounding walls are computer screens now, but still circular, cubic, or curved to fill out the user’s field of view. Glasses are much more advanced compared to the Holmes Card Viewer, but work on the same principles. Nevertheless, some considerable changes have taken place in the interfaces of media spaces and in the strategies of inducing immersion.

Evidently, recent immersive media such as virtual environments draw on regimes of viewing and on visual strategies of immersion in artificial space. And in doing so, they can be connected to immersive cultural practices within the evolution of media realities. What distinguishes today’s virtual environments from their ancestors, is that the concept of immersion in artificial space has been expanded from calm aesthetic contemplation and mere optical illusions for a passive observer, to a means of participation and interaction for the active user, where participation is enabled by interface with position tracking and various new input devices. Hence, a Virtual Environment is defined as

“A synthetic, spatial (usually 3D) world seen from a first-person point of view. The view in a Virtual Environment is under the real-time control of the user.”⁴²

As has been shown by empirical research, it is exactly that real-time control and the possibility to participate that leads to much higher ratings on perceived immersion and presence or “being there” than solely advanced 3D images.⁴³ But not any kind of control, participation, or interaction works equally well as an immersive interface strategy. I have argued that besides perceptive illusions and viewpoint control, the central strategy of spatial media consists of addressing the body. Accordingly, the main interaction technique with interfaces of immersive media is to make the virtual environment accessible for the user through movement in space. But this immersive media practice, which again relates to the much older topos of exploring artificial space by traveling, is no longer characterized as being moved around passively in rides. Instead, recent interfaces enable users to actively navigate, explore, and manipulate the environment, using special input devices for locomotion such as treadmills, bicycles etc., as well as all kinds of steering devices fitting for the presented environment, and input devices for object manipulation like special touch controllers, 3D mice, or data gloves. Right now, the most advanced devices in the high-tech sector are so called “force feedback devices” that couple input with immediate haptic or tactile feedback. Instead of trying to induce immersion by presenting ever more realistic image spaces, interfaces of immersive media have to address the body by enabling kinesthetic action. Or as Myron Krueger points out in an interview: “Whereas the HMD folks thought that 3D scenery was the essence of reality, I felt that the degree of physical involvement was the measure of immersion.”⁴⁴

References:

Altick, Richard Daniel. The Shows of London. Cambridge, MA: Harvard University Press, 1978.

Bolter, Jay David, and Richard Grusin. Remediation: Understanding New Media. Cambridge, MA: MIT Press, 1999.

Bowman, Doug A., Ernst Kruijf, Joseph J. LaViola, Jr., and Ivan Poupyrev. 3D User Interfaces: Theory and Practice. Boston: Addison-Wesley, 2005.

Clay, Peter E.. Surrogate Travel via Optical Videodisc. Thesis Urb. Stud B.S., Boston MA: MIT, 1978.

Crowther, Bosley. “New Movie Projection System Shown Here; Giant Wide Angle Screen Utilized.” New York Times, October 1, 1952. http://www.nytimes.com/movie/review?res=9E0DE6DF1E3CE23BBC4953DFB6678389649EDE

Cruz-Neira, C., Daniel J. Sandin, and Thomas A. DeFanti. „Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVE.“ In Proceedings of SIGGRAPH ´93, 135-142. ACM, 1993.

Defanti, Tom A., and Daniel J. Sandin. „Final Report to the National Endowment of the Arts.“ Technical Report US NEA R60-34-163, Chicago: University of Illinois at Chicago Circle, 1977.

Fisher, Scott S. “Virtual Environments: Personal Simulations & Telepresence.” In Virtual Reality: Theory, Practice and Promise, edited by S. Helsel and J. Roth. Westport: Meckler Publishing, 1991.

Grau, Oliver. Virtual Art. From Illusion to Immersion. Cambridge, MA: MIT Press, 2003.

Grau, Oliver. “Immersion and Interaction. From circular frescoes to interactive image spaces.” In MediaArtNet - 1: Survey of Media Art. Joint project of ZKM and Goethe Institute, edited by Dieter Daniels, and Rudolf Frieling, 268-291 (German version), 292-313 (English version). Vienna: Springer-Verlag, 2004.

Heilig, Morton L. Stereoscopic-Television Apparatus for Individual Use. U.S. Patent No. 2,955,156, October 4, 1960.

Heilig, Morton L. “El Cine del Futuro: The Cinema of the Future.” Presence 1, no. 3 (1992): 279–294, reprinted from Espacios (1955): 23–24.

Huhtamo, Erkki. “Armchair Traveller on the Ford of Jordan. The Home, the Stereoscope and the Virtual Voyager.” Mediamatic Magazine 8, no. 2,3 (1995).

Krueger, Myron W. “Responsive Environments.” In AFIPS 46 National Computer Conference Proceedings, 423-433. Montvale: N.J. AFIPS Press, 1977. Reprinted in: Wardrip-Fruin, Noah, and Nick Montfort, ed. The New Media Reader (Cambridge, MA: MIT Press 2003), 379-389.

Krueger, Myron W. Artificial Reality. Reading MA: Addison-Wesley, 1983.

Lanier, Jaron. “Beyond Goggles and Gloves.” Byte 22, no. 9 (1997): 32-42.

Lambourne, Lionel. Victorian Painting. London: Phaidon Press, 1999.

Laurel, Brenda, and S. Joy Mountford. “Introduction.” In The Art of Human-Computer Interface Design, edited by Brenda Laurel, xi-xvi. Reading: MA Addison-Wesley, 1999.

Licklider, Joseph Carl Robnett. “The Computer as a Communication Device.” Science and Technology 76 (April, 1968): 21-31. http://memex.org/licklider.pdf.

Manovich, Lev. “An Archeology of a Computer Screen.” Die Zukunft des Körpers I. Kunstforum International 132 (November 1995 - January 1996): 124- 135.

Naimark, Michael. “Aspen the Verb: Musings on Heritage and Virtuality.” Presence, Special Issue on Virtual Heritage 15, no. 3 (2006): 330-335. 

Oculus VR LLC. “Oculus Rift”. Accessed August 2017. https://www.oculus.com/rift/

Sutherland, Ivan E. “The Ultimate Display.” In Proceedings of IFIP Congress, 506-508. 1965.

Sutherland, Ivan E. “A Head-Mounted Three Dimensional Display.” In Proceedings of the Fall Joint Computer Conference, AFIPS Conference Proceedings 33. 757-764. ACM, 1968.

Turner, Jeremy. “Myron Krueger Live. Interview by Jeremy Turner.“ CTheory a104 (2002).

Walker, John. “Through the Looking Glass.” In The Art of Human-Computer Interface Design, edited by Brenda Laurel, 439-447. Reading, MA: Addison-Wesley, 1999.

Wallach, Hans and D.N. O'Connell. “The kinetic depth effect.” Journal of Experimental Psychology 45 (1953): 205-217.

Witmer, Bob G., and Michael J. Singer. “Measuring Presence in Virtual Environments: A Presence Questionnaire” Presence: Teleoperators and Virtual Environments 7, no. 3 (1998): 225-240.

Woletz, Julie. Human-Computer Interaction. Kulturanthropologische Perspektiven auf Interfaces. Darmstadt: Büchner, 2016.

Zimmermann, Thomas, G., Jaron Lanier, Chuck Blanchard, Steve Bryson, and Young Harvill. “A Hand Gesture Interface Device.” In Proceedings Human Factors in Computer Systems and Graphics Interface, 189-192. ACM, 1987.