THERE IS NO INTERFACE (WITHOUT A USER). A CYBERNETIC PERSPECTIVE ON INTERACTION
“Interaction is seen as a one-way street, conveying a design model to a user, who is acting by that model either because they adapted to it, or because the model replicates their given structure. This is the cognitivist heritage of the HCI discourse responsible for the idea that interfaces can actually be designed.”
The interface in itself does not exist. This is not to say that any phenomenon must be perceived in order to exist, but rather that interfaces quite literally only come into being if they are used. They are effects of interaction and thus they are ultimately created by their users.
Of course, academic and professional disciplines like human-computer interaction and interaction design assume the opposite: namely that interfaces are designed (and exist) before they are used, possibly even creating their users. This article1 traces the development of this view, as well as offering an alternative to it that fundamentally understands any interface as “cybernetic interface.”2
GENEALOGIES OF INTERACTION
When during the late 1990s, the new millennium prompted countless retrospectives and outlooks, Terry Winograd contributed a chapter to a book about the next fifty years of computer science titled, “From Computing Machinery to Interaction Design.”3 Following an idea of evolutionary progress, this title described a goal directed development from the computing machinery of the past towards a future of interaction.
This trajectory constitutes the standard account of the history of interaction. Often the field is seen as following a teleological development of progress, during which computers became more and more interactive, and interaction became more intuitive, rich, and natural. This development is often explicated as a genealogy. Depending on the focus and goals of their narrators, there are genealogies of interaction focusing on a succession of hardware generations, interaction paradigms, theoretic frameworks, or visionaries pushing the field to the next level.
An early and paradigmatic account that focuses on hardware is John Walker’s genealogy of five “User Interaction Generations”4 published in the early 1990s. This account starts with the “plug boards” and “dedicated set-ups” of early computing.5 These were followed by the (in)famous era of batch processing – a time when programming meant punching holes into cards, handing batches of these cards to a mainframe operator and waiting for hours to be handed back a printed result.
It is only the third of these generations that was interactive. As to be expected from a proper generation, it was a child of the previous one and generated by it: When, this account goes, the algorithms allocating a mainframe’s computing time to several batch jobs got more and more advanced, it became clear that it would be possible to divide the computing time of a mainframe even further. Divided into small enough pieces, that follow each other in rapid succession, it would seem to several people that they would have exclusive control over the whole machine. This idea, named “time sharing,” did not divide a computer’s resources between several batch jobs but between several humans (re-defining these, as we will see below, as “users”) who could now engage in “conversational interactivity” with the machine.6
While the conversations of the time sharing generation happened as exchanges of written text, the fourth of Walker’s generation of interaction introduced graphical displays that concentrated textual commands into visual menus. The fifth and final generation then spawned the graphical user interfaces of personal computing that, in various iterations, keep accompanying us on our desktops, laptops, and phones until today.7
Paul Dourish’s “History of Interaction,”8 which is much more contemporary in style, follows a very similar path “from soldering to mouse.” Focusing on the mode of interaction instead of hardware generations, the first generation here was defined by “electrical interaction” (using cables, plugs, and the soldering iron) with Walker’s dedicated set-ups. This was followed by the era of “symbolic interaction” that was marked by the use of punch cards and batch processing – which were often programmed using the symbols of assembler languages instead of the raw zeros and ones of machine code. This generation, in turn, led to the “textual interaction” with the terminals of time sharing systems. “Graphical interaction” here again marks the final step in an evolution starting with machinery and ending with today’s interactive surfaces.
Apart from their implicit assumption that interactivity progressively increases, these, and most other histories of human-computer interaction (HCI) have one thing in common: They all assume an origin of interaction. While operating a computer during Walker’s second generation meant batch processing, the third generation introduced time sharing and with it, interaction. In Dourish’s terminology, this transition corresponds to the shift from “symbolic” (based on assembler language and punchcards) to “textual interaction” (based on conversational interactivity via command line). Only when computers, after time-sharing was introduced, started to react (seemingly) exclusively and directly to human input, did they become interactive: “Arguably, this is the origin of »interactive« computing.”9
Of course, history is not that simple. A closer look behind the narratives postulating a teleological development of interaction instead reveals contradictory and asynchronous developments, as well as chronological overlaps.10 Especially the origin of interactive computing itself can be described differently, for instance, by looking at the first interactive computer ever built – which happens to be one of the first computers at all. It is, as this look reveals, the very first generation of computing machinery that defined interactivity up to this day, including its problems.
INTERACTIVE COMPUTERS AS FEEDBACK MACHINES
During the 1940s, the MIT Servomechanisms Lab started to build a flight simulator. As the leading paradigm for automatic computation at that time was analog computing, the flight simulator was planned to be based on that: “a cockpit or control cabin connected, somehow, to an analog computer.”11 “Analog computing” in this context did not only imply calculating with analog (i.e. continuous) values, it rather implied an entirely different approach toward calculation: It relied on building electrical and mechanic systems, that, as analogues or analogies, could stand in for the systems they were built to simulate. Because building such analog computers entailed accurately following and amplifying changing physical signals, it largely depended on another development: the rise of the use of negative feedback as the de facto standard method for handling electro-mechanical systems. In fact, during the early twentieth century, negative feedback became so important in both control and communication engineering that both disciplines merged into one feedback based control theory – in a paradigm shift that yielded the era of “classical”12 control. This development, in turn, constituted the nucleus of what Norbert Wiener would later call cybernetics – a science of “control and communication in the animal and the machine”13 that would become thinkable mainly because the application of negative feedback and the associated mathematical formalisms seemed to be powerful enough to tackle any form of “behavior” – of living and non-living systems.14 Because feedback implies using the output of a system as its own input, the systems of cybernetics exhibited “circular causality”15 – a circular interdependence of input and output, entailing that agency within the system is distributed and cannot be pinned down to specific agents.
In building the simulator, moving axes and disks, and changing voltages and currents were used as analogies to the complex dynamics of a plane in flight. As these analogies constituted electro-mechanical motion, coupling them to the moving controls of a cockpit and the motion of their human operators was self-evident. However, during the development of this “Aircraft Stability and Control Analyzer” (ASCA)16 the first digital computers were under construction as well. The engineers at MIT observed this development and Jay Forrester, one of the project leads, became more and more interested in digital computation – so interested, in fact, he sacrificed the core of the project (building a flight simulator) to his new interest (building a digital computer): The development of the analog computer was halted, and a “general purpose, high speed”17 digital computer was built. As it was one of the first of its kind, the engineers building it were constantly “pushing the state of the art,”18 developing new building blocks for digital computation, such as memory mechanisms. Caught up in this task, however, they increasingly lost sight of the fact they were trying to build a flight simulator. This was especially problematic, as the ASCA’s cockpit still was the analog machine the project started with. Whereas coupling the motion and continuously changing electrical signals of an analog computer to the analog instruments of a cockpit did not pose a categorical problem, this had changed with digital computing. The digital and discrete state changes of the new computer had to be translated into continuous motion of the instruments, while the reactions of the operators on these instruments, in turn, had to be translated into digital states.19 “These problems were not impossible, but neither did established solutions exist. The digital computer was too new,”20 one of the engineers in the project later wrote. In consequence, the project management acknowledged that it was not about building a flight simulator anymore and the cockpit of the ASCA was scrapped.21 The computer was renamed “Whirlwind”22 and became a general-purpose digital computer not usable for flight simulation anymore. As it thus became a computer without application, it later would be turned from flight simulation to air defense and become the foundation for SAGE, the “Semi-Automatic Ground Environment” air defense system – the largest computer built to date that was in use until 1983.23
What set Whirlwind apart from the other first-generation digital computers of its time was its heritage in analog computing and flight simulation: It was conceived as a machine that reacts to changes in an environment (the cockpit) by incorporating any change happening here into its calculating. In addition, it would have the results of these calculations directly, and in real-time affect the environment. In other words; it was a digital computer that was to function like the control systems of analog computing and cybernetics – as a digital computer that can react to its environment in real-time.
This is remarkable, given that theoretical computer science operates with a conception of “machine,” explicated as with the Turing machine, that does not know time or any reciprocal interaction between calculation and its environment. Only relatively recently did theoretical computer science start to acknowledge, that the actual computing machines we have been using from the very beginning had done something that goes beyond Turing’s definition of computation – by incorporating interaction with an environment.24
Whirlwind thus was a strange hybrid: A digital computer that also tried to be a cybernetic feedback system, in constant dialog with the environment it controlled. If we follow Winograd’s juxtaposition of computing machinery and interaction, it was both: a machine and interactive – a feedback machine.
INTERRUPTION AND COUPLING: A BLACK ART
Even after having scrapped the cockpit, Whirlwind was still a machine to be used by human operators in real-time and as such posed two problems: How to integrate real-time input from the environment into an ongoing digital computation, and how to couple the process of digital computation to the action and perception of human operators. The engineers of Whirlwind approached these novel (or even “too new”) problems pragmatically.
The fundamental problem of having the machine react to its environment was tackled introducing a basic technique into computer engineering whose heritage is alive until today: Whirlwind could interrupt what it was working on, turn to any new data that may have arrived in the meantime, integrate that data (by copying it into memory), and continue where it had left off.25 Coupling the machine to the environment thus became a function of interruption – which, as hardware interrupt, later became a core feature of any interactive computer.
The problem of coupling computation to human operation was instead approached by introducing what later would be defined as one of the teleological ends of the development of interactivity: Whirlwind produced graphical representations that could be touched. This was made possible when the engineers in the project coupled memory registers of the computer to the x/y-control of the magnetic fields of a cathode ray tube (CRT). Whirlwind could thus paint symbolic representations of data onto screen: “One of the things that I think we did first was to connect a visual display to a computer.”26 This great leap into our screen-based present happened with the pragmatic naturalness of something “I think we did first,” simply because all prerequisites for it were already in place: The second world war had established various modes of coupling (analog) radar data to CRTs. Project Whirlwind could build on this foundation and even use the leftover CRTs of the war.27 In addition, CRTs had already been coupled to digital computers: In the “Williams Tube,” the afterglow of the light painted onto a screen was used as a short-term memory device that was not meant to be looked at by humans, but nevertheless constituted computer control of light on a screen.28
In order to close the loop between representation and action, the images painted by Whirlwind onto its CRTs were accompanied by a device to touch them: a “light-gun” (figure 1). The device realized this by feeding back the computer’s visual output to its own interrupt: The “gun” was designed not to shoot but to pick up light. Pointed at a visual representation on screen, it would pick up the light emitted when the computer drew this very representation. If an operator now pressed a button, the computer was interrupted while drawing it. It thus “knew” which item was selected and could take this selection into account for further computation.29 Even the light gun, although pioneered here as an interaction device, had technically already been built before it became part of the configuration of interactive computing – as it was originally used to test the Williams Tube memory devices for errors.30
Coupling Whirlwind to people was thus both: the pragmatic problem-solving of engineers using parts and components at hand, and a revolutionary prototype for most interactivity to come. But while it offered the basic capability of having human action become part of an ongoing computation, it did not solve any problems of how exactly this setup should be used. Instead, computer science had unexpectedly introduced a new class of problems, as the representations and couplings it made possible now had to be designed. It became a field of design, a “black art” in which “engineering design,” “creative design,” and scientific methods came (and still come) together.31
DOWNLOAD THE PDF
Suggested citation: Scherffig, Lasse (2018). “There is no Interface (without a User). A cybernetic Perspective on Interaction.” In: Interface Critique Journal Vol.1. Eds. Florian Hadler, Alice Soiné, Daniel Irrgang. DOI: 10.11588/ic.2018.0.44739
This article is released under a Creative Commons license (CC BY 4.0).
Lasse Scherffig is the Art and Technology Department Chair at San Francisco Art Institute, where he teaches since 2015. He previously taught at Bauhaus University Weimar and Academy of Media Arts Cologne. His work has been published and exhibited internationally.
Scherffig, Lasse. “Feedbackmaschinen. Kybernetik und Interaktion.” PhD thesis, KHM Köln, 2017.
Scherffig, Lasse. "Moving into View: Enacting Virtual Reality.“ Mediatropes 6, no. 1 (2016): 1-29.
(IN)HUMAN FACTORS: THE USER AS NEW HUMAN
In spite of Whirlwind, the narratives of the progressive incline of interactivity are not plainly wrong. Although interactive computing existed before time-sharing,32 MIT’s Whirlwind was a singular development and most of computer science for a long time stuck to building machines running algorithms that produce answers without being interrupted. Important early developments, such as Ivan Sutherland’s Sketchpad33 and especially Douglas Engelbart’s NLS, were running against this mainstream that was so dominant it took the field until the 1980s to acknowledge “interaction” as an independent area of inquiry. One of the first books carrying human-computer interaction (HCI) in its title was “The Psychology of Human-Computer Interaction”34 by Stuart Card, Thomas Moran, and Allen Newell. The role of the latter in establishing HCI is remarkable, as he serves as a link back to the first interactive computer as well as pointing towards the future of the field.
Early in his career, Newell worked at RAND’s Systems Research Laboratory. Here, he was in charge of training the operators of the SAGE system – and thus the first professional operators of interactive computers.35 This work was conducted together with Herbert Simon, with whom Newell would continue working on a number of subsequent projects. While building a training environment for the SAGE operators, Newell used computer modeling to simulate the input into the training system, consisting of human operators and simulated computer consoles. His simulation created sequences of “radar blips,” as they would have shown up on the real screens of the SAGE air defense system. The realization that computers could do something like this, and thus “more than arithmetic”36 would prove highly influential for Newell and Simon. The fact that in training these computer operators, computer modeled input data shown on computer screens37 would be perceived and interpreted by human observers, led Newell and Simon to the far-reaching conclusion that all participants of this system were essentially involved in the same task: processing information. Just as Newell’s digital simulation processed information in order to produce the fake radar blips, the human operators looked at these blips and perceived them as information to be processed and acted upon. In other words: “Within the simulated training environment, Newell came to view the human operators too as »information processing systems« (IPS), who processed symbols just like his program »processed« the symbols of simulated radar blips.”38
This is the crucial outcome of the training for the first interactive computers. Subsequently, Newell and Simon authored a number of papers that took this idea further, developing an understanding of human thinking that was driven by the verdict that it is a form of the symbolic information processing exhibited by computers. This culminated with the “Physical Symbol System Hypothesis,” declaring intelligence to be a feature of all forms of physical systems that are able to manipulate symbols – be it human or machine.39 This argument, at the time, was part of the development of a new scientific field of studying the human mind that, at least for a long time, understood thinking as rule-based information processing: cognitive science.40 The field from the very beginning “subsumes various computational theories of mental phenomena. Their computational nature is what unifies the multiple disciplines in the field and may count for much of its success in recent years.”41 In this sense, the human trained to perform in front of the computer became the model for the thinking human in general – a human acting as a computer.
This is what Newell brought back to working with interaction: He proposed to XEROX PARC an “Applied Information-processing Psychology Project (AIP)”42 that promised to apply cognitive science to the black art of designing interaction. The project started in 1974, led by Card and Moran, who were consulted by Newell. One of its results was the publication of “The Psychology of Human-Computer Interaction” by the three.
The center of this project was not longer the computer operator. Instead, it was the “user” of the computer interface. Card, Moran, and Newell stated: “But the user is not an operator. He does not operate the computer, he communicates with it to accomplish a task.”43 This attribution of agency to the computer (as an equal partner in communication) probably followed from the nature of the interactive computer as feedback machine that exhibits circular causality between machine and (human) environment. For the authors, however, the relationship of user and computer was defined solely by the postulated equivalence of all information processing systems.
In proposing the project to XEROX, Newell suggested to marry the empirical methods of human factors with the formal (and computational) models of cognitive science, creating “a technical understanding of the user himself and of the nature of human-computer interaction.”44 This would be a “science of the user rooted in cognitive theory.”45
In doing so, he seemed to be aware that this user was not a given, but something that was created by the systems being used – after all, it was a training environment for early computer users that gave rise to the idea of the human as information processing system. In the memo proposing the AIP to XEROX he thus wrote: “There is emerging a psychology of cognitive behavior that will permit calculation of behavior in new situations and with new humans…”46 Since this user was to be subject to the technical understanding provided by computational theories of mental phenomena, what emerged here was a view of the human being using the computer, as a computer.
The human factors of human-computer interaction, and human- or user-centered design thus become readable as the “inhuman factors”47 of thinking humans as machines – and making them act accordingly. The training required to become the new human that an interface demands, in this sense, can be seen as a “subtle enslavement”, and a “total, unavowed disqualification of the human in favor of the definitive instrumental conditioning of the individual.”48
COGNITIVE ENGINEERING VERSUS CONCRETE THINKING
It is this convergence of computer and cognitive science that served as the “origin myth” of human-computer interaction.49 The field did, for a long time, embrace cognitive science and its methods, effectively becoming a form of “cognitive engineering” as Donald Norman defined it in a seminal paper: “neither Cognitive Psychology, nor Cognitive Science, nor Human Factors. It is a type of applied Cognitive Science, trying to apply what is known from science to the design and construction of machines.”50
Being based on the cognitive science idea of what a human is, cognitive engineering was seen as a form of “user-centered” design. At the center of this idea stands a juxtaposition of the mental and the physical. Interaction, the argument goes, is an act of mediating between a user’s mental goals and the physical states of a system. This mediation happens in a loop of “execution” and “evaluation,” while execution is based on action sequences a user formulates according to their goals.51 Formulating these action sequences is possible because users possess a “mental model” of how they assume a system functions.52
The task of the interface designer as cognitive engineer now is to make sure that this mental model is correct – so that an action sequence will lead to the expected and intended results. They must bridge the gulf between execution and evaluation.53 Creating an interface thus becomes an act of communication where a designer’s “design model”54 must be communicated in a way yielding the appropriate mental model. In terms of information processing, this means that by its design a system must provide the information that, once perceived and processed, leads to the appropriate actions that fulfill a given goal.
According to Norman, there are two ways of achieving this: “(M)ove the system closer to the user; move the user closer to the system.”55 Of course, user-centered design wants to move the system closer to the user, by creating systems whose physical states behave in an “intuitive” or “natural” way, close to the mental intentions of their users. This, of course, implies that the latter can be formulated in terms of the former. A user’s non-physical goals and intentions must be translatable into physical actions and system states, thus reproducing the assumption that computer users ultimately can be understood on the same ground as the computers they use.
The relationship between the psychological states of a user and the physical states of a system has been described as “directness.”56 This term entered HCI discourse when Ben Shneiderman in 1983 was puzzled by “[c]ertain interactive systems” which “generate glowing enthusiasm among users.”57 What set these systems apart was the interactivity already introduced by Whirlwind: “(D)irect manipulation” of graphical representations without the need to type text – the origin (as constructed here) of interaction thus once more got reinterpreted as a milestone of the progressive incline of the field. But as opposed to the pragmatic engineering behind Whirlwind’s early interfaces, Shneiderman’s discussion of direct manipulation followed a cognitivist pattern, having a clear idea of the human user as rational problem solver in mind: Shneiderman reproduced Norman’s idea of interaction as psychophysical mediation by identifying a “problem domain” of “semantic” intentions and a “program domain” of “syntactic” manipulations at the interface. Direct manipulation, he argued, enables users to interact directly with the objects of the problem domain – by, for instance, enabling a writer to directly interact with paragraphs of text, instead of having to deal with the commands meant to manipulate these paragraphs. Direct manipulation would hence be a (or maybe the) realization of Norman's “move the system closer to the user” by minimizing the distance of the problem and program domain.
Not surprisingly, Norman himself later joined the discussion, expanding Shneiderman’s work in cooperation with James Hollan and Edwin Hutchins.58 This argument started with the assertion that, “[w]e see promise in the notion of direct manipulation, but as of yet we see no explanation of it.”59
Trying to formulate this explanation as a full-fledged “cognitive account”60 of direct manipulation, they reformulate Shneiderman’s distance of syntax and semantics as an “information processing distance” between human intentions and machine states61 – a distance that direct manipulation is minimizing. These interfaces, in this view, are easier to use because what we want do with them corresponds to the way it is done.
This, however, may not be enough to explain the “glowing enthusiasm” described by Shneiderman. Instead, the authors acknowledged that direct manipulation seems to entail an experiential component that can not be explained by information processing alone. It features a feeling of “engagement,”62 that is hard to come by: „Although we believe this feeling of direct engagement to be of critical importance, in fact, we know little about the actual requirements for producing it.“63 Referring to Brenda Laurel’s work that applied Aristotelian poetics to HCI, they concluded that a feeling of “first-personness”64 must be responsible for the feeling of engagement. For Laurel, this feeling was based on the interplay of user and interface, as „[a]n interface [...] is literally co-created by its human user every time it is used.“65
Direct manipulation hence seems to contain a playful component and a residue of the non-rational. It is not about a cognitive distance between mental intention and physical representation and action alone, it also is about a subjective experience that is created through the cyclic dependence of user action and machine response. This non-rational (or non-cognitivist) residue, however, seemed to deeply bother Hutchins, Hollan and Norman, who stated:
“On the surface, the fundamental idea of a direct manipulation interface to a task flies in the face of two thousand years of development of abstract formalisms as a means of understanding and controlling the world. Until very recently, the use of computers has been an activity squarely
in that tradition. So the exterior of direct manipulation, providing as it does for the direct control of a specific task world, seems somehow atavistic, a return to concrete thinking.”66
This return to concrete thinking subsequently became even more prominent when “tangible user interfaces” and other forms of non-screen-based interactivity emerged. When, for instance, physical objects in research projects at MIT and elsewhere became phicons67 – physical icons that represent data and computational processes – researchers at XEROX PARC coined the term, “interfaces for really direct manipulation.”68 If tangible user interfaces use real-world objects as representations of computation, the hope was, they would feel ultimately natural and the information processing distance would be reduced to zero.
This, however, makes two things apparent: First, if the mouse and screen felt natural during the 1980s and tangible user interfaces felt more (or really) natural during the early 2000s, naturalness itself must be understood as a fluid category depending on what feels natural for the “new human” of each era. Interfaces like the touch screen in this light must be understood as being products of a naturalization creating the very human for which they feel natural. Second, the whole discussion of tangible interaction neglects the fact that all interfaces in one form or another have been tangible: We have never “directly” manipulated a paragraph of text but always had to deal with pens and marks on paper, keyboard and screen, fingers on a touchscreen. The atavistic syntax of executing manual actions always existed and was always different from any semantic goal or intention.
What, instead, differentiates tangible user interfaces from graphical user interfaces and these from the command line is something much more profane: It is the simple spatio-temporal distance of human action and computer reaction as well as their perceived similarity. The incremental progress of interaction, postulated by the genealogies of interactivity, is another clue suggesting that what is really interesting about interactivity is the closure of the gap in space and time between human and computer action. In particular, the theoretical reflection on non-screen based interfaces had understood this at an early stage. Already in 2000, Michel Beaudouin-Lafon has concluded that what is really important about the experience of computer interfaces is the “spatial and temporal offset,” the “ratio between the number of degrees of freedom” and the “similarity between the physical actions of the users on the instrument and the response of the object.”69
No matter if we hail the natural or intuitive interface as bridging the gap between user and system, or if we condemn the interface as a form of conditioning that ultimately naturalizes a non-human mode of action and perception, we presuppose the interface as the agent of this process. If interfaces are seen as forming the new human after their own image (by moving them closer to the system) or if they supposedly assist a given human by modeling their non-physical goals or semantics, they are assumed to be sources of information that are perceived, processed and acted upon. Interaction is seen as a one-way street, conveying a design model to a user, who is acting by that model either because they adapted to it, or because the model replicates their given structure. This is the cognitivist heritage of the HCI discourse responsible for the idea that interfaces can actually be designed.
When, however, the engineers in project Whirlwind coupled digital computation to symbolic representation and human action back to computation, they not only wrapped its human operators in the feedback loop of the circular systems of cybernetics: They also created a setting in which the representations would be wrapped in a loop of human action and perception.
The motion on a computer screen is not real motion but a cinema-like sequence of still images, which psychology denotes as “apparent motion.” Apparent motion has been a subject of experimental psychology since the cinematograph and cinema rendered it ubiquitous, providing the experimental systems for studying it and for using it as a tool to study perception in general.70 One seminal early work was Max Wertheimer’s experimental studies of the perception of movement,71 which today is seen as one of the founding texts of Gestalt psychology.72 While Werheimer pioneered the experimental investigation of apparent motion, Gestalt psychologists like Paul Linke and later Paul von Schiller studied the phenomenon with a focus on a fringe case of it: The perception of “ambiguous motion,” which is present whenever the direction of an apparent motion stimulus can not be decided objectively (figure 2). Such stimuli are interesting because they afford more than one possible perceptual interpretation, while subjectively only one direction of motion is perceived at a time. They thus reveal how the sensory system is treating stimuli in deciding how they are to be perceived, making them “invaluable tools for the study of the neural basis of visual awareness, because they allow us to distinguish neural responses that correlate with basic sensory features from those that correlate with perception.”73
Trying to establish the “laws” of how visual perception deals with ambiguous motion, von Schiller preempted a number of results of contemporary experimental psychology about how form, color and initial position of ambiguous motion stimuli influence the way we perceive them.74 During his experiments, however, he did make one especially remarkable observation: His subjects were able to actively control the perceived direction of motion if they moved their heads or hands. This was a case, he remarked in a footnote only, where motor activity shapes the Gestalt of optical perception.75
About sixty years later, this little noted observation was confirmed by modern psychology: In a brief article in the journal “Investigative Ophthalmology & Visual Science,” Ishimura and Shimojo report that “Voluntary Action Captures Visual Motion.”76 In a series of studies,77 they had shown that hand movements capture (as in: influence) the way we perceive visual motion. Their experiments, of course, were conducted with a computer, coupling motion on a physical interface to visual representation on screen. A few years later, Andreas Wohlschläger continued this research, analyzing more features of the effect.78 Later, it has also been shown that this does not only hold for the relation of hand and eye, but that auditory and tactile perception can be influenced by motion of the hands, eyes, head or feet as well.79
What these studies showed is not only that our motor actions directly influence what we perceive. They also showed that this influence is stronger, the closer action and perception happen in space and time and the more their features (like their spatial orientation) align. The strongest influence was measured when manual motion and computer reaction happened simultaneously and overlapped each other. More importantly even, they also showed that the effect is even present if a motor action merely is planned, but not executed. It could also be changed through training: After using a mouse whose control of the cursor on screen was inverted such that a motion to the right yielded an on-screen motion to the left, subjects exhibited a corresponding change in action capture, so that a motion to the left influenced ambiguous motion to the right. Apparently, the effect takes into account the expected results an action has. Action capture thus demonstrates that the more an action is related to the reaction it provokes (in terms of spatio-temporal distance, orientation, and its expected results), the more it influences perception of that action.
From the point of view of physiology, it has long been known that the neural activity causing motion, which originates in the motor cortex of the brain, is not only communicated to the muscles executing motion, but also to sensory areas. Motor signals are accompanied by an “efference copy”80 or “corollary discharge”81 that relays them to parts of the brain responsible for perception. This is thought to be part of a process in which the expected results of an action are compared to what actually is perceived, in a feedback loop resembling the one of cybernetic control systems.82
Motor activity thus is directly inscribed into the perception of its results. The reactions we expect an activity to have is driving its perception, based on their spatio-temporal relation and perceived similarity.
FACTORING THE HUMAN BACK IN: CYBERNETIC INTERACTIONS
Years after Hutchins worked with Norman on a cognitive account of direct manipulation, he diverged from classical cognitive science. As if he could not longer ignore the “concrete thinking” conducted by the hands on the physical interface, he turned to “embodied” and “enactive” cognitive science, trying to understand thinking as a process involving bodies engaged in the culturally structured world surrounding them.83 Analyzing the reasoning and actions of humans performing nautical navigation, he observed that “[t]he traditional »action-neutral« descriptions of mental representations seem almost comically impoverished alongside the richness of the moment-by-moment engagement of an experienced body with a culturally constituted world.”84 He thus shifted his focus on how the “actions of the hands”85 drive insight and even constitute the physical symbols or representations we are working with: “To apprehend a material pattern as a representation of something is to engage in specific culturally shaped perceptual processes.”86
This view corresponded to the way enactive cognitive science understands how our actions are ultimately responsible for the perceived features of objects, such as their shape. As Kevin O’Regan and Alva Noë wrote in a seminal text on how an action-centric view of cognitive science could look like: “The idea we wish to suggest here is that the visual quality of shape is precisely the set of all potential distortions that the shape undergoes when it is moved relative to us, or when we move relative to it.”87
This also holds for the interface. Although very few research has been devoted to studying how an interface is perceived while it is used, there is a remarkable PhD thesis by Dag Svanæs titled “Understanding Interactivity.”88 In explicit tradition of Gestalt psychology and its qualitative methods, Svanæs, as part of this thesis, conducted experiments in which subjects (or users) interacted with abstract minimalist systems of black and white squares called “Square World.”89
Users interacted with these worlds using a mouse, clicking on the squares and observing the subsequent changes in the world on screen. Governed by more or less complex state-transition-diagrams, the squares in the world changed their color (from black to white or back, see figure 3). Svanæs recorded the user actions while correlating these with their verbal descriptions of what, according to them, was happening.
Among his observations was an interesting shift in his users’ perceived “locus of agency,” which moved from describing actions in the Square World (“it gets colored”) towards locating oneself as acting in it (“I turn it on”).90 He understood these, using Merleau-Ponty’s terminology, as a gradual extension of the users body space by which the interface became incorporated. This, according to him, is direct manipulation: An extension of the perceived locus of agency into an interface, which would explain why interaction can feel direct although it is mediated by physical interfaces (like the mouse) that are distant from their effects (apparent motion on screen).
For his subjects, with this incorporation came an “understanding” of the Square Worlds that grew from the sequence of their interactions. Users clicked, observed, clicked again and at some point would formulate a conceptualization of what they were dealing with, by saying, for instance, “It is a switch.”91 Notably, the switch was not there from the beginning. There was no symbolic representation of a switch to be seen and interpreted as such. Instead, it appeared to be encapsulated into the action sequence:
“When the subjects said »It is a switch«, they did not come to this conclusion from a formal analysis of the State Transition Diagram of the example. Nor did they conclude it from the visual appearance of the square, as the squares all looked the same. The switch behavior slowly emerged from the interaction as the square repeated its response to the subject’s actions.”92
By physically engaging in the “syntax” of moving a mouse and pressing its buttons, the subjects established the “semantics” of the Square World by literally enacting it: co-creating the perceived objects in the world through their actions. As these objects existed only through being used, Svanæs described them as having Gestalt properties, naming them “Interaction Gestalts.”94 In Svanæs's words: “At the perceptual level closest to the computer are the rapid mouse movements and button clicks that the subjects did when they explored new examples. At the cognitive level above emerge the Interaction Gestalts that result from the interactions.”95
In light of this, direct manipulation, in all its instances from Whirlwind’s light-gun to mouse and keyboard, tangible user interfaces and today’s ubiquitous touch-screens, can be seen as not the reduction of a psychophysical distance of material syntax and mental semantics. It can rather be understood as an interplay of syntax and semantics, perceptual level and cognitive level that together create the Gestalt of the interface.
As interfaces exhibit different levels of interactivity (few would disagree that a touchscreen somehow feels more interactive than a keyboard), they also exhibit different degrees of what really makes their interactivity direct: “spatial and temporal offset,” the “ratio between the number of degrees of freedom” and the “similarity between the physical actions of the users on the instrument and the response of the object.”96 These factors supporting directness of interaction turn out to be the same factors supporting the influence of our actions on the perception of their results. Interaction thus seems to depend on how closely action and perception are fused by an interface, while this fusing is subject to their physical qualities and our acquired expectations. Wrapped in their reciprocal dependence, they create the Gestalt of that very interface.
It is the circular causality of cybernetic feedback, inherent to interactive computing since the very beginning, that encapsulates user and interface in a loop within which objects emerge through the process of acting with them. No matter how supposedly natural the latest interface might be, in the very moment when computers became feedback machines they set the stage for creating naturalness and its user in the reciprocal interplay of action, computer reaction and perception.
Any button we touch on our phones and tablets is, just like the switch in Svanæs’ experiments, a button only because it is used as such. The interface in itself therefore only exists subjectively and is quite literally co-created, or enacted, every time it is used. While interaction design constantly creates new humans, it never has them or its interfaces fully under control. It may hence be time to start rethinking human-computer interaction as something that is, and always has been, fundamentally participatory.
Beaudouin-Lafon, Michel. “Instrumental Interaction: An Interaction Model for Designing Post-WlMP User Interfaces.” In Proceedings of CHI, 446-453. ACM, 2000.
Bennett, Stuart. A History of Control Engineering 1930–1955. Hitchin: Peter Peregrinus Ltd., 1993.
Card, Stuart K., and Thomas P. Moran. “User Technology: From Pointing to Pondering.” In Proceedings of the ACM Conference on The History of Personal Workstations, 183-198. ACM, 1986.
Card, Stuart K., Thomas P. Moran, and Allen Newell. The Psychology of Human-Computer Interaction. Hillsdale, NJ/London: Lawrence Erlbaum Associates, 1983.
Carter, Olivia, Talia Konkle, Qi Wang, Vincent Hayward, and Christopher Moore. “Tactile Rivalry Demonstrated with an Ambiguous Apparent-Motion Quartet.” Current Biology 18 (2008): 1050–1054.
Chapman, Robert L., John L. Kennedy, Allen Newell, and William C. Biel. “The Systems Research Laboratory’s Air Defense Experiments.” Management Science 5, no. 3 (1959): 250-269.
Dourish, Paul. Where the action is. The foundations of embodied interaction. Cambridge, MA: MIT Press, 2001.
Dunne, Anthony. Hertzian tales. Electronic products, aesthetic experience, and critical design. Cambridge, MA: MIT Press, 2008.
Eagleman, David M. “Visual illusions and neurobiology.” Nature Reviews Neuroscience 2 (2001): 920-926.
Everett, Robert. “Whirlwind.” In A History of Computing in the Twentieth Century, edited by J. Howlett, Gian Carlo Rota, and Nicholas Metropolis, 365-384. Orlando: Academic Press, 1980.
Fishkin, Kenneth P., Anuj Gujar, Beverly L. Harrison, Thomas P. Moran, and Roy Want. “Embodied user interfaces for really direct manipulation.” Communications of the ACM 43, no. 9 (2000): 74–80.
Friedewald, Michael. Der Computer als Werkzeug und Medium: Die geistigen und technischen Wurzeln des Personal Computers. Berlin/Diepholz: Verlag für Geschichte der Naturwissenschaften und der Technik, 1999.
Gardner, Howard. The Mind’s New Science. A History of the Cognitive Revolution. New York: Basic Books, 1985.
Hellige, Hans Dieter. „Krisen- und Innovationsphasen in der Mensch-Computer-Interaktion.“ In Mensch-Computer-Interface. Zur Geschichte und Zukunft der Computerbedienung, edited by Hans Dieter Hellige, 11-92. Bielefeld: transcript, 2008.
Hoffmann, Christoph. „φ-Phänomen Film. Der Kinematograph als Ereignis experimenteller Psychologie um 1900.“ In Die Adresse des Mediums, edited by Stefan Andriopoulos, Gabriele Schabacher, and Eckhard Schumacher, 236-252. Cologne: DuMont, 2001.
Hutchins, Edwin L., James D. Hollan, and Donald A. Norman. “Direct Manipulation Interfaces.” Human-Computer Interaction 1 (1985): 311–338.
Hutchins, Edwin. “Enaction, Imagination, and Insight.” In Enaction: Towards a New Paradigm for Cognitive Science, edited by John Robert Stewart, Olivier Gapenne, and Ezequiel A. Di Paolo, 425-450. Cambridge, MA: MIT Press, 2010.
Ishimura, G. “Visuomotor factors for action capture.” Investigative Ophthalmology and Visual Science (Supplement) 36 (1995): S357.
Ishimura, G., and S. Shimojo. “Voluntary action captures visual motion.” Investigative Ophthalmology and Visual Science (Supplement) 35 (1994): 1275.
Laurel, Brenda K. “Interface as mimesis.” In User-Centered System Design: New Perspectives on Human-Computer Interaction, edited by Donald A. Norman and Stephen Draper, 67-85. Hillsdale, NJ: Lawrence Erlbaum Associates, 1986.
Miller, George A. “The cognitive revolution: a historical perspective.” Trends in Cognitive Sciences 7, no. 3 (2003): 141-144.
MIT. Whirlwind I: A high-speed Electronic Digital Computer. Promotional brochure. Cambridge, MA: MIT, 1951.
Newell, Allen, and Herbert A. Simon. “Computer science as empirical inquiry: symbols and search.” Communications of the ACM 19, no. 3 (1976): 113-126.
Noble, Douglas D. “Mental Materiel. The militarization of learning and intelligence in US education.” In Cyborg Worlds. The Military Information Society, edited by Les Levidow and Kevin Robins, 13-41. London: Free Association Books, 1989.
Norman, Donald A. “Cognitive Engineering.” In User-Centered System Design: New Perspectives on Human-Computer Interaction, edited by Donald A. Norman and Stephen Draper, 31-61. Hillsdale, NJ: Lawrence Erlbaum Associates, 1986.
O’Regan, Kevin, and Alva Noë. “A Sensorimotor Account of Vision and Visual Consciousness.” Behavioral and Brain Sciences 24 (2001): 939-1031.
Pias, Claus. „Computer Spiel Welten.“ Dissertation, Bauhaus-Universität Weimar, 2000.
Pold, Søren Bro. “Interface Perception: The Cybernetic Mentality and Its Critics: Ubermorgen.com.” In Interface Criticism: Aesthetics Beyond Buttons, edited by Christian Ulrik Andersen and Søren Bro Pold, 91-113. Aarhus: Aarhus University Press, 2011.
Redmond, Kent C., and Thomas M. Smith. Project Whirlwind: the history of a pioneer computer. Bedford, MA: Digital Press, 1980.
Repp, Bruno H., and Günther Knoblich. “Action Can Affect Auditory Perception.” Psychological Science 18, no. 1 (2007): 6–7.
Rosenblueth, Arturo, Norbert Wiener, and Julian Bigelow. “Behavior, Purpose and Teleology.” Philosophy of Science 10 (1943): 18–24.
Scherffig, Lasse. „Feedbackmaschinen. Kybernetik und Interaktion.“ Dissertation, Kunsthochschule für Medien Köln, 2017.
Scherffig, Lasse. “Moving into View: Enacting Virtual Reality.” Mediatropes 6, no. 1 (2016): 1-29.
Schumann, Frank. “Embodied Cognitive Science: Is it Part of Cognitive Science? Analysis within a Philosophy of Science Background.” PICS. Publications of the Institute of Cognitive Science 3 (2004).
Shneiderman, Ben. “Direct Manipulation: A Step Beyond Programming Languages.” In The New Media Reader, edited by Noah Wardrip-Fruin and Nick Montfort, 486-498. New York, NY/London: W. W. Norton & Company, 2001.
Simon, Herbert A. “Allen Newell. 1927-1992.” National Academy of Sciences Biographical Memoirs (1997): 139-173.
Sperry, Roger W. “Neural Basis of the Spontaneous Optokinetic Response Produced by Visual Inversion.” Journal of Comparative and Physiological Psychology 43, no. 6 (1950): 482–489.
Steinman, Robert M., Zygmunt Pizlo, and Filip J. Pizlo. “Phi is not beta, and why Wertheimer’s discovery launched the Gestalt revolution.” Vision Research 40 (2000): 2257–2264.
Svanæs, Dag. “Understanding Interactivity: Steps to a Phenomenology of Human-Computer Interaction.” Dissertation, Norges Teknisk-Naturvitenskapelige Universitet Trondheim, 2000.
Ullmer, Brygg, Hiroshi Ishii, and Dylan Glas. “mediaBlocks: Physical Containers, Transports, and Controls for Online Media.” In Proceedings of SIGGRAPH, 379-386. ACM, 1998.
von Foerster, Heinz. “Cybernetics of Epistemology.” In Understanding Understanding: Essays on Cybernetics and Cognition. Heinz von Foerster, 229–246. New York: Springer, 2003.
von Holst, Erich and Horst Mittelstaedt. “The Principle of Reafference: Interactions Between the Central Nervous System and the Peripheral Organs.” In Perceptual Processing: Stimulus Equivalence and Pattern Recognition, edited by Peter C. Dodwell, 464-476. New York: Appleton-Century-Crofts, 1971.
von Schiller, Paul. „Stroboskopische Alternativversuche.“ Psychologische Forschung 17 (1933): 179-214.
Walker, John. “Through the looking glass.” In The Art of Human-Computer Interface Design, edited by Brenda Laurel, 439-448. Redding, MA: Addison-Wesley, 1990.
Wegner, Peter. “Why interaction is more powerful than algorithms.” Communications of the ACM 40, no. 5 (1997): 80-91.
Wertheimer, Max. „Experimentelle Studien über das Sehen von Bewegung.“ Zeitschrift für Psychologie und Physiologie der Sinnesorgane 61 (1912): 161-265.
Wiener, Norbert. Cybernetics: or Control and Communication in the Animal and the Machine. Cambridge, MA: MIT Press, 1961.
Wieser, C. R. Cape Cod System and Demonstration, Technical report. Cambridge, MA: Lincoln Laboratory – Division 6, 1953.
Winograd, Terry. “From Computing Machinery to Interaction Design.” In Beyond Calculation: The Next Fifty Years of Computing, edited by Peter Denning and Robert Metcalfe, 149-162. Berlin/New York: Springer, 1997.
Wohlschläger, Andreas. “Visual motion priming by invisible actions.” Vision Research 40 (2000): 925–930.
Yabe, Yoshiko and Gentaro Taga. “Treadmill locomotion captures visual perception of apparent motion.” Experimental Brain Research 191, no. 4 (2008): 487–494.
1 A German language discussion of similar questions, that is much broader in scope, can be found in Lasse Scherffig, „Feedbackmaschinen. Kybernetik und Interaktion“ (Dissertation, Kunsthochschule für Medien Köln, 2017).
2 Søren Bro Pold, “Interface Perception: The Cybernetic Mentality and Its Critics: Ubermorgen.com,” in Interface Criticism: Aesthetics Beyond Buttons, ed. Christian Ulrik Andersen and Søren Bro Pold (Aarhus: Aarhus University Press, 2011), 91.
3 Terry Winograd, “From Computing Machinery to Interaction Design,” in Beyond Calculation: The Next Fifty Years of Computing, ed. Peter Denning and Robert Metcalfe (Berlin & New York: Springer, 1997).
4 John Walker, “Through the looking glass,” in The Art of Human-Computer Interface Design, ed. Brenda Laurel (Redding, MA: Addison-Wesley, 1990), 439.
5 Ibid. 439-440.
6 Ibid. 441.
7 Ibid. 441-442.
8 Paul Dourish, Where the action is. The foundations of embodied interaction (Cambridge, MA: MIT Press, 2001), 1-17.
9 Ibid. 10.
10 Hans Dieter Hellige, „Krisen- und Innovationsphasen in der Mensch-Computer-Interaktion,“ in Mensch-Computer-Interface. Zur Geschichte und Zukunft der Computerbedienung, ed. Hans Dieter Hellige (Bielefeld: transcript, 2008), 15-20.
11 Kent C. Redmond and Thomas M. Smith, Project Whirlwind: the history of a pioneer computer (Bedford, MA: Digital Press, 1980), 32.
12 Stuart Bennett, A History of Control Engineering 1930–1955 (Hitchin: Peter Peregrinus Ltd., 1993), 17.
13 Norbert Wiener, Cybernetics or: Control and Communication in the Animal and the Machine (Cambridge, MA: MIT Press, 1961).
14 Arturo Rosenblueth, Norbert Wiener, and Julian Bigelow, “Behavior, Purpose and Teleology,” Philosophy of Science 10 (1943): 18–24.
15 For a detailed discussion, see Heinz von Foerster, “Cybernetics of Epistemology,” in von Foerster, Understanding Understanding: Essays on Cybernetics and Cognition (New York: Springer, 2003): 229–246.
16 Redmond and Smith, Project Whirlwind, 51.
17 MIT, Whirlwind I: A high-speed Electronic Digital Computer, promotional brochure (Cambridge, MA: MIT, 195), 6.
18 Robert Everett, “Whirlwind,” in A History of Computing in the Twentieth Century, ed. J. Howlett, Gian Carlo Rota, and Nicholas Metropolis (Orlando: Academic Press, 1980), 365.
19 Redmond and Smith, Project Whirlwind, 49.
20 Ibid. 49. Emphasis by author.
21 Ibid. 60.
22 Ibid. 43-44.
23 Ibid. 206.
24 Peter Wegner, “Why interaction is more powerful than algorithms,” Communications of the ACM 40, no. 5 (1997): 83.
25 Everett, “Whirlwind,” 377.
26 Ibid. 375.
27 Ibid. 379.
28 Claus Pias, „Computer Spiel Welten“ (Dissertation, Bauhaus-Universität Weimar, 2000), 55-56.
29 C. R. Wieser, Cape Cod System and Demonstration, Technical Report, (Cambridge, MA: Lincoln Laboratory – Division 6, 1953), 2.
30 Michael Friedewald, Der Computer als Werkzeug und Medium: Die geistigen und technischen Wurzeln des Personal Computers (Berlin & Diepholz: Verlag für Geschichte der Naturwissenschaften und der Technik, 1999), 103.
31 Hellige, „Krisen- und Innovationsphasen,“ 16.
32 In fact, “time-sharing” is an after-the-fact conceptualization of what was done in the project, as the term was first used by an engineer working on the already interactive SAGE system. See Friedewald, Der Computer als Werkzeug und Medium, 128.
33 Which was programmed on a TX-2 – a direct descendant of Whirlwind. See Friedewald, Der Computer als Werkzeug und Medium, 110-118.
34 Stuart K. Card, Thomas P. Moran, and Allen Newell, The Psychology of Human-Computer Interaction (Hillsdale, NJ & London: Lawrence Erlbaum Associates, 1983).
35 Douglas D. Noble, “Mental Materiel. The militarization of learning and intelligence in US education,” in Cyborg Worlds. The Military Information Society, ed. Les Levidow and Kevin Robins (London: Free Association Books, 1989), 19.
36 Herbert A. Simon, “Allen Newell. 1927-1992,” National Academy of Sciences Biographical Memoirs (1997): 146.
37 Which, ironically, in the training system were simulated by complex analog display machinery showing sequences of pre-rendered screens. See Robert L. Chapman, John L. Kennedy, Allen Newell, and William C. Biel, “The Systems Research Laboratory’s Air Defense Experiments,” Management Science 5, no. 3 (1959): 256-262.
38 Noble, “Mental Materiel,” 19.
39 Allen Newell and Herbert A. Simon, “Computer science as empirical inquiry: symbols and search,” Communications of the ACM 19, no. 3 (1976): 116.
40 Newell and Simon presented a Logic Theory Machine at the famous Symposium on Information Theory at MIT in 1956, which often is understood as the founding event of cognitive science. For this standard account of the history of the field see Howard Gardner, The Mind’s New Science. A History of the Cognitive Revolution, (New York: Basic Books, 1985), 28, and George A. Miller, “The cognitive revolution: a historical perspective,” Trends in Cognitive Sciences 7, no. 3 (2003): 141-144.
41 Frank Schumann, “Embodied Cognitive Science: Is it Part of Cognitive Science? Analysis within a Philosophy of Science Background,” PICS. Publications of the Institute of Cognitive Science 3 (2004): 12.
42 Stuart K. Card and Thomas P. Moran, “User Technology: From Pointing to Pondering,” in Proceedings of the ACM Conference on The History of Personal Workstations (ACM: 1986), 183.
43 Card, Moran, and Newell, The Psychology of Human-Computer Interaction, 7.
44 Ibid. 183.
46 As quoted in Card und Moran, “User Technology,” 183.
47 Anthony Dunne, Hertzian tales. Electronic products, aesthetic experience, and critical design (Cambridge, MA: MIT Press 2008), 21.
48 Paul Virilio as cited in Dunne, Hertzian tales, 21.
49 Dourish, Where the action is, 61.
50 Donald A. Norman, “Cognitive Engineering,” in User-Centered System Design: New Perspectives on Human-Computer Interaction, ed. Donald A. Norman and Stephen Draper (Hillsdale, NJ: Lawrence Erlbaum Associates, 1986), 31.
51 Norman, “Cognitive Engineering,” 41.
52 Ibid. 46.
53 Ibid. 38.
54 Ibid. 46.
55 Ibid. 43.
56 Ibid. 52.
57 Ben Shneiderman, “Direct Manipulation: A Step Beyond Programming Languages,” in The New Media Reader, ed. Noah Wardrip-Fruin and Nick Montfort (New York, NY & London: W. W. Norton & Company, 2001), 486.
58 Edwin L. Hutchins, James D. Hollan, and Donald A. Norman, “Direct Manipulation Interfaces,” Human-Computer Interaction 1 (1985): 311–338.
59 Ibid. 316.
61 Ibid. 311.
62 Ibid. 332.
63 Ibid. 332-333.
64 Ibid. 318. See also Brenda K. Laurel, “Interface as mimesis,” in User-Centered System Design: New Perspectives on Human-Computer Interaction, ed. Donald A. Norman and Stephen Draper (Hillsdale, NJ: Lawrence Erlbaum Associates, 1986), 67–85.
65 Laurel, “Interface as mimesis,” 73.
66 Hutchins, Hollan and Norman, “Direct Manipulation Interfaces,” 337.
67 Brygg Ullmer, Hiroshi Ishii, and Dylan Glas, “mediaBlocks: Physical Containers, Transports, and Controls for Online Media,” in Proceedings of SIGGRAPH (ACM: 1998), 379.
68 Kenneth P. Fishkin, Anuj Gujar, Beverly L. Harrison, Thomas P. Moran, and Roy Want, “Embodied user interfaces for really direct manipulation,” Communications of the ACM 43, no. 9 (2000): 74–80.
69 Michel Beaudouin-Lafon, “Instrumental Interaction: An Interaction Model for Designing Post-WlMP User Interfaces,” in Proceedings of CHI (ACM: 2000), 446–453.
70 Christoph Hoffmann, „φ-Phänomen Film. Der Kinematograph als Ereignis experimenteller Psychologie um 1900,“ in Die Adresse des Mediums, ed. Stefan Andriopoulos, Gabriele Schabacher, and Eckhard Schumacher (Cologne: DuMont, 2001), 236.
71 Max Wertheimer, „Experimentelle Studien über das Sehen von Bewegung,“ Zeitschrift für Psychologie und Physiologie der Sinnesorgane 61 (1912).
72 Robert M. Steinman, Zygmunt Pizlo, and Filip J. Pizlo, “Phi is not beta, and why Wertheimer’s discovery launched the Gestalt revolution,” Vision Research 40 (2000): 2257–2264.
73 David M. Eagleman, “Visual illusions and neurobiology,” Nature Reviews Neuroscience 2 (2001): 922.
74 Paul von Schiller, „Stroboskopische Alternativversuche,“ Psychologische Forschung 17 (1933): 180.
75 von Schiller, „Stroboskopische Alternativversuche,“ 196,
76 The note only covers one sixth of a page in the issue. G. Ishimura and S. Shimojo, “Voluntary action captures visual motion,” Investigative Ophthalmology and Visual Science (Supplement) 35 (1994): 1275.
77 Continued with G. Ishimura, “Visuomotor factors for action capture,” Investigative Ophthalmology and Visual Science (Supplement) 36 (1995): S357.
78 Andreas Wohlschläger, “Visual motion priming by invisible actions,” Vision Research 40 (2000): 925–930.
79 Bruno H. Repp and Günther Knoblich, “Action Can Affect Auditory Perception,” Psychological Science 18, no. 1 (2007): 6–7; Olivia Carter, Talia Konkle, Qi Wang, Vincent Hayward, and Christopher Moore, “Tactile Rivalry Demonstrated with an Ambiguous Apparent-Motion Quartet,” Current Biology 18 (2008): 1050–1054; Yoshiko Yabe and Gentaro Taga, “Treadmill locomotion captures visual perception of apparent motion,” Experimental Brain Research 191, no. 4 (2008): 487–494.
80 Erich von Holst and Horst Mittelstaedt, “The Principle of Reafference: Interactions Between the Central Nervous System and the Peripheral Organs,” in Perceptual Processing: Stimulus Equivalence and Pattern Recognition, ed. Peter C. Dodwell (New York: Appleton-Century-Crofts, 1971), 41.
81 Roger W. Sperry, “Neural Basis of the Spontaneous Optokinetic Response Produced by Visual Inversion,” Journal of Comparative and Physiological Psychology 43, no. 6 (1950): 482–489.
82 For a thorough discussion see Lasse Scherffig, “Moving into View: Enacting Virtual Reality,” Mediatropes 6, no. 1 (2016).
83 For his own introduction to “embodiment” and “enaction” see Edwin Hutchins, “Enaction, Imagination, and Insight,” in Enaction: Towards a New Paradigm for Cognitive Science, ed. John Robert Stewart, Olivier Gapenne, and Ezequiel A. Di Paolo (Cambridge, MA: MIT Press, 2010), 428.
84 Hutchins, “Enaction, Imagination, and Insight”, 445.
85 Ibid. 443.
86 Ibid. 429-430.
87 Kevin O’Regan and Alva Noë, “A Sensorimotor Account of Vision and Visual Consciousness,” Behavioral and Brain Sciences 24 (2001): 940.
88 Dag Svanæs, “Understanding Interactivity: Steps to a Phenomenology of Human-Computer Interaction” (Dissertation, Norges Teknisk-Naturvitenskapelige Universitet Trondheim, 2000).
89 Svanæs, “Understanding Interactivity,” 128.
90 Ibid. 159.
91 Ibid. 206.
93 Ibid. 147.
94 Ibid. 218.
95 Ibid. 206.
96 Beaudouin-Lafon, “Instrumental Interaction,” 446–453.