“Interaction is seen as a one-way street, conveying a design model to a user, who is acting by that model either because they adapted to it, or because the model replicates their given structure. This is the cognitivist heritage of the HCI discourse responsible for the idea that interfaces can actually be designed.”

The interface in itself does not exist. This is not to say that any phenomenon must be perceived in order to exist, but rather that interfaces quite literally only come into being if they are used. They are effects of interaction and thus they are ultimately created by their users.

Of course, academic and professional disciplines like human-computer interaction and interaction design assume the opposite: namely that interfaces are designed (and exist) before they are used, possibly even creating their users. This article¹ traces the development of this view, as well as offering an alternative to it that fundamentally understands any interface as “cybernetic interface.”²

GENEALOGIES OF INTERACTION

When during the late 1990s, the new millennium prompted countless retrospectives and outlooks, Terry Winograd contributed a chapter to a book about the next fifty years of computer science titled, “From Computing Machinery to Interaction Design.”³ Following an idea of evolutionary progress, this title described a goal directed development from the computing machinery of the past towards a future of interaction.

This trajectory constitutes the standard account of the history of interaction. Often the field is seen as following a teleological development of progress, during which computers became more and more interactive, and interaction became more intuitive, rich, and natural. This development is often explicated as a genealogy. Depending on the focus and goals of their narrators, there are genealogies of interaction focusing on a succession of hardware generations, interaction paradigms, theoretic frameworks, or visionaries pushing the field to the next level.

An early and paradigmatic account that focuses on hardware is John Walker’s genealogy of five “User Interaction Generations”⁴ published in the early 1990s. This account starts with the “plug boards” and “dedicated set-ups” of early computing.⁵ These were followed by the (in)famous era of batch processing – a time when programming meant punching holes into cards, handing batches of these cards to a mainframe operator and waiting for hours to be handed back a printed result.

It is only the third of these generations that was interactive. As to be expected from a proper generation, it was a child of the previous one and generated by it: When, this account goes, the algorithms allocating a mainframe’s computing time to several batch jobs got more and more advanced, it became clear that it would be possible to divide the computing time of a mainframe even further. Divided into small enough pieces, that follow each other in rapid succession, it would seem to several people that they would have exclusive control over the whole machine. This idea, named “time sharing,” did not divide a computer’s resources between several batch jobs but between several humans (re-defining these, as we will see below, as “users”) who could now engage in “conversational interactivity” with the machine.⁶

While the conversations of the time sharing generation happened as exchanges of written text, the fourth of Walker’s generation of interaction introduced graphical displays that concentrated textual commands into visual menus. The fifth and final generation then spawned the graphical user interfaces of personal computing that, in various iterations, keep accompanying us on our desktops, laptops, and phones until today.⁷

Paul Dourish’s “History of Interaction,”⁸ which is much more contemporary in style, follows a very similar path “from soldering to mouse.” Focusing on the mode of interaction instead of hardware generations, the first generation here was defined by “electrical interaction” (using cables, plugs, and the soldering iron) with Walker’s dedicated set-ups. This was followed by the era of “symbolic interaction” that was marked by the use of punch cards and batch processing – which were often programmed using the symbols of assembler languages instead of the raw zeros and ones of machine code. This generation, in turn, led to the “textual interaction” with the terminals of time sharing systems. “Graphical interaction” here again marks the final step in an evolution starting with machinery and ending with today’s interactive surfaces.

Apart from their implicit assumption that interactivity progressively increases, these, and most other histories of human-computer interaction (HCI) have one thing in common: They all assume an origin of interaction. While operating a computer during Walker’s second generation meant batch processing, the third generation introduced time sharing and with it, interaction. In Dourish’s terminology, this transition corresponds to the shift from “symbolic” (based on assembler language and punchcards) to “textual interaction” (based on conversational interactivity via command line). Only when computers, after time-sharing was introduced, started to react (seemingly) exclusively and directly to human input, did they become interactive: “Arguably, this is the origin of »interactive« computing.”⁹

Of course, history is not that simple. A closer look behind the narratives postulating a teleological development of interaction instead reveals contradictory and asynchronous developments, as well as chronological overlaps.¹⁰ Especially the origin of interactive computing itself can be described differently, for instance, by looking at the first interactive computer ever built – which happens to be one of the first computers at all. It is, as this look reveals, the very first generation of computing machinery that defined interactivity up to this day, including its problems.

INTERACTIVE COMPUTERS AS FEEDBACK MACHINES

During the 1940s, the MIT Servomechanisms Lab started to build a flight simulator. As the leading paradigm for automatic computation at that time was analog computing, the flight simulator was planned to be based on that: “a cockpit or control cabin connected, somehow, to an analog computer.”¹¹ “Analog computing” in this context did not only imply calculating with analog (i.e. continuous) values, it rather implied an entirely different approach toward calculation: It relied on building electrical and mechanic systems, that, as analogues or analogies, could stand in for the systems they were built to simulate. Because building such analog computers entailed accurately following and amplifying changing physical signals, it largely depended on another development: the rise of the use of negative feedback as the de facto standard method for handling electro-mechanical systems. In fact, during the early twentieth century, negative feedback became so important in both control and communication engineering that both disciplines merged into one feedback based control theory – in a paradigm shift that yielded the era of “classical”¹² control. This development, in turn, constituted the nucleus of what Norbert Wiener would later call cybernetics – a science of “control and communication in the animal and the machine”¹³ that would become thinkable mainly because the application of negative feedback and the associated mathematical formalisms seemed to be powerful enough to tackle any form of “behavior” – of living and non-living systems.¹⁴ Because feedback implies using the output of a system as its own input, the systems of cybernetics exhibited “circular causality”¹⁵ – a circular interdependence of input and output, entailing that agency within the system is distributed and cannot be pinned down to specific agents.

In building the simulator, moving axes and disks, and changing voltages and currents were used as analogies to the complex dynamics of a plane in flight. As these analogies constituted electro-mechanical motion, coupling them to the moving controls of a cockpit and the motion of their human operators was self-evident. However, during the development of this “Aircraft Stability and Control Analyzer” (ASCA)¹⁶ the first digital computers were under construction as well. The engineers at MIT observed this development and Jay Forrester, one of the project leads, became more and more interested in digital computation – so interested, in fact, he sacrificed the core of the project (building a flight simulator) to his new interest (building a digital computer): The development of the analog computer was halted, and a “general purpose, high speed”¹⁷ digital computer was built. As it was one of the first of its kind, the engineers building it were constantly “pushing the state of the art,”¹⁸ developing new building blocks for digital computation, such as memory mechanisms. Caught up in this task, however, they increasingly lost sight of the fact they were trying to build a flight simulator. This was especially problematic, as the ASCA’s cockpit still was the analog machine the project started with. Whereas coupling the motion and continuously changing electrical signals of an analog computer to the analog instruments of a cockpit did not pose a categorical problem, this had changed with digital computing. The digital and discrete state changes of the new computer had to be translated into continuous motion of the instruments, while the reactions of the operators on these instruments, in turn, had to be translated into digital states.¹⁹ “These problems were not impossible, but neither did established solutions exist. The digital computer was too new,”²⁰ one of the engineers in the project later wrote. In consequence, the project management acknowledged that it was not about building a flight simulator anymore and the cockpit of the ASCA was scrapped.²¹ The computer was renamed “Whirlwind”²² and became a general-purpose digital computer not usable for flight simulation anymore. As it thus became a computer without application, it later would be turned from flight simulation to air defense and become the foundation for SAGE, the “Semi-Automatic Ground Environment” air defense system – the largest computer built to date that was in use until 1983.²³

What set Whirlwind apart from the other first-generation digital computers of its time was its heritage in analog computing and flight simulation: It was conceived as a machine that reacts to changes in an environment (the cockpit) by incorporating any change happening here into its calculating. In addition, it would have the results of these calculations directly, and in real-time affect the environment. In other words; it was a digital computer that was to function like the control systems of analog computing and cybernetics – as a digital computer that can react to its environment in real-time.

This is remarkable, given that theoretical computer science operates with a conception of “machine,” explicated as with the Turing machine, that does not know time or any reciprocal interaction between calculation and its environment. Only relatively recently did theoretical computer science start to acknowledge, that the actual computing machines we have been using from the very beginning had done something that goes beyond Turing’s definition of computation – by incorporating interaction with an environment.²⁴

Whirlwind thus was a strange hybrid: A digital computer that also tried to be a cybernetic feedback system, in constant dialog with the environment it controlled. If we follow Winograd’s juxtaposition of computing machinery and interaction, it was both: a machine and interactive – a feedback machine.

INTERRUPTION AND COUPLING: A BLACK ART

Even after having scrapped the cockpit, Whirlwind was still a machine to be used by human operators in real-time and as such posed two problems: How to integrate real-time input from the environment into an ongoing digital computation, and how to couple the process of digital computation to the action and perception of human operators. The engineers of Whirlwind approached these novel (or even “too new”) problems pragmatically.

The fundamental problem of having the machine react to its environment was tackled introducing a basic technique into computer engineering whose heritage is alive until today: Whirlwind could interrupt what it was working on, turn to any new data that may have arrived in the meantime, integrate that data (by copying it into memory), and continue where it had left off.²⁵ Coupling the machine to the environment thus became a function of interruption – which, as hardware interrupt, later became a core feature of any interactive computer.

The problem of coupling computation to human operation was instead approached by introducing what later would be defined as one of the teleological ends of the development of interactivity: Whirlwind produced graphical representations that could be touched. This was made possible when the engineers in the project coupled memory registers of the computer to the x/y-control of the magnetic fields of a cathode ray tube (CRT). Whirlwind could thus paint symbolic representations of data onto screen: “One of the things that I think we did first was to connect a visual display to a computer.”²⁶ This great leap into our screen-based present happened with the pragmatic naturalness of something “I think we did first,” simply because all prerequisites for it were already in place: The second world war had established various modes of coupling (analog) radar data to CRTs. Project Whirlwind could build on this foundation and even use the leftover CRTs of the war.²⁷ In addition, CRTs had already been coupled to digital computers: In the “Williams Tube,” the afterglow of the light painted onto a screen was used as a short-term memory device that was not meant to be looked at by humans, but nevertheless constituted computer control of light on a screen.²⁸

In order to close the loop between representation and action, the images painted by Whirlwind onto its CRTs were accompanied by a device to touch them: a “light-gun” (figure 1). The device realized this by feeding back the computer’s visual output to its own interrupt: The “gun” was designed not to shoot but to pick up light. Pointed at a visual representation on screen, it would pick up the light emitted when the computer drew this very representation. If an operator now pressed a button, the computer was interrupted while drawing it. It thus “knew” which item was selected and could take this selection into account for further computation.²⁹ Even the light gun, although pioneered here as an interaction device, had technically already been built before it became part of the configuration of interactive computing – as it was originally used to test the Williams Tube memory devices for errors.³⁰

Coupling Whirlwind to people was thus both: the pragmatic problem-solving of engineers using parts and components at hand, and a revolutionary prototype for most interactivity to come. But while it offered the basic capability of having human action become part of an ongoing computation, it did not solve any problems of how exactly this setup should be used. Instead, computer science had unexpectedly introduced a new class of problems, as the representations and couplings it made possible now had to be designed. It became a field of design, a “black art” in which “engineering design,” “creative design,” and scientific methods came (and still come) together.³¹

(IN)HUMAN FACTORS: THE USER AS NEW HUMAN

In spite of Whirlwind, the narratives of the progressive incline of interactivity are not plainly wrong. Although interactive computing existed before time-sharing,³² MIT’s Whirlwind was a singular development and most of computer science for a long time stuck to building machines running algorithms that produce answers without being interrupted. Important early developments, such as Ivan Sutherland’s Sketchpad³³ and especially Douglas Engelbart’s NLS, were running against this mainstream that was so dominant it took the field until the 1980s to acknowledge “interaction” as an independent area of inquiry. One of the first books carrying human-computer interaction (HCI) in its title was “The Psychology of Human-Computer Interaction”³⁴ by Stuart Card, Thomas Moran, and Allen Newell. The role of the latter in establishing HCI is remarkable, as he serves as a link back to the first interactive computer as well as pointing towards the future of the field.

Early in his career, Newell worked at RAND’s Systems Research Laboratory. Here, he was in charge of training the operators of the SAGE system – and thus the first professional operators of interactive computers.³⁵ This work was conducted together with Herbert Simon, with whom Newell would continue working on a number of subsequent projects. While building a training environment for the SAGE operators, Newell used computer modeling to simulate the input into the training system, consisting of human operators and simulated computer consoles. His simulation created sequences of “radar blips,” as they would have shown up on the real screens of the SAGE air defense system. The realization that computers could do something like this, and thus “more than arithmetic”³⁶ would prove highly influential for Newell and Simon. The fact that in training these computer operators, computer modeled input data shown on computer screens³⁷ would be perceived and interpreted by human observers, led Newell and Simon to the far-reaching conclusion that all participants of this system were essentially involved in the same task: processing information. Just as Newell’s digital simulation processed information in order to produce the fake radar blips, the human operators looked at these blips and perceived them as information to be processed and acted upon. In other words: “Within the simulated training environment, Newell came to view the human operators too as »information processing systems« (IPS), who processed symbols just like his program »processed« the symbols of simulated radar blips.”³⁸

This is the crucial outcome of the training for the first interactive computers. Subsequently, Newell and Simon authored a number of papers that took this idea further, developing an understanding of human thinking that was driven by the verdict that it is a form of the symbolic information processing exhibited by computers. This culminated with the “Physical Symbol System Hypothesis,” declaring intelligence to be a feature of all forms of physical systems that are able to manipulate symbols – be it human or machine.³⁹ This argument, at the time, was part of the development of a new scientific field of studying the human mind that, at least for a long time, understood thinking as rule-based information processing: cognitive science.⁴⁰ The field from the very beginning “subsumes various computational theories of mental phenomena. Their computational nature is what unifies the multiple disciplines in the field and may count for much of its success in recent years.”⁴¹ In this sense, the human trained to perform in front of the computer became the model for the thinking human in general – a human acting as a computer.

This is what Newell brought back to working with interaction: He proposed to XEROX PARC an “Applied Information-processing Psychology Project (AIP)”⁴² that promised to apply cognitive science to the black art of designing interaction. The project started in 1974, led by Card and Moran, who were consulted by Newell. One of its results was the publication of “The Psychology of Human-Computer Interaction” by the three.

The center of this project was not longer the computer operator. Instead, it was the “user” of the computer interface. Card, Moran, and Newell stated: “But the user is not an operator. He does not operate the computer, he communicates with it to accomplish a task.”⁴³ This attribution of agency to the computer (as an equal partner in communication) probably followed from the nature of the interactive computer as feedback machine that exhibits circular causality between machine and (human) environment. For the authors, however, the relationship of user and computer was defined solely by the postulated equivalence of all information processing systems.

In proposing the project to XEROX, Newell suggested to marry the empirical methods of human factors with the formal (and computational) models of cognitive science, creating “a technical understanding of the user himself and of the nature of human-computer interaction.”⁴⁴ This would be a “science of the user rooted in cognitive theory.”⁴⁵

In doing so, he seemed to be aware that this user was not a given, but something that was created by the systems being used – after all, it was a training environment for early computer users that gave rise to the idea of the human as information processing system. In the memo proposing the AIP to XEROX he thus wrote: “There is emerging a psychology of cognitive behavior that will permit calculation of behavior in new situations and with new humans…”⁴⁶ Since this user was to be subject to the technical understanding provided by computational theories of mental phenomena, what emerged here was a view of the human being using the computer, as a computer.

The human factors of human-computer interaction, and human- or user-centered design thus become readable as the “inhuman factors”⁴⁷ of thinking humans as machines – and making them act accordingly. The training required to become the new human that an interface demands, in this sense, can be seen as a “subtle enslavement”, and a “total, unavowed disqualification of the human in favor of the definitive instrumental conditioning of the individual.”⁴⁸

COGNITIVE ENGINEERING VERSUS CONCRETE THINKING

It is this convergence of computer and cognitive science that served as the “origin myth” of human-computer interaction.⁴⁹ The field did, for a long time, embrace cognitive science and its methods, effectively becoming a form of “cognitive engineering” as Donald Norman defined it in a seminal paper: “neither Cognitive Psychology, nor Cognitive Science, nor Human Factors. It is a type of applied Cognitive Science, trying to apply what is known from science to the design and construction of machines.”⁵⁰

Being based on the cognitive science idea of what a human is, cognitive engineering was seen as a form of “user-centered” design. At the center of this idea stands a juxtaposition of the mental and the physical. Interaction, the argument goes, is an act of mediating between a user’s mental goals and the physical states of a system. This mediation happens in a loop of “execution” and “evaluation,” while execution is based on action sequences a user formulates according to their goals.⁵¹ Formulating these action sequences is possible because users possess a “mental model” of how they assume a system functions.⁵²

The task of the interface designer as cognitive engineer now is to make sure that this mental model is correct – so that an action sequence will lead to the expected and intended results. They must bridge the gulf between execution and evaluation.⁵³ Creating an interface thus becomes an act of communication where a designer’s “design model”⁵⁴ must be communicated in a way yielding the appropriate mental model. In terms of information processing, this means that by its design a system must provide the information that, once perceived and processed, leads to the appropriate actions that fulfill a given goal.

According to Norman, there are two ways of achieving this: “(M)ove the system closer to the user; move the user closer to the system.”⁵⁵ Of course, user-centered design wants to move the system closer to the user, by creating systems whose physical states behave in an “intuitive” or “natural” way, close to the mental intentions of their users. This, of course, implies that the latter can be formulated in terms of the former. A user’s non-physical goals and intentions must be translatable into physical actions and system states, thus reproducing the assumption that computer users ultimately can be understood on the same ground as the computers they use.

The relationship between the psychological states of a user and the physical states of a system has been described as “directness.”⁵⁶ This term entered HCI discourse when Ben Shneiderman in 1983 was puzzled by “[c]ertain interactive systems” which “generate glowing enthusiasm among users.”⁵⁷ What set these systems apart was the interactivity already introduced by Whirlwind: “(D)irect manipulation” of graphical representations without the need to type text – the origin (as constructed here) of interaction thus once more got reinterpreted as a milestone of the progressive incline of the field. But as opposed to the pragmatic engineering behind Whirlwind’s early interfaces, Shneiderman’s discussion of direct manipulation followed a cognitivist pattern, having a clear idea of the human user as rational problem solver in mind: Shneiderman reproduced Norman’s idea of interaction as psychophysical mediation by identifying a “problem domain” of “semantic” intentions and a “program domain” of “syntactic” manipulations at the interface. Direct manipulation, he argued, enables users to interact directly with the objects of the problem domain – by, for instance, enabling a writer to directly interact with paragraphs of text, instead of having to deal with the commands meant to manipulate these paragraphs. Direct manipulation would hence be a (or maybe the) realization of Norman's “move the system closer to the user” by minimizing the distance of the problem and program domain.

Not surprisingly, Norman himself later joined the discussion, expanding Shneiderman’s work in cooperation with James Hollan and Edwin Hutchins.⁵⁸ This argument started with the assertion that, “[w]e see promise in the notion of direct manipulation, but as of yet we see no explanation of it.”⁵⁹

Trying to formulate this explanation as a full-fledged “cognitive account”⁶⁰ of direct manipulation, they reformulate Shneiderman’s distance of syntax and semantics as an “information processing distance” between human intentions and machine states⁶¹ – a distance that direct manipulation is minimizing. These interfaces, in this view, are easier to use because what we want do with them corresponds to the way it is done.

This, however, may not be enough to explain the “glowing enthusiasm” described by Shneiderman. Instead, the authors acknowledged that direct manipulation seems to entail an experiential component that can not be explained by information processing alone. It features a feeling of “engagement,”⁶² that is hard to come by: „Although we believe this feeling of direct engagement to be of critical importance, in fact, we know little about the actual requirements for producing it.“⁶³ Referring to Brenda Laurel’s work that applied Aristotelian poetics to HCI, they concluded that a feeling of “first-personness”⁶⁴ must be responsible for the feeling of engagement. For Laurel, this feeling was based on the interplay of user and interface, as „[a]n interface [...] is literally co-created by its human user every time it is used.“⁶⁵

Direct manipulation hence seems to contain a playful component and a residue of the non-rational. It is not about a cognitive distance between mental intention and physical representation and action alone, it also is about a subjective experience that is created through the cyclic dependence of user action and machine response. This non-rational (or non-cognitivist) residue, however, seemed to deeply bother Hutchins, Hollan and Norman, who stated:

“On the surface, the fundamental idea of a direct manipulation interface to a task flies in the face of two thousand years of development of abstract formalisms as a means of understanding and controlling the world. Until very recently, the use of computers has been an activity squarely

in that tradition. So the exterior of direct manipulation, providing as it does for the direct control of a specific task world, seems somehow atavistic, a return to concrete thinking.”⁶⁶

This return to concrete thinking subsequently became even more prominent when “tangible user interfaces” and other forms of non-screen-based interactivity emerged. When, for instance, physical objects in research projects at MIT and elsewhere became phicons⁶⁷ – physical icons that represent data and computational processes – researchers at XEROX PARC coined the term, “interfaces for really direct manipulation.”⁶⁸ If tangible user interfaces use real-world objects as representations of computation, the hope was, they would feel ultimately natural and the information processing distance would be reduced to zero.

This, however, makes two things apparent: First, if the mouse and screen felt natural during the 1980s and tangible user interfaces felt more (or really) natural during the early 2000s, naturalness itself must be understood as a fluid category depending on what feels natural for the “new human” of each era. Interfaces like the touch screen in this light must be understood as being products of a naturalization creating the very human for which they feel natural. Second, the whole discussion of tangible interaction neglects the fact that all interfaces in one form or another have been tangible: We have never “directly” manipulated a paragraph of text but always had to deal with pens and marks on paper, keyboard and screen, fingers on a touchscreen. The atavistic syntax of executing manual actions always existed and was always different from any semantic goal or intention.

What, instead, differentiates tangible user interfaces from graphical user interfaces and these from the command line is something much more profane: It is the simple spatio-temporal distance of human action and computer reaction as well as their perceived similarity. The incremental progress of interaction, postulated by the genealogies of interactivity, is another clue suggesting that what is really interesting about interactivity is the closure of the gap in space and time between human and computer action. In particular, the theoretical reflection on non-screen based interfaces had understood this at an early stage. Already in 2000, Michel Beaudouin-Lafon has concluded that what is really important about the experience of computer interfaces is the “spatial and temporal offset,” the “ratio between the number of degrees of freedom” and the “similarity between the physical actions of the users on the instrument and the response of the object.”⁶⁹

PERCEIVING ACTION

No matter if we hail the natural or intuitive interface as bridging the gap between user and system, or if we condemn the interface as a form of conditioning that ultimately naturalizes a non-human mode of action and perception, we presuppose the interface as the agent of this process. If interfaces are seen as forming the new human after their own image (by moving them closer to the system) or if they supposedly assist a given human by modeling their non-physical goals or semantics, they are assumed to be sources of information that are perceived, processed and acted upon. Interaction is seen as a one-way street, conveying a design model to a user, who is acting by that model either because they adapted to it, or because the model replicates their given structure. This is the cognitivist heritage of the HCI discourse responsible for the idea that interfaces can actually be designed.

When, however, the engineers in project Whirlwind coupled digital computation to symbolic representation and human action back to computation, they not only wrapped its human operators in the feedback loop of the circular systems of cybernetics: They also created a setting in which the representations would be wrapped in a loop of human action and perception.

The motion on a computer screen is not real motion but a cinema-like sequence of still images, which psychology denotes as “apparent motion.” Apparent motion has been a subject of experimental psychology since the cinematograph and cinema rendered it ubiquitous, providing the experimental systems for studying it and for using it as a tool to study perception in general.⁷⁰ One seminal early work was Max Wertheimer’s experimental studies of the perception of movement,⁷¹ which today is seen as one of the founding texts of Gestalt psychology.⁷² While Werheimer pioneered the experimental investigation of apparent motion, Gestalt psychologists like Paul Linke and later Paul von Schiller studied the phenomenon with a focus on a fringe case of it: The perception of “ambiguous motion,” which is present whenever the direction of an apparent motion stimulus can not be decided objectively (figure 2). Such stimuli are interesting because they afford more than one possible perceptual interpretation, while subjectively only one direction of motion is perceived at a time. They thus reveal how the sensory system is treating stimuli in deciding how they are to be perceived, making them “invaluable tools for the study of the neural basis of visual awareness, because they allow us to distinguish neural responses that correlate with basic sensory features from those that correlate with perception.”⁷³