Instant Sensemaking, Immersion and Invisibility: Notes on the Genealogy of Interface Paradigms

by Florian Hadler + Daniel Irrgang

Abstract

This paper discusses current interface paradigms, the genealogy of the GUI and their underlying semiotic or diagrammatic principles. By conceptualizing the interface as a historical phenomenon, as a coded zone of differentiation and distinction, it opens a perspective on the interface that is both historically and theoretically informed, taking into consideration not only its social, economical or technological conditions, but also its philosophical and theoretical discourses. The consolidation of contemporary aspects, genealogical investigations and theoretical explorations provide outlines of a theoretical framework for the studies of interfaces and their historical developments.

Introduction

When talking about interface paradigms it seems useful to first interrogate the concepts and theories of the interface itself. The term ‘interface’ has become as common as mysterious, its usage varying across different approaches and contexts. The more it is taken for granted, the more it seems to escape our understanding (Hadler and Haupt: 2016). Therefore, we suggest a discussion of different theories of the interface before analyzing selected paradigms, not so much to extrapolate an established definition, but to render visible the dynamics and contradictions within the concept of the interface, its perspectives and applications and, last but not least, its contingency and historical dependencies. We selected a set of interface theories from backgrounds as various as philosophy, literary theory, architecture, media studies and computer science.

One of the most common understandings of the interface is that of a surface, of a two-dimensional space that can be described as a significant surface (Flusser 1973). The conceptualization of the interface as a surface implies certain characteristics, such as the spatialization of meaning and, accordingly, the notion of the semantic value of relations like proximity, overlap, hierarchy, dependency and sequence (Drucker 2011). Its theoretical frameworks draw from methodologies such as comic book analysis, graphical reading, frame analysis, cognitive semantics and diagrammatics. An obvious advantage of the spatialization of meaning is the combination of an overview with the possibility to easily grasp details; perceptual ‘synthesis followed by analysis’, as Flusser puts it (Flusser 1973: 101). But this conceptualization of the interface as a significant surface seems deficient, as the interface involves not only the arrangement of information on a screen, but also the interaction between two or more entities. Many interface theories, therefore, not only focus on the materiality or appearance of interfaces, but also necessarily include the relation between the subjects of the interface, which are either user and technology, user and user by means of technology, or any other entities that are linked together. This not only puts into perspective the interface as a zone of interaction, but also transcends the notion of the interface as a surface, as this broader scope is not limited to graphical user interfaces, as implied in the notion of a surface. Rather, it also includes gateways between databases, code modules or other forms of machine based communication. The interface, in this perspective, renders incoherent modes of communication coherent.

The interface as a gateway, as a process or transition, is heavily informed by information sciences, cybernetics and systems theory. Once again, though, there are certain limitations thinking about the interface as a passive gateway or threshold, as it does not only link different entities, but at the same time constitutes them as participating interactors. It is a form of relation (Hookway 2014) but also a form of differentiation and distinction (Weibel 2001), a field of transition (Galloway 2012), a mediating environment and dynamic space (Drucker 2011), a fertile nexus (Dagognet 1982), that links and separates at the same time. That means, the interface divides its subjects by linking them together, it is ‘not only defined by but also actively defines what is humane and what is machine’ (Hookway 2014: 12). The interface includes ‘processes of holding apart and drawing together, of confining and opening up, of disciplining and enabling, of excluding and including’ (ibid.: 4).

This concept of the interface as a zone of division and transition leads to the conclusion that the interface is not only a mode of interaction or communication, but at the same time a structured set of codes, an artifact of complex processes and protocols, a space of power where creative, social, political, economical, as well as technological genealogies are inscribed. This renders the interface a historical phenomenon. As zone of transition, translation and transcoding, the interface becomes necessary in the light of the ever increasing complexity of technological artifacts. Since these artifacts tend to become black box-like apparatuses, incomprehensible in their functional principles for the majority of the users, they call for graspable interfaces to make the interactions with the apparatus possible again. This tendency can be described as a dialectic of structural complexity and functional simplicity (Flusser 1973). The architecture and construction of the artifacts are more and more intricate whereas the usage becomes easier and rather conventional. A phenomenon most significant in the field of human-computer interaction (HCI).

Rapid progresses in user interface design made the former calculating apparatus, the computer terminals operated by highly qualified professionals, usable ‘for the rest of us’, as Apple claimed in their advertising campaign for the Macintosh in 1984, introducing the graphical user interface to the consumer market. Our effort will be to make visible the current inscriptions of the interface by tracing some of its historical predecessors, its inspirations and conditions. Before looking at one of the major streams of the genealogy of the interface, the development of the graphical user interface, we provide a brief overview of some current paradigms in interface design and production.

Some Paradigms

The term paradigm in this context refers to Thomas S. Kuhn’s concept, who uses it in two distinct, but intertwined senses: it is both a ‘disciplinary matrix’, a set of conscious or unconscious values, and a specific element within this set, serving as an example for the more abstract whole set of values and techniques (Kuhn 1962). These two sides of the paradigm mirror the hermeneutic circle, the production of knowledge in the human sciences. Knowledge of the particular presupposes knowledge of the whole and knowledge of the whole presupposes knowledge of the particular. Giorgio Agamben argues that the hermeneutic circle is, in fact, a paradigmatic circle: ‘There is no duality here between “single phenomenon” and “the whole” […]: the whole only results from the paradigmatic exposition of individual cases’ (Agamben 2009: 27). We will employ this concept of paradigm as a tool of interpretation to current developments in interface design, looking both at concrete examples and singularities while at the same time deducting more general observations of the whole field of the interface.

The current developments in the area of consumer technology and software services are certainly diverse, but there seem to be certain underlying paradigms that are visible throughout a broad range of paradigmatic products, such as the touch paradigm of the smartphone, the tablet and other devices that operate with haptic input, or the natural language paradigm for services like Siri, Facebook Search and Amazon Echo. Some of these (partly not so) new paradigms can be easily accessed and applied to software developments in the form of preformulated guidelines, that explicitly state their underlying principles and guiding concepts. Every big platform that provides cross-device operating systems makes sure that developers are aware of their constantly updated guidelines. These design and development resources – provided by the big players Apple, Google and Microsoft as online databases – became more and more relevant with the development of the app-economy starting shortly before the launch of the iPhone in 2007. With the rise of third-party developers producing software for existing cross-device operating systems, their products increasingly added value to the platforms on which they operate. In order to be attractive to the consumer, these platforms have to make sure that they provide a broad range of third party apps and services with a consistently good user experience.

One of the biggest and most obvious paradigms across all devices and applications is simplicity. It is way too broad to be treated as a general trend, but it groups together various aspects of developments. Overall simplicity enables the user to cope with complexity, to reduce decision-making and to allow a seamless flow of information and control. The approaches which aim to break down complexity, to create context sensitive modes of decisions and intuitive control can be differentiated. We suggest three distinct but overlapping sub-categories to look at the overall paradigm of simplicity.

Bildschirmfoto 2016-11-23 um 20.56.43

Figure 1. The Strava application tracks athletic activities, analyzes and compares the data and provides diagrammatic overviews on smartphones or smartwatches (2015). (Source: Strava for Apple Watch product presentation; http://www.strava.com/apple-watch, accessed on April 26, 2015).

_________________

Instant Sensemaking

The first category we propose to call the paradigm of Instant sensemaking, concerns the intuitive and instant readability of and interactivity with data and content, as it can be seen in the numerous dashboards and diagrammatic representations of live actions, delivering immediate examinations and results and therefore not only allowing, but demanding the user’s direct response (Irrgang and Hadler 2014). The Google AdWords Dashboard or any other online campaign tracking tool applies this paradigm – as well as the numerous self-quantifying apps to monitor certain user behaviors such as strava or the integrated health service from Apple’s iOS (Fig. 1). Instant sensemaking is further supported by the touch paradigm, as can be seen on mobile devices, tablets or trackpads for picture editing, gaming and many other services, which provides the feeling of ‘direct manipulation’ as Apple calls it: ‘When people directly manipulate onscreen objects instead of using separate controls to manipulate them, they’re more engaged with their task and it’s easier for them to understand the results of their actions’.[1]

Immersion

The second category or sub-paradigm of simplicity, prominently featured across various interface guidelines, is closely related to instant sensemaking. What Apple calls ‘engagement’ and Google calls ‘continuity of experience’[2] and is, at the same time, well known in film theory and practice as the ‘découpage classique’ or ‘continuity editing’, is what can be called the paradigm of immersion. Immersion becomes more and more relevant for third party developers, as it builds a lasting attraction for the user. Metrics such as ‘returning users’ and ‘retention period’ are crucial for monetization of apps, either through advertising or in-app purchases. The more immersive a service is, the more users will spend both time and money. Immersion is achieved not only by instantaneous and constant feedback, but also by social interaction with other users, by coherence of the design, as well as, by the use of intuitive control elements. The intuitiveness of the control elements is produced through the application of graphical ‘metaphors’ such as shadows, translucency, weight or other tactile attributes. In 2014 Apple shifted from the skeumorphistic approach of their first guidelines towards the so called ‘flat design’ principles; Google calls their approach ‘material design’, combining both elements of skeumorphism and flatness (Fig. 2). The application of metaphors is features strongly in both guidelines, and is explained as follows:

„A material metaphor is the unifying theory of a rationalized space and a system of motion.    The material is grounded in tactile reality, inspired by the study of paper and ink, yet technologically advanced and open to imagination and magic. Surfaces and edges of the material provide visual cues that are grounded in reality. The use of familiar tactile attributes helps users quickly understand affordances. Yet the flexibility of the material creates new affordances that supercede those in the physical world, without breaking the rules of physics. The fundamentals of light, surface, and movement are key to conveying how objects move, interact, and exist in space and in relation to each other. Realistic lighting shows seams, divides space, and indicates moving parts.“[3]

When virtual objects and actions in an app are metaphors for familiar experiences – whether these experiences are rooted in the real world or the digital world – users quickly grasp how to use the app. It’s best when an app uses a metaphor to suggest a usage or experience without letting the metaphor enforce the limitations of the object or action on which it’s based. iOS apps have great scope for metaphors because people physically interact with the screen. Metaphors in iOS include: Moving layered views to expose content beneath them/Dragging,   flicking, or swiping objects in a game/Tapping switches, sliding sliders, and spinning pickers/Flicking through pages of a book or magazine.[4]

The use of metaphors that reach beyond the simple graphic metaphors of icons, makes the modes and forms of interaction immediately understandable, without forcing the user to read manuals before using the software, enhancing, thus, both intuition and immersion.

Google Material Design applies lighting, shadows, translucency and layering as spatial metaphors to create immersive and intuitive design principles (2015). (Source: Google Material Design, part ‘Introduction: Principles’; http://www.google.de/design/spec/material-design/introduction.html#introduction-principles, accessed on April 26, 2015).

Google Material Design applies lighting, shadows, translucency and layering as spatial metaphors to create immersive and intuitive design principles (2015). (Source: Google Material Design, part ‘Introduction: Principles’; http://www.google.de/design/spec/material-design/introduction.html#introduction-principles, accessed on April 26, 2015).

_________________

Invisibility

The third category we propose, is a direct result of the immersion and instant sensemaking, but reaches beyond the simple design of screen elements. We call it the paradigm of invisibility. It is common knowledge among user interface designers that the best interface does not exist or, as Don Norman puts it: the real problem of the interface is that it is an interface. Interface design strives for an interface so intuitive that it appears in a constant oscillating state of both guiding the user and disappearing from his awareness. The goal, therefore, of each interface design is to disappear, to withdraw itself, to become imperceptible and ultimately invisible (cf. Zielinski 1997). This is partly achieved by immersion, which causes the user to forget that he is actually using a device or an app, just as a viewer forgets he is actually watching a movie or a player forgets he is playing a game. But it is also a paradigm that informs the development of ubiquitous computing, the ‘backgroundability’ of technology, such as RFID chips, near-field-communication between devices and objects and wearable technology such as smartwatches or smartbands, stepcounters or even intrusive technology, like cochlear implants or other bodily enhancements. Voice or gesture control simulates ‘natural’ interaction as applied by Apple Siri, or the far field voice recognition as done by Amazon Echo, that constantly listens to every conversation in order to take up cues. Conversational search, as cis currently done by Googl, deep integration such as the access of contacts, photos or other user data by third party apps or actionable notifications that allow interaction on the homescreen without opening the app, are all signs of this paradigm of invisibility, which is often referred to as ‘shytech’ – technology that hides itself. Invisibility relies heavily on cloud computing, on big data analysis, semantic search and learning algorithms, in order to reduce the number of decisions and to act proactively with regards to user needs, taking into account contexts such as user preferences and other external data. The usage of cloud based operations, of invisible and hidden computation, refers to the dialectic of structural complexity and functional simplicity as described by Flusser and others.

With the three paradigms related to simplicity, instant sensemaking, immersion and invisibility, we outlined some of the current trends and developments in interface design, software development and service production. Obviously our list is not complete, but offers an instructive picture of what is happening in the field of contemporary interface design. One could just as well elaborate on the rise of haptic interfaces, that are increasingly used for in-car-systems, gaming devices or virtual reality setups. Or dig deeper into natural language processing, artificial intelligence, semantic search and self-learning algorithms, that are essential for more intuitive and context sensitive search programs. And last but not least, the rise of the mobile device deeply affects standards of layout, typography and spacing for displaying content and enabling interaction, resulting in responsive, flexible and adaptive concepts that rearrange and resize content autonomously.

Instead of collecting more and more examples, we will turn now to examine the origins of the graphical user interface, which are directly related to the general paradigm of simplicity and the sub-paradigm of instant sensemaking. In doing so we hope to sketch a more detailed genealogy of the user interface of today. Based on that genealogical sketch we will take a closer look at their underlying iconic or diagrammatic and metaphorical principles.

Some Origins

The best example for the above described shift of the apparatuses becoming increasingly structurally complex while showing a strong tendency to appear functionally simple can be found in the dawn of the personal computer. The former calculating apparatus for highly specialized tasks turned into a device usable for daily office work with the presentation of the Xerox Star workstation in 1981. Only a few years later, personal computers entered the consumer market with the launch of the Apple Lisa (1983) and the Apple Macintosh (1984). While the Xerox Star was relatively expensive, the Apple personal computers were not only affordable for private households, but also especially designed for consumers, proposing an integration of the computer to everyday life. On a technological level, the driving force of this radical diversification – eventually becoming an important element of the strong socio-technological impact often reduced to the buzz word information society – consisted of basically two corresponding elements: the computer mouse and the graphical user interface (GUI). The Apple advertising campaign for the Macintosh was the first campaign for a computer that not only mentioned but even focused on the user interface (Johnson 1997).

Both the mouse and the GUI were vital for Macintosh’s success. The structurally complex computer, formerly only operable through complex sequential command-lines, could now be used and controlled by functionally simple interactions with the computer mouse as pointing device, dealing with iconic and graphical representations on a clearly organized surface. In the ensemble ‘human-interface-computer’, the command-line interface (CLI) was replaced, at least for the bigger part of tasks, by interactions guided by a GUI. This new way of human-computer interaction (HCI) made possible a cognitively ‘more intuitive’ approach of direct interaction and feedback, determining the interaction principles of computer interfaces and mobile devices until today. Whilst additionally creating a new kind of computer user (cf. Moran et al. 1983)[5] – and soon replacing a highly specialized practice with a general cultural technique.

Although it was the Xerox Star that introduced the computer mouse and GUI to the market in 1981, both technologies existed long before. The mouse was invented in 1963, by Douglas C. Engelbart and Bill English, at Stanford Research Institute. The basic principles of GUI, such as manipulating objects on a screen by using a pointing device for receiving direct feedback,[6] were developed even earlier by Ivan E. Sutherland. The Sketchpad system, designed at the MIT Lincoln Lab in the beginning of the 1960s, enabled its user to interactively draw geometrical figures on a computer screen by using a light pen (Sutherland 2003). Like Apple’s approach twenty years later, Sutherland’s goal was to make computers usable by a new group of less technology savvy people (Blackwell and Rodden 2003).

Sutherland’s work, including the theoretical ideas published in his dissertation, a research report on Sketchpad, influenced the work of the Learning Research Group at the Xerox Palo Alto Research Center (PARC). The head of the group, Alan C. Kay, studied with Sutherland and was directly influenced by the assumptions underlying the Sketchpad project (Packer and Jordan 2001). In the early 1970s, at Xerox PARC, Kay elaborated these assumptions by applying the developmental psychology theories of Jean Piaget and Jerome Bruner that emphasize the crucial role of images and symbolic reasoning in learning processes (Kay 2001). He published his basic insights – defining, in effect, the Learning Research Group’s agenda for the following decade – in the article ‘A Personal Computer for Children of All Ages’ (1972). In the latter he describes the Dynabook, an early vision of a tablet computer so easy to handle it could be used by children. In the mid-1970s, the synthesis and elaboration of Kay’s research eventually lead to the first known fully functional GUI, a graphical development environment for the object-oriented programming language Smalltalk (Fig. 3). The interface ran on the Xerox Alto, a computer work station with bitmap screen, keyboard, and mouse. It introduced the familiar GUI elements of overlapping windows, scrollbars and pop-up menus. Under the supervision of Kay, the Smalltalk GUI was developed at Xerox PARC by Dan Ingalis, Adele Goldberg, Larry Tesler, and others. Finally, David Canfield Smith added the icons (document, folder, dashboard references, etc.) to the graphical concept, which completed the research group’s interaction principle of ‘Windows, Icons, Menus, Pointing devices’ (Kay 1993) and eventually led to the first commercial GUI of the Xerox Star (Fig. 4.).

In his short reflection on User Interface. A Personal View (1989), Alan Kay identifies an important stream in the genealogy of user interface design: the research on human factors in industry (Kay 2001). The optimization of the machine to its human operator – and vice versa – arose as an interdisciplinary approach soon referred to as human factors and ergonomics (Murrell 1965) in the middle of the 20th century.[7] Similar to Branden Hookway, who in his recent book Interface (2014) emphasizes the Kinalog Display System, a navigational instrument of airplanes that intuitively provides the pilot with iconic information about the plane’s horizontal orientation,[8] as an important example in the history of interface design, Kay identifies the aeronautical technology of the 1950s, applying ergonomic principles, as an early attempt at user interface design. The extreme situation of a pilot dealing with a complex machine while flying at high velocity, demands for dashboards that enable an instant sensemaking for immediate decisions. Like Hookway and others, Kay considers the interface not from a merely technological perspective, but rather as a site where technological and human preconditions meet in structured moments of sensemaking and interaction. Thus he concludes:

„Therefore, let me argue that the actual dawn of user interface design first happened when computer designers finally noticed, not just that end users had functioning minds, but that a better understanding of how those minds worked would completely shift the paradigm of interaction.“ (Kay 2001: 123)

Bildschirmfoto 2016-11-23 um 20.58.25

Figure 3: Screenshot of the GUI for the object-oriented programming language ‘Smalltalk’ on the Xerox Alto workstation (ca. 1980). (Source: Computer Histoy Museum, Mountain View, CA, USA – online exhibition ‘Input & Output’; http://www.computerhistory.org/revolution/input-output/14/347/1859, accessed on April 27, 2015).

_________________

In his studies of the history of HCI, Jonathan Grudin identified human factors and ergonomics (HF&E), which expanded its research field to HCI at the end of the 1960s, as one of the three pillars in the history of research investigating the relationship between computers and its (various groups of) users (Grudin 2005, 2012). The other two pillars identified by Grudin are ‘information systems in management science’ (IS), evolving around the same time, and ‘computer-human interaction’ (CHI), the most influential work being Xerox PARC. While HF&E and IS soon focused on the improvement of efficiency, CHI, although closely collaborating with researchers of HF&E until 1980, headed towards another direction and introduced cognitive aspects to HCI research – in order to figure out how the user’s mind works, as Kay puts it.

The Learning Research Group led by Kay did not approach the question of a sufficient interface from a technological perspective, but rather started their work with examining the insights of psychology and cognitive studies. By applying developmental psychology theories, the research group started to learn about the preconditions of the user’s mind when approaching a computer interface in order to figure out how potential cognitive barriers of abstraction could be lowered. The researchers studied Jean Piaget’s theories of the understanding and learning processes of children. Piaget suggested that a child’s cognitive development does not proceed in a continuous or linear way, but rather in stages roughly tied to its age. In each stage, the child is capable of understanding certain things and correlations while it is not capable, yet, of understanding others. The developmental and educational psychologist Jerome S. Bruner elaborated Piaget’s model of stages by identifying a different mentality in each stage: enactive (kinesthetic), iconic, and symbolic (cf. Atherton 2013). Kay adopted these mindsets of learning stages to formulate a type of best-practice for practical learning, that ‘it is best to learn something kinesthetically, then iconically, and finally the intuitive knowledge will be in place that will allow the more powerful but less vivid symbolic processes to work at their strongest’ (Kay 2001: 126f). The application of Piaget’s and Bruner’s models on different domains of understanding gave Kay and his colleagues important indications on how a computer interface could connect with these domains in order to provide intuitive usability. Based on these indications, they formulated the goal for their HCI research: ‘Doing with Images makes Symbols. The slogan also implies – as did Bruner – that one should start with – be grounded in – the concrete “doing with Images”, and be carried into the more abstract “make Symbols”’ (ibid.: 128).

Bildschirmfoto 2016-11-23 um 21.00.17

Figure 4: Scans of polaroid photographs of the screen of the Xerox Star 8010 workstation, taken in 1981. Source: Digi Barn Computer Museum; http://www.digibarn.com/collections/screenshots/xerox-star-8010/index.html, accessed on April 27th, 2015).

_________________

‘Doing with Images’ points to the principle of direct manipulation, the use of the mouse to navigate with and within the graphical elements on the screen – an operational iconicity (Stjernfelt 2007) in a very direct sense of the term. The goal of the research group was to design the act of navigation as intuitively as possible. Kay explains this approach towards intuitive interaction with the example of overlapping windows, a principle that was invented out of necessity: since the screen of the Xerox Alto was too small to display all the windows necessary for a complex task next to each other, Kay proposed to overlap the windows on the screen similar to sheets of paper on a desk, placing the important window (or task) on the top (or in the front).[9] The act of clicking the window to place it on the top of the other windows and eventually performing a task references the common spatial gesture of selecting a desired object, taking it in the center of one’s visual field, examining it for the requirements needed, and possibly doing something with it. Kay explicates the implications of this kind of interaction:

„This interaction was modeless in a special sense of the word […] – the user could always get   to the next thing desired without any backing out. The contrast of the nice modeless      interactions of windows with the clumsy command syntax of most previous systems directly suggested that everything should be made modeless.“ (Kay 2001: 129)

The modeless performance of tasks through the use of common gestures and spatial operations is a vital characteristic of GUI. This is made possible by its topological structure, that is, the specific arrangement of the graphical or iconic elements on the surface represented by the GUI, where each position of an element in relation to other elements has a specific meaning. Although this surface is two-dimensional, it indicates a three-dimensional space, e.g. through the visualization of windows as overlapping layers; in contemporary interfaces used in current OS such as iOS or Android, the simulated casting of a shadow or translucency is an explicit reference to spatiality, described in the guidelines from Google and Apple as tactile, spatial or material metaphors. The operation of drag and drop, the act of moving elements from one ‘location’ on the GUI to another, is another example for such a metaphorical spatial operation. The quick perception of and intuitive interaction with a GUI is taken, to a large extent, from this topological or spatial potential. The fact that Kay and his team obviously took this potential for granted (it is not explicitly discussed or questioned in their research reports) while topologically ‘arranging’ graphical elements on the virtual surface of the screen might be explained by the elementary character of visuospatial representations and interactions we experience on a daily basis.

Apparently, there is no research in the canon of HCI that focuses on this topological basis and precondition of GUI. There is, however, a body of empirical research on spatial references with regards to the navigation with mouse and cursor in terms of hand-eye coordination (cf. MacKenzie 2013: 75f) or with regards to spatial metaphors such as the desktop metaphor, whose implications we will discuss in the next paragraph (for a practical application of spatial metaphors in interaction design see Preim and Dachselt 2010: 110 f). In fact, for a long time, especially in the 1980s, the predominant paradigm of HCI research was based on the analogy of interaction between human and computer as a ‘dialogue’ (e.g. Martin 1973) or as ‘communication’ (e.g. Smith 1980). The problem with this analogy, which evokes a reciprocal relationship between user and computer, was already pointed out by Smith (1980):

„In human-computer communication […] the dissimilarities of the components form an obstacle. When interacting with a computer, the human has a model of the task environment, a set of goals and (typically) a diffuse model or image of how the computer represents the task. […] [The computer] does not have a model of the users, their expectancies and preferred ways of reaching their goals, nor can it adept them.“ (ibid.: 25)

The conception of HCI as dialogue may have its origin in James Martin’s Design of Man-Computer Dialogues (1973), an early and influential HCI book, which, despite its notion of dialogue, warns of the ‘unfortunate tendency […] to overly anthropomorphize the computer and its capabilities’ (ibid.: 7). With the rise of graphical user interfaces and later, the World Wide Web and its ‘navigating’ and ‘exploring’ users, it is probable that spatial metaphors slowly replaced the dialogue analogy.

A common principle of GUI based on a spatial metaphor is the desktop. The graphical elements of the GUI for Smalltalk and especially the icons developed for the Xerox Star interface established a strong and, until recently, consistent reference to the classical office space: paper sheets, folders, painting devices and writing tools dominate the GUI visuality. This is how the developers of the Xerox Star GUI described their invention in 1982:

„We decided to create electronic counterparts to the physical objects in an office: paper, folders, file cabinets, mail boxes, and so on – an electronic metaphor for the office. We hoped that this would make the electronic ‘world’ seem more familiar, less alien, and require less training.“ (Smith et al. 1982: 256)

However, the desktop metaphor seems to lose its dominant position with the rise of mobile devices, relying on dashboard-like icons for instant sensemaking, or touch technology, applying simple and plain shapes as reference points for direct interaction. The metaphorical capacities of iconic and graphical representations for processes of learning and understanding the functionality of a GUI directly builds on the application of Piaget’s and Bruner’s theories to GUI design by Kay and his team, especially on the stage of iconic mentality. However, Steven Johnson (1997) points out that the success of this metaphorical or iconic approach to GUI is not the highest possible similarity to or even simulation of reality. Rather, a certain level of abstraction is necessary and vital for providing an intuitive interaction. In fact, there is a difference between interacting with the clearly arranged document icons and more or less clearly arranged windows opened on the computer screen, and dealing with the usual mess of papers, notes, books, and other objects on ones writing desk. It seems that a kind of balance is necessary when designing a ‘well-tempered’ GUI: an iconic similarity should be provided, giving the user references to the meaning of the virtual objects by referencing familiar objects in the physical world; at the same time, an operational iconicity needs to be considered by keeping the GUI visually simple and with clearly arranged elements, utilizing its topological advantages.

Since its early days, HCI research focusses on an explorative mode of ‘trial and error’ by empirically verifying or falsifying hypotheses in user experiments (cf. MacKenzie 2013). In order to complement such an approach with a stronger theoretical framework, it is necessary to examine the basic preconditions of GUI technology and understand the principles that lead to the success of this interface paradigm. A paradigm taken to a whole new level with the rise of contemporary touch technologies, enabling an even more immediate manipulation. One does not need to start such a basic research from scratch. There are other fields and disciplines dealing with similar questions of representation, operation, and interaction, such as research in diagrammatics or the (linguistic) study of metaphorical projections. Let us take a closer look at such possible answers in the next section.

Some Principles

Paradigms and origins are two perspectives of looking at interfaces – one is contemporary, one is historical or genealogical. We suggest a third one, a rather ‘timeless’ perspective, what we call principles, describing theoretical frameworks that deal with the very basic conditions of human cognition with regards not only to interfaces, but to comprehension in general. The interface, in this perspective, is not just a process, device or surface for human-machine interaction, but an instrument to see, understand and act. It is an access to a world that becomes more and more mediated through all kinds of ubiquitous interfaces. The interface in this sense might be the edge or periphery of the world and therefore a general attribute or condition of a human being, a constitution of subjects. To rephrase Wittgenstein’s statement ‘The subject does not belong to the world; rather, it is a limit of the world’ (Wittgenstein 1922: sect. 5.632), one could say that an interface does not belong to the world, but designates its limits.

Diagrams

With regards to the topological structure of representations and their interactive capacity, or operational iconicity, one may consult the insights of the research on diagrammatics, a genuinely interdisciplinary approach towards the representational and operational dimensions of diagrams.[10] Diagrammatics became increasingly popular in recent years. One explanation for this rising interest might be its reception in the aftermath of the icon turn and the spatial or topological turn in humanities – since a diagrammatic representation deals with its object in a visuospatial manner: it visualizes the relations of its object, often not visible by the naked eye, by representing structural elements on a two-dimensional surface as concrete relations. Sometimes, this mode of representation makes diagrams more effective than sentential representations such as text when it comes to the necessity of fast information retrieval or instant sensemaking. The far-reaching implications of this simple observation was described by Bertrand Russell nearly a century ago:

„There is, however, a complication about language as a method of representing a system, namely that words which mean relations are not themselves relations, but just as substantial or unsubstantial as other words. In this respect a map, for instance, is superior to language, since the fact that one place is to the west of another is represented by the fact that the corresponding place on the map is to the left of the other; that is to say, a relation is represented by a relation. But in language this is not the case […]; the word ‘precedes’, though it means a relation, is not a relation. I believe that this simple fact is at the bottom of the hopeless muddle which has prevailed in all schools of philosophy as to the nature of relations.“ (Russell 1988 [1922]: 152)

The specific spatial or topological structure of diagrammatic representations is described by the philosopher Sybille Krämer (2009: 95) as a ‘language of space’ [Sprache des Raumes], deriving its cognitive power from the two-dimensionality and simultaneity of the presented relations. Among other characteristics, Kråmer identifies the ‘two-dimensionality’ or ‘spatiality’ and the ‘directionality’ as basic modes of the presented elements. The way elements are positioned on the two-dimensional surface is not arbitrary but has a specific meaning: up or down, in center or periphery, as part of a cluster or in isolation, in close or distant proximity to other elements – relations are represented by relations. This topological organization of diagrammatic representations allows an instant sensemaking based on the specific meaning of the displayed relations. This cognitive advantage is characterized by the cognitive science approach towards diagrammatics as locational indexing of information:

„This form of indexing means that information that tends to be needed for the same inference can usually be found in adjacent locations in a diagram, so reducing the amount of search required to find the information. Further, perceptual inferences with diagrams allow the power of the highly parallel human visual system to replace more cumbersome serial logical inferences.“ (Cheng et al. 2001: 86).

The cognitive advantages of diagrammatic representations apply to the cognitive advantages of graphical user interfaces as described above. Both topologically arranged representations, they make use of locational indexing of information. Moreover, they apply basic categories of bodily orientation in space (up/down, in/out, etc.) to the abstract domain of understanding and operating with icons and other signs. There is a fundamental connection between abstract symbolic operations and concrete bodily experiences in a physical environment for understanding and creating meaning. The relation between embodied (spatial/temporal) experience, perception, and cognition has been at the center of phenomenology, at least since Edmund Husserl. Its theoretical application, together with positions from cognitive science, forms a current stream in interface studies (cf. Gallagher 2014, Harrell and Chow 2011).

Objects, artifacts, phenomena

Phenomenology after Husserl focuses on how Sachen (Husserl) – objects, artifacts, or phenomena – experienced by the human consciousness ‘shape’ perception and cognition. Its aim is, briefly, to identify basic modes and functions of consciousness. With regards to the topics discussed in this paper, Husserl’s (1991) work Ding und Raum [Thing and Space] is particularly interesting. Here Husserl investigates the meaning of objects in an environment for perceptual experience, emphasizing basic categories for any phenomenological experience, such as the relation of the human body to (moving) artifacts in an environment and the question how the body is affected by or reacts to such artifacts. Husserl describes the bodily experience and its spatial and temporal relativity as basic preconditions of the human attempt to make sense of phenomena. Martin Heidegger stressed this ‘pragmatic engagement’ (Gallagher 2014: s. p.) towards the world as our primary mode of making sense of our environment, rather than a distant observation of it.[11] The role of the body and its motor capacities in our attempt to make sense of the world is made explicit in the work of Maurice Merleau-Ponty (cf. Harrell and Chow 2011).

We cannot go much deeper into the recent history of phenomenology here. For the sake of the argument, however, it is important to briefly reflect on the adaption of phenomenological concepts in current studies of cognitive science and on its implications for interface or interaction design. As Gallagher (2014) summarizes, there is a series of evolving studies exploring the bodily or sensory-motor basis of perception and cognition – a basis that could be described as a pragmatic apriority: ‘I see and understand an object in terms of what I can do with it.’ (ibid.: s. p.) With respect to interaction and interface design, this precondition might become most clear in the concept of affordance (ibid.).[12] The cognitive and cultural effects of the mind-object relation is phrased by Gallagher with reference to the ‘extended mind hypothesis’, assuming an interdependence between technology and culture or human cognition: tools and other artifacts do not only facilitate cognitive processes but also shape the human mind in a specific way. Such a connection is already described by the media or cultural theory of Marshall McLuhan (1964) and Vilém Flusser (2015), while the idea of a non-trivial interdependence of technology and culture has been articulated at least from the philosophies of technology by Arnold Gehlen and Helmuth Plessner or the cultural studies perspective of Raymond Williams. With regards to phenomenology, Gallagher concludes that ‘the cognitive process is in some cases constituted and in some cases simply facilitated depending on the nature of the body-environment coupling’ (Gallagher 2014, s. p.).

While adapting such a phenomenological and cognitive science perspective to the design of gestural interfaces, Harrell and Chow (2011) argue that bodily motor actions also enter the level of conceptual meaning. With reference to HCI, they describe basic navigational patterns such as drag and drop or swiping a menü on a touch screen as a direct reference to spatial experiences: ‘We need to “see” something being “moved” from one location to another. In a GUI environment […] a user no longer notices details of the action after many repetitions’ (ibid., s. p.).

Surprisingly, these current efforts to apply phenomenology and cognitive science to the study of interfaces do not utilize a field of study that investigates the implications of bodily or spatial experience on a very basic conceptual level: Cognitive semantics (cf. Johnson and Lakoff 1980), from a linguistics perspective, focuses on the role of metaphorical projections in perception and cognition. Spatially organized image schemata play a special role for cognitive semantics, as pointed out by Mark Johnson (1987), a theory he later elaborated, together with George Lakoff (1999), as a philosophy of the ‘embodied mind’. We will have a closer look at Johnson’s early approach in the next section.

Metaphors

It is a linguistic paradigm that metaphorical projections are basic principles for processes of understanding and creation of meaning (cf. Johnson and Lakoff 1980, Kövecses 2010). The desktop metaphor might be the best known example of such an iconic creation of meaning in GUI design. However, it might not be the iconic or metaphoric reference to a familiar office environment what gives GUI visualizations their strength. Although there is no doubt about the advantages of providing user orientation – by e.g. designing a document on the screen as a graphical icon similar to a sheet of paper – what might be even more striking for the cognitive potential of GUI are the basic metaphorical categories as described by Johnson (1987) with regards to spatial image schemata.

Johnson provides a phenomenological analysis of spatial relations as experienced by the human body in its physical environment. Thus he identifies ‘image-schematic experiential structures and their figurative elaborations and projections onto abstract domains of understanding’ (ibid.: xxxviii). Johnson’s spatial image schemata are determined by basic spatial formations, as reflected by the containment schema, and by basic spatial operations, as reflected by the in-out schema – to mention just two examples of schemata. The containment schema is a cognitive structure determined by our bodily experience of either being in a closed space (a ‘container’) or of perceiving an object inside of another object. According to Johnson, the containment schema ‘concerns the nature of negation’ (ibid.: 40): every object in the container belongs to the same category, objects outside of the container do not, ‘so that whatever is within the category is in the appropriate container’ (ibid.). The in-out schema is closely connected to containment, but rather relating to the operation of taking something in or out of a container. These schemata are metaphorically projected to abstract domains of understanding in order to structure an experience and to make sense out of it, as can be seen in metaphorical expressions such as ‘entering into a conversation’ which makes use of the in-out schema (ibid. 31).

The containment schema and the in-out schema are two excellent examples of schemata at work in the interaction with a GUI. The first one applies to documents in a folder or to objects in a window. Here the graphical representation makes use of the containment schema and thus allows to categorize elements not only by spatial indexing but also by separating the elements from elements of another category (by putting them to a specific folder or window). The act of putting the object in its graphical ‘container’, by using the drag and drop or copy and paste functionality, involves, on the other hand, the in-out schema. Finally, the close connection between both schemata and GUI principles becomes particularly clear in a research report by the Xerox Star GUI developers from 1982, where the windows principle is metaphorically explained:

„You can ‘open’ an icon to deal with what it represents. This enables you to read documents,   inspect the content of folders and file drawers […]. When opened, an icon expands into a   larger form called ‘window’, which displays the icon’s contents. Windows are the principal mechanism for displaying and manipulating information.“ (Smith et al. 1982: 256)

The assumption that there are basic metaphorical projections at work, and that the prominent desktop metaphor might not be that powerful after all, gives a whole new dimension to the importance of topological principles for GUI design. It is supported by current GUI developments, e.g. for mobile devices or for the operation system Microsoft Windows 8, optimized for touch interaction, where graphical or iconic references to an office environment are drastically reduced (Fig. 5).[13] The adaption of the cognitive semantics approach – investigating the influence of bodily experience and spatial categories on domains of abstract conceptual understanding – to a phenomenological and cognitive science approach towards interface design could be fruitful for both fields of study.

Bildschirmfoto 2016-11-23 um 21.01.46

Figure 5: Clear and flat design of the GUI Microsoft Windows 8, 2015. (Source: Microsoft Windows 8 product presentation; http://windows.microsoft.com/en-us/windows/home, accessed on April 28th, 2015).

_________________

Conclusions

We suggested three different approaches to the interface: a contemporary one, looking at the paradigms of current interfaces; a historical one, examining the contingency and conditions of the current developments; and a principal one, looking at the very basic conditions of human perception. Although each approach might be different in its focus and vocabulary, they appear to be deeply intertwined. The current paradigms are obviously related to their genealogical predecessors, and represent perspectives which all relate to debates about basic human cognition. By juxtaposing these three perspectives, it has become clear that a comprehensive theory of the interface as surface, process and constitutive nexus has to take into consideration not only one, but all three aspects and therefore can not be limited to any one discipline. Rather, it demands an interdisciplinary approach that draws from semiotics, diagrammatics, cognitive science, cognitive semantics, phenomenology, ergonomics, and which also considers philosophical and historical debates about perception and cognition, economics, technology, engineering, architecture and many more. It seems reasonable to combine these perspectives into a truly multidisciplinary approach in order to understand interfaces in their dynamic developments – and to make them visible again, especially when they are about to disappear and to withdraw themselves from our perception. We strove to contribute to the already existing interdisciplinary discourse on interfaces, hoping that more comprehensive approaches will emerge from different disciplinary backgrounds, that combine technological, sociological, economical and aesthetical aspects into a perspective that considers interfaces as truly dynamic, cultural phenomena.

 

REFERENCES

Agamben, Giorgio 2009. The Signature of All Things. On Method. New York: Zone Books.

Atherton, James S. 2013. Learning and Teaching; Piaget’s developmental theory. Available from: http://www.learningandteaching.info/learning/piaget.htm [accessed April 29, 2015]

Bauer, Matthias and Christoph Ernst 2011. Diagrammatik. Einführung in ein kultur- und medienwissenschaftliches Forschungsfeld. Bielefeld: Transcript.

Bender, John and Michael Marrinan 2010. The Culture of Diagram. Stanford, CA: Stanford University Press.

Blackwell, Alan and Kerry Rodden 2003. Preface to this Electronic Edition. In: Sutherland, Ivan E. Sketchpad: A man-machine graphical communication system (Technical Report 574 ). Cambridge, MA: University of Cambridge Computer Laboratory, 3-5. Available from: http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-574.pdf. [accessed April 20, 2015]

Cheng, Peter C.-H., Ric K. Lowe and Mike Scaife 2001. Cognitive Science Approaches To Understanding Diagrammatic Representations. In: Alan F. Blackwell (ed) Thinking with diagrams. Dordrecht, Boston and London: Kluwer Academic Publishers, 79-94.

Dagognet, François 1982. Faces, Surfaces, and Interfaces (Problèmes et controverses). Paris: Vrin.

Drucker, Johanna 2011. Humanities Approaches to Interface Theory. Culture Machine 12: 1-20.

Flusser, Vilém 2015. Rückschlag der Werkzeuge auf das Bewusstsein. In Graham Harman, Die Rache der Oberfläche: Heidegger, McLuhan, Greenberg. Cologne: Walther König, 28-32.

Flusser, Vilém 1973. Line and Surface. Main Currents in Modern Thought 29 (3): 100-106.

Fogel, Lawrence J. 1959. A New Concept: The Kinalog Display System. Human Factors: The Journal of the Human Factors and Ergonomics Society 1 (2): 30-37.

Gallagher, Shaun 2014. Phneomenology. In: Mads Soegaard and Rikke Friis Dam (ed) The Encyclopedia of Human-Computer Interaction. Aarhus:  / Interaction Design Foundation. Available from: https://www.interaction-design.org/encyclopedia/phenomenology.html [accessed August 3, 2015].

Galloway, Alexander 2012. The Interface Effect. Malden, MA: Polity Press.

Grudin, Jonathan 2005. Three Faces of Human-Computer Interaction. IEEE Annals of the History of Computing 27 (4): 46-62.

Grudin, Jonathan 2012. A Moving Target – The Evolution of Human-Computer Interaction. In: Julie A. Jacko (ed) Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications. Boca Raton, FL: CRC Press, xxvii-lx.

Hadler, Florian and Joachim Haupt (eds) 2016. Interface Critique. Berlin: Kadmos.

Harrel, D. Fox and Kenn K. N. Chow 2011. Enduring Interaction: Ann Approach to Analysis and Design of Animated Gestural Interfaces in Creative Computing Systems. Proceedings of the 8th ACM Conference on Creativity and Cognition, November 3-6, 2011: 95-104.

Hookway, Branden 2014. Interface. Cambridge, MA and London: MIT Press.

Husserl, Edmund 1991. Ding und Raum. Hamburg: Felix Meiner.

Irrgang, Daniel and Florian Hadler 2014. Nonlinearity, Multilinearity, Simultaneity. Notes on Epistemological Structures. In: Hudson Moura, Ricardo Sternberg, Regina Cunha, Cecilia Queiroz and Martin Zeilinger (eds) Interactive Narratives, New Media, and Social Engagement. Toronto: University of Toronto, CD-ROM.

Johnson, Mark 1987. The Body in the Mind. The Bodily Basis of Meaning, Imagination, and Reason. Chicago and London: The University of Chicago Press.

Johnson, Steven 1997. Interface Culture. How New Technology Transforms the Way We Create & Communicate. San Francisco: Harper.

Kay, Alan C. 1993. The Early History of Smalltalk. ACM SIGPLAN Notices 28 (3): 69-96.

Kay, Alan C. 2001 [1990]. User Interface. A Personal View. In: Randall Packer and Ken Jordan (eds) multiMEDIA. From Wagner to virtual reality. New York, NY: W.W. Norton, 121-131.

Kovecses, Zoltàn 2010. Metaphor. New York, NY: Oxford University Press.

Kråmer, Sybille 2009. Operative Bildlichkeit. Von der “Grammatologie” zu einer “Diagrammatologie”? Reflexionen über erkennendes “Sehen”. In: Martina Heßler and Dieter Mersch (eds) Logik des Bildlichen. Zur Kritik der ikonischen Vernunft. Bielefeld: Transcript, 94-122.

Kråmer, Sybille 2012. Punkt, Strich, Fläche. Von der Schriftbildlichkeit zur Diagrammatik. In: Sybille Kråmer, Eva Cancik-Kirschbaum and Rainer Totzke (eds) Schriftbildlichkeit. Wahrnehmbarkeit, Materialität und Operativität von Notationen. Berlin: Akademie, 79-100.

Kuhn, Thomas S. 1962. The Structure of Scientific Revolutions. Chicago: University of Chicago Press

Lakoff, George and Mark Johnson 1980. Metaphors We Live By. London: The University of Chicago Press.

Lakoff, George and Mark Johnson 1999. Philosophy in the Flesh. The Embodied Mind and Its Challenge to Western Thought. New York, NY: Basic Books.

MacKenzie, I. Scott 2013. Human-Computer Interaction. An Empirical Research Perspective. Waltham, MA: Morgan Kaufmann.

Martin, James 1973. Design of Man-Computer Dialogues. Englewood Cliffs, NJ: Prentice-Hall.

McLuhan, Marshall 1964. Understanding Media. The Extension of Man. New York, NY: McGraw-Hill.

Moran, Thomas P., Stuart K. Card and Allen Newell 1983. The Psychology of Human-Computer Interaction. Hillsdale, NJ and London: L. Erlbaum.

Murrell, Hywel 1965. Ergonomics. Man in his Working Environment. London: Chapman and Hall.

Packer, Randall and Ken Jordan 2001. Introduction to Alan Kay. In: Randall Packer and Ken Jordan (eds) multiMEDIA. From Wagner to virtual reality. New York, NY: W.W. Norton, 122.

Posner, Roland 2009. Diagrammatische Zeichen. Zeitschrift für Semiotik 31 (3-4): 213-229.

Russell, Bertrand 1988. Vagueness (Read before the Jowett Society, Oxford, November 25, 1922). In: John G. Slater (ed) Bertrand Russell. Collected Papers, vol. 9. London and Boston: Unwin Hyman, 147-154.

Smith, David C., Charles Irby, Ralph Kimball and Eric Harslem 1982. The Star User Interface: An Overview, in: AFIPS ‘82 (Proceedings of the June 7-10, 1982 National Computer Conference): 515-528.

Smith, Hugh T. and T. R. G. Green (eds) 1980. Human Interaction With Computers. London et al.: Academic Press.

Smith, Hugh T. 1980. Human-Computer Communication. In: Hugh T. Smith and T. R. G. Green (eds) Human Interaction With Computers. London et al.: Academic Press, 5-38.

Stjernfelt, Frederik 2007. Diagrammatology. An Investigation on the Borderlines of Phenomenology, Ontology, and Semiotics. Dordrecht: Springer.

Sutherland, Ivan E. 2003 [1963]. Sketchpad: A man-machine graphical communication system (Technical Report 574 ). Cambridge, MA: University of Cambridge Computer Laboratory. Available from: http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-574.pdf. [accessed April 20, 2015]

Weibel, Peter 2011. The Art of Interface Technology. In: Hans H. Diebner, Timothy Druckrey and Peter Weibel (eds) Sciences of the Interfaces. Karlsruhe: ZKM, 272-281.

Wittgenstein, Ludwig 1961 [1922]. Tractatus Logico-Philosophicus. London: Routledge & Kegan Paul.

Zielinski, Siegfried 1997. Künste und Apparate. Plädoyer für die Dramatisierung einer Schnittstelle. In: Internationales Forum für Gestaltung Ulm (ed) Mensch, Masse, Medien. Interaktion oder Manipulation. Frankfurt a. M.: Anabas, 113-128.

 

Footnotes:

[1] Apple iOS Developer Library, iOS Developer Guidelines: Design Principles (https://developer.apple.com/library/ios/documentation/UserExperience/Conceptual/MobileHIG/Principles.html#//apple_ref/doc/uid/TP40006556-CH4-SW1, accessed on April 26, 2015).

[2] Google Material Design, Introduction: Principles (http://www.google.com/design/spec/material-design/introduction.html#introduction-principles, accessed on April 26, 2015).

[3] Google Material Design, Introduction: Principles (http://www.google.com/design/spec/material-design/introduction.html#introduction-principles, accessed on April 26, 2015).

[4] Apple iOS Developer Library, iOS Developer Guidelines: Design Principles (https://developer.apple.com/library/ios/documentation/UserExperience/Conceptual/MobileHIG/Principles.html#//apple_ref/doc/uid/TP40006556-CH4-SW1, accessed on April 26, 2015).

[5] The compendium The Psychology of Human-Computer Interaction by Thomas P. Moran, Stuart K. Card, and Allen Newell is considered as the groundbreaking publication establishing a user-centered view in computer science and related fields. (cf. Grudin 2005) However, investigating the ‘human factor’ in computing did already play a role in research of ergonomics and human factors since the 1950s. An early explicit articulation of the user in the scope of ‘human interaction with computers’ can be found in the anthology with the same title by H. T. Smith and T. R. G. Green (1980).

[6] The aspect of feedback is central when it comes to HCI in general. In Human Interaction With Computers, H. T. Smith (1980) points out the attraction of feedback experienced by the user: ‘It is the very tightness of the person-computer feedback loop that sucks in the user […]’ (ibid.: 12). The user does not even need to understand the computer system tasks at work, ‘[a]s long as there is a unique and observable response of the system to each user input […]’ (ibid. 26).

[7] The evolving of research groups investigating human factors in industry, which eventually lead to the foundation of the ‘Human Research Society’ in 1949, soon renamed to ‘Ergonomics Research Society’ in 1950 (Murrell 1965), can be discussed as a product of two dominant capitalist discourses of the 20th century: economic optimization and, since the beginning of the 1950s, cybernetics. However, the growing importance of ergonomics as research on the optimization of man-machine processes in order to deal with the ever increasing structural complexity of the apparatuses might mark an important milestone in the formation of media ecologies as discussed by Neil Postman in the 1970s or by Erich Hörl today.

[8] Lawrence J. Fogel proposed the Kinalog Display System in Human Factors: The Journal of the Human Factors and Ergonomics Society in 1959.

[9] The analogy of sheet staples is already quite close to the famous desktop metaphor that became a central element of GUI standards. Strangely, the principle of overlapping windows was not adapted for the Xerox Star. Here, windows can not overlap but have to be placed next to each other, limiting the number of parallely open windows on the screen significantly. The principle of overlapping windows was later adapted by Apple for the Lisa and Macintosh GUI.

[10] The bulk of the studies in diagrammatics, in disciplines such as media or cultural studies, art history, philosophy, semiotics, logics, or cognitive science, is (at least in parts) informed by a semiotics perspective: As a sub-group of the icon (Peirce), diagrams as signs signify their object not by means of visual similarity such as form or color, but by structural correspondence. Moreover, studies in diagrammatics are not only about a specific strategy of visualization. Rather, they also examine the operational iconicity (Krämer 2009 & 2012, Stjernfelt 2007) of diagrams or, in other words, what one can do with a diagram in order to learn something new about its object. This diagrammatically mediated thought experiment, aiming at an imaginary change of the object by rearranging and modifying the diagram (Bauer and Ernst 2011, Bender 2010), can be described as a specific mode of ‘behavior in rehearsal’ [‘Probehandeln’ (Posner 2009: 2014)] wherein the diagrammatic representation plays the role of a catalyzer.

[11] Heidegger’s Zuhandenheit (ready-to-hand), the ’vanishing’ of a well-working tool while using it (e. g. the carpenter that is not constantly aware of the hammer while using it – it gets an inherent part of the practice of nailing), or Vorhandenheit (present-at-hand), the object which shows itself resisting ist use (e. g. the hammer is broken or badly designed and difficult to use), are two categories which should be considered in studies about the usability of interfaces and in design processes in general.

[12] An object design intuitively implying its use – e. g. a button element in a GUI, designed to show the same sensory (visual) characteristics as a “physical” button which can be pressed down by ones thumb. Design concepts relying on affordance form the basis of skeuomorphism, especially in interface design: Skeuomorphic design elements do not have a specific function except implying or signifying a specific use of interface elements (e. g. a music production application emulating a synthesizer which is designed similar to an analogue Moog synthesizer, including patch cables and wooden chassis). On an analytical level, design strategies of affordance or skeumorphism relate to McLuhans hypothesis that every new medium includes (aspects of) its predecessors („the older medium”). (McLuhan 1964) Here we disagree with McLuhan: Such design strategies could be described as practices of adaption (of technologies, signs, materiality, practices, etc.) rather then, as McLuhan stated, a process of replacing old media (in other words: media convergence theory vs. media extinction theory).

[13] The fact that users of Windows 8 complained about the new very clear and flat design, which generated a lot of media attention, might be explained with the loss of graphical references ingrained in usage by the classical Windows environments, which resulted in cognitive dissonances while coping with the new graphical structure of Windows 8.

Whole paper as PDF

Copyright © 2015 Florian Hadler & Daniel Irrgang. Licenced under the

Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd). Available at https://creativecommons.org/licenses/by-nc-nd/4.0/

This text first appeared in:

Punctum, 1(1): 7-25, 2015
DOI: 10.18680/hss.2015.0002