More than a feeling

This is a draft of a column I wrote for the ACM’s interactions magazine. It will appear late 2011.

_______________________________________________________

Mildly irritated. Frustrated. Somewhat annoyed. Profoundly agitated.

This was the trajectory of feeling I experienced this afternoon as I ascended and descended a customer service phone tree in the hopes of reaching a human who could (or would) answer my question. As I reached a flushed state of agitation, I had a meta-moment. I was catapulted out of my immediate experience into a view onto self from a distance. You are seeing red . Your reaction is unreasonable. Stop. Get a grip. Take a deep breath. Hang up the phone. [1]

As a result of this incident, in the longstanding tradition of armchair philosophy I started to ponder: What are feeling, affect, emotion? Was I really being unreasonable having an emotional reaction, by feeling agitated? What are reasonable and unreasonable in this context? Would others have been swept away with agitation and simply shouted at the person who finally answered? Would I have felt better if I had done that? Aside from hiring more people, what, if anything, could the company have done to prevent this ascension to annoyance? What was clear to me was that a tinny rendition of Antonio Vivaldi’s Four Seasons was not cutting it in the keep-Elizabeth-calm department.

For decades human computer interaction professionals have been thinking about cognition and how to present choices in ways that are intuitively obvious, and we’ve worried about how to measure emotions like frustration for almost as long. We know that perceived options change and that people make different choices among the same set of options depending on their emotional state.

Well, I pondered, perhaps we can detect subtle cues about someone’s state, and based on that, present more copacetic options and different interaction experiences. It is postulated that people have visceral, pre-cognitive positive or negative reactions to things [2]. Perhaps someone’s state can be detected before the person themself realizes how they feel? Perhaps, there could be sensors in my phone handset that detect my emotional state and initiate cheer-up, calm-down or fall in love sequences? Evidence suggests that visceral responses aren’t just in response to an artifact or situation; they can be modulated by social factors. “Affective priming” studies suggest that others’ emotional states can affect our decision making even if we are unaware of it: people take more risks after being subliminally exposed to smiling faces than to frowning ones [3]. If the service I was calling had subliminally played laughter down the phone line, just maybe my ascension to agitation would have been prevented.

As I idly entertained myself with these technospeculative reveries, I came across a discussion of innovations in sensor technologies which can reliably tell us what people are feeling, can accurately map their emotional state. Frankly, I had cast aside my reveries as overly deterministic and somewhat creepy, the stuff of potions and hexes, and way too much of a wicked design/engineering problem [4]. Such a scenario, I reasoned, requires accurate detection, a good model of emotion and mood and it’s impact on thought and action, and an understanding of how the person’s reactions and their surrounding context are going to interfere with any initiated interactive sequence. I decided to turn to research to see if there exist grounds for these optimistic, techno-detection-of-emotion, media narratives.

The plethora of disciplines interested in emotion (for example, biology, linguistics, psychology, sociology, cultural studies, anthropology, design, literary studies, performing arts…..) suggest many have pondered: What is the relationship between emotion, thought and action? How can we detect and measure emotion? How can we assess the influence of emotion on thinking and action? Some conclude that emotions are an annoying flaw in an otherwise perfect reasoning system. Others assert that emotion/feeling and reason are intertwined, that it is futile to assert emotion-free cognition is even possible. In 1994, António Damásio argued from neuroscientific evidence that emotions play a critical role in cognition and provide the scaffolding for the construction of social cognition and underlie all human consciousness [5]. Armed with Damásio’s perspective, I delved deeper into theories on emotion, hoping to find more evidence as the nature and measurement of emotional influence on human behavior. The theories can be crudely bucketed into three types:

• naturalistic theories, which maintain that emotions are products of natural processes which are independent of social norms and conscious interpretation; they result from hormones, neuro-muscular feedback from facial expressions, and genetic mechanisms,

• interactionist theories, which acknowledge culture’s importance yet suggest that enduring and universal biological mechanisms provide for core emotional features (social aspects are cast as derivative, contingent, and variable)

and

• social constructionist theories, which maintain that emotions depend on a social consciousness concerning when, where, and what to feel as well as when, where, and how to act.

There are more or less extreme versions of each; the more extreme flaunt clear ideological stances. For example, naturalistic theorists like Robert Zajonc associate emotions with spontaneity, creativity, sociability and passion, they are more authentic and are preferred over trained, ‘cold’ reason [6]. A whole research agenda exists to illustrate that we process emotional cues (preferences) separately from making reason-driven decisions (inferences) . Much of the interactionist perspective is rooted in the work of Charles Darwin, the father of evolutionary theory, whose view on emotions is that they are gestures that hark back to basic human behaviours—love is the vestige of copulation, angry teeth-baring a vestige of biting. Returning to my own telephonic experience, I confess I did not bare my teeth at the handset. However, had I done so, my Darwininan ancestral counterpart would simply have bitten it.

Other researchers focus a great deal on responses in the sympathetic nervous system; elevated activity is known as arousal, and arousal is deemed positive or negative. Reading articles in this tradition, I have to say there seems to be a conflation of body arousal with emotion and feeling. In my view, arousal tells you something happened, but not how the individual felt, what emotion they experienced nor how those around behaved in response to the same incident; these factors are likely to have a major influence on the emotion as felt and whether it persists or not. Moving from detection of usually non-visible sensory nervous system arousal to visually available body reactions, Paul Ekman’s work on emotion and facial expression is perhaps the most famous work from the interactionist camp. Ekman suggests there are a few core or “basic”, universal, biologically generated, emotion-revealing facial expressions—these reflect anger, disgust, fear, sadness, surprise and joy [7]. Ekman acknowledges that most of the elicitors and expressions (“display rules”) of emotions are socially learned and can thus vary.

Ekman’s allusion to ‘variance’ is also interesting, of course. People modulate their behaviors quite substantially depending on social context—raising children is largely about trying to infuse a framework for socially appropriate emotional displays given different social contexts. Modification of spontaneous physical expressions is part of enculturation, and cultures vary. Anthropologists like Dewight Middleton propose different cultures have ‘emotional styles’, which not only guide what we consider to be reasonable feelings for any situation but also how to enact that experienced feeling [8]. These ways of enacting feeling are part of what are called the ‘techniques of the body’, a concept introduced by Marcel Mauss in 1973 to describe highly developed body actions that embody aspects of a given culture—how one walks, eats and emotes are learned through explicit instruction, through subtle approval/disapproval cues and through postural mimicry, and they can reflect one’s gender and class. Our physical expression of emotion including our facial expression is modulated by the company we are in, a problem for systems that claim to detect anything but the most extreme of emotional displays with any degree of universal reliability [9].

Mapping the physiological to the psychological to the social and cultural is important; however, to replace biological determinism with cultural determinism is not satisfactory either. Social constructionists assert that there are two kinds or classes of human emotions; one class is universal and has analogues in animals and can be seen in human infants (e.g., joy, sadness, fear), and the other is adult human emotions which are culturally variable. Through enculturation emotions lose any ‘natural’ or ‘spontaneous’ quality because they are mediated by culturally located social consciousness. Arlie Russell Hochschild’s work is a good illustration of this [10]. Her position is that society and culture provide “feeling rules” and prototypes of feeling that focus on particular kinds of emotion . In the West we feel grief at funerals but we’re supposed to look happy at parties no matter how dull they are; of course, we are perfectly able to embrace other ways of behaving at funerals and parties should we choose to do so. In addition to modeling appropriate postures and behaviors as discussed above, people verbally communicate culturally sanctioned or appropriate reactions in saying things like “You must be delighted”, “Don’t fret over that” or “You should be absolutely incensed” when others narrate incidents. Adult emotions, Hochschild says, involve personal appraisal of things like attachment (to things, situations, people and outcomes), agency and approval/disapproval. Such appraisal involves other entities: for example, envy is a name for noting an attachment or desire for something, and noting that another has it; sadness focuses on something or someone liked/loved that is not available. Hochchild addresses a range of complex emotional states including sadness and grief, nostalgia, depression, frustration, anger, fear, indignation, disgust, contempt, guilt, anguish, love, envy, jealousy, shame and anxiety. Familiar to most adults, these emotions do not map easily to reductive concepts like arousal or positive/negative affect. Rather, they require that we consider the personal, social and cultural locatedness of feeling and emotion. Hochschild also introduces the very useful concept of “emotional labor”—the effort to try to actually feel the “right” feeling for a situation, and to try to induce the “right” feeling in others [11]. Appraisal and emotional labor are not just about the current situation or the past; they also involve imagined futures. Personally, my breaking point on the phone tree debarcle was when I realized I could not be relied upon to engage in the emotional labour of masking my irritation. As a result, my concern was that I’d say something I’d regret when a person finally came on the phone. Imagining the future-self being embarrassed is a great way to restrain the inner jackass.

Walter Benjamin in Illuminations wrote “In every case the storyteller is a man who has counsel for his readers.” It’s true. My main message is not, however, ‘avoid phone trees at all costs’ ; that’s my secondary message [12]. Rather, as content and communication tools proliferate and we design people’s experiences with and through those tools, it behooves us to know why and how people react emotionally and express, perceive and communicate their feelings, not just verbally but also pre- or para-verbally. We need to be sensitive to norms in different cultures, and what happens when we design across them. And, while automated detection of arousal and affect are a very promising for connecting with people who have little insight into their own feelings or cannot communicate their emotions easily, let’s please be wary of reducing the entirety of human emotional experience to simplistic, deterministic, electrochemical models.

Cycling back, to solve the phone-tree problem, my solution would not be to save up for an fmri scanner, nor to measure my galvanic skin response, nor have a webcam trained to my face to detect my expression. Without claiming any universality but knowing myself pretty well, I’d hook something up to listen to my voice, my intonation [13]. Following the old saying, “It’s not what you said, it’s how you said it”, tone of voice includes the timbre, rhythm, loudness, breathiness and hoarseness of how something is uttered. Personally, I seem to follow the general rules: softer tones and pitches are associated with friendliness, higher tones and pitches signal upbeat and happy, and clipped and louder tones signal irritation. Let’s suffice to say when I finally hung up the phone, I was not intoning in a kittenesque fashion, nor in a light, cheerful tone. I was gearing up to bark.

NOTES

[1] In their 1980 book Metaphors We Live By, they suggest the body is directly reflected in metaphors we use. They suggest that “our concept of anger is embodied via the autonomic nervous system and that the conceptual metaphors and metonymy used in understanding and are by no means arbitrary: instead they are motivated by a physiology”. Feeling anger is “seeing red” because it reflects bodily reactions—when people get angry, as a result of soaring cortisol levels, they heat up and often faces flush red.

[2] See Don Norman’s Emotional Design for a lovely exposition on this.

[3] People who are asked to make more complex financial decisions—for example, whether to gamble $1 for $.50 chance of winning $2.50—are primed with subliminal happy faces or sad faces. Those shown happy faces were likely to choose the investment than people primed with angry faces.

[4] ““Wicked problem” is a phrase originally used in social planning to describe a problem that is difficult or impossible to solve because of incomplete, contradictory, and changing requirements that are often difficult to recognize. Moreover, because of complex interdependencies, the effort to solve one aspect of a wicked problem may reveal or create other problems.” See http://en.wikipedia.org/wiki/Wicked_problem

[5] Antonio Demasio (1994) Descartes’ Error: Emotion, Reason, and the Human Brain

[6] In his 1980 American Psychologist paper, “Feeling and Thinking: Preferences need no Inferences”, Zajonc asserts “affect dominates social interaction, and it is the major currency in which social intercourse is transacted.”

[7] Neither Darwin nor Ekman nor their followers, I note, have managed to give us any advice on emotion recognition in the world of cosmetic modification. The American Society for Aesthetic Plastic Surgery reports over 2 million procedures in 2010 using Botulinum Toxin Type A (the active ingredient of products like Botox) to reduce facial lines. Eyebrow raising and frowning, clear signals of interest and affect, are not what they used to be when this research thread was first hatched.

[8] Dewight Middleton, Emotional Style: The Cultural Ordering of Emotions, 2009, Ethos, American Anthropological Association

[9] I also note that, to date, the most successful biofeedback systems present motivated people with data that allow them annotate and moderate their own behaviours; they don’t presume to mimic the careful collaborative choreography of emotions as we might achieve with a sensitive human friend. See research into the Quantified Self.

[10] The Managed Heart: The Commercialization of Human Feeling (1982) Berkeley: The University of California Press.

[11] See also http://en.wikipedia.org/wiki/Arlie_Russell_Hochschild.

[12] If you are going to use one, refer to gethuman.com first; it’ll tell you what to do to get to a human operator in the most expeditious of ways.

[13] I note that well-publicised affect detecting companies like Affectiva (whose technology is based on research conducted in Rosalind Picard’s Affective Computing Lab at MIT) do not use voice, but rather focus on the reactions of the sympathetic nervous system and facial expressions. This is, in large part, because they technologies were originally designed for people for whom speech is difficult; e,g, those who suffer from autism or babies. See http://www.affectiva.com/q-sensor/#videos.

Missing the point in gesture-based interaction

This is a draft of a column I wrote for the ACM’s interactions magazine. It will appear mid 2011.

_____________________________________________________

“Zhège” she said, pointing emphatically at the top right of her iPhone screen.  She leaned further into the gap between the passenger and driver seat of the taxi. Then, lifting her head, she pointed forward through the windscreen in a direction that, I assumed, was where we were hoping soon to be headed.

The taxi driver looked at her quizzically.

Undeterred, she repeated the motion, accompanied by a slower, more carefully enunciated rendition of the word: “zhège”. This time she added a new motion. She pointed at the bottom left of her iPhone screen, at herself, at the taxi driver himself, and then at the ground below us. Balletic though this motion was, it did not reduce the look of confusion on the driver’s face.

Gently taking the device from her hand, he studied the screen. A moment later, his expression changed. He smiled and nodded. He stretched out the index finger on his right hand, pointed to the location on the screen she had isolated, and said “zhège”. He handed the device back to her, flipped on the meter, and grasped the steering wheel. A second later we accelerated out of the taxi rank. He had understood the point of her point(s).

My traveling partner, Shelly, and I know precisely 6 words of Chinese. ‘Zhège’ is one of them. We cannot actually pronounce any of the words we know with any consistency. Sometimes, people nod in understanding. Mostly they don’t. However, the scenario I painted above is how we navigated two weeks in China. The word ‘navigated’ is intentional–it is about the physical and conceptual traversal of options. We navigated space and location. We navigated food. We navigated products. We navigated shopping locations, shopping possibilities and shopping traps (always a concern for tourists, wherever they may be). We did all this navigation speechless; owing to our linguistic ignorance, we accomplished it by pointing. We pointed at menus. We pointed at paper and digital maps. We pointed at applications on our phone screens. We pointed at ourselves. We pointed at desired products. We pointed in space toward unknown distant locations… Basically, we pointed our way to just about all we needed and/or wanted, and we got our way around Beijing with surprisingly few troubles.

Pointing of this kind is a deictic gesture. The Wikipedia definition for ‘deixis’ is the “phenomenon wherein understanding the meaning of certain words and phrases in an utterance requires contextual information. Words are deictic if their semantic meaning is fixed but their denotational meaning varies depending on time and/or place.” [1] In simpler language, if you point and say “this”, what “this” refers to is fixed to the thing at which you are pointing. In the scenario above, it was the location on a map where we wanted to go. Linguists, anthropologists, psychologists and computer scientists have chewed deixis over for decades, examining when the words “this” and “that” are uttered, how they function in effective communication and what happens when misunderstandings occur. In his book Lectures on Deixis, Charles Fillmore describes deixis as “lexical items and grammatical forms which can be interpreted only when the sentences in which they occur are understood as being anchored in some social context, that context defined in such a way as to identify the participants in the communication act, their location in space, and the time during which the communication act is performed”. Stephen Levinsohn in his 1983 book, Pragmatics, states that deixis is “the single most obvious way in which the relationship between language and context is reflected”.

Pointing does not necessitate an index finger. If conversants are savvy to each other’s body movements–that is, their body ‘language’–it is possible to point with a minute flicker of the eyes. A twitch can be an indicator of where to look for those who are tuned in to the signals. Arguably, the better you know someone, the more likely you will pick up on subtle cues because of well-trodden interactional synchrony. But even with unfamiliar others, where there is no shared culture or shared experience, human beings as a species are surprisingly good at seeing what others are orienting toward, even when the gesture is not as obvious as an index finger jabbing the air. Perhaps it is because we are a fundamentally social species with all the nosiness that entails; we love to observe what others are up to, including what they are turning their attention toward. Try it out sometime, stop in the street and just point. See how many people stop and look in the direction at which you are pointing.

Within the field of human-computer interaction–HCI–much of the research on pointing has been done in the context of remote collaboration and telematics. However, pointing has been grabbing my interest of late as a result of a flurry of recent conversations where it has been suggested that we are on the brink of a gestural revolution in HCI. In human-device/application interaction, deictic pointing establishes the identity and/or location of an object within an application domain. Pointing may be used in conjunction with speech input–but not necessarily. Pointing does not necessarily imply touch, although touch-based gestural interaction is increasingly familiar to us as we swipe, shake, slide, pinch and poke our way around our applications. Pointing can be a touch-less, directive gesture, where what is denoted is determined through use of cameras and/or sensors. Most people’s first exposure to this kind of touch-less gesture-based was when Tom Cruise swatted information around by swiping his arms through space in the 2002 film Minority Report. However, while science fiction interfaces often inspire innovations in technology–it is well worth watching presentations by Nathan Shedroff and Chris Noessel and by Mark Coleran on the relationship between science fiction and the design of non-fiction interfaces, devices and systems–there really wasn’t anything innovative in the 2002 Minority Report cinematic rendition of gesture-based interaction, nor in John Underkoffler’s [1] presentation of the non-fiction version of it, g-speak, in a TED Talk in 2010. Long before this TED talk, Richard Bolt created the ”Put that there” system in 1980 (demoed at the CHI conference in 1984). In 1983 Gary Grimes at Bell Laboratories patented the first glove that recognized gestures, the “Digital Data Entry Glove”. Pierre Wellner’s work in the early 1990’s explored desktop based gesture based interaction and Thomas Zimmerman and colleagues used gestures to identify objects in virtual worlds using the VPL DataGlove in the mid 1980’s.

This is not to undermine the importance of Underkoffler’s demonstration; gesture-based interfaces are now more affordable and more robust than these early laboratory prototypes. Indeed, consumers are experiencing the possibilities everyday. Devices like the Nintendo Wii and the Kinect for Xbox 360 system from Microsoft are driving consumer exuberance and enthusiasm for the idea that digital information swatting by arm swinging is around the corner. Anecdotally, an evening stroll around my neighbourhood over a holiday weekend will reveal that a lot of people are spending their evenings jumping around gesticulating and gesturing wildly at large TV screens, trying to beat their friends at flailing.

There is still much research to be done here, however. The technologies, their usability but also the conceptual design space needs exploration. For example, current informal narratives around gesture-based computing regularly suggest that gesture-based interactions are more “natural” than other input methods. But, I wonder, what is “natural”? When I ask people this, I usually I get two answers: better for the body and/or simpler to learn and use. One could call these physical and cognitive ergonomics. Frankly, I am not sure I buy either of these yet for the landscape of current technologies. I still feel constrained and find myself repeating micro actions with current gesture-based interfaces. Flicking the wrist to control the Wii does not feel “natural” to me, neither in terms of my body nor in terms of the simulated activity in which I am engaged. Getting the exact motion on any of these systems feels like cognitive work too. We may indeed have species specific and genetic predispositions to being able to pick up certain movements more easily than others, but that doesn’t make most physical skills “natural” as in “effortless”. Actually, with the exception of lying on my couch gorging on chocolate biscuits, I am not sure anything feels very natural to me. I used to be pretty good at the movements for DDR (Dance Dance Revolution) but I would not claim these are movements in any sense natural, and these skills were hard won with hours of practice. It took hours of stomping in place before stomping felt “natural”. Postures and motions that some of my more nimble friends call “simple” and “natural” require focused concentration for me.  “Natural” also sometimes gets used to imply physical skill transfer from one context of execution to another. Not so. Although there is a metaphoric or inspired-by relationship to the ‘real’ physical work counterparts, with the Wii, I know I can win a marimba dancing competition by sitting on the sofa twitching and I can scuba-dive around reefs while lying on the floor more or less motionless, twitching my wrist.

An occupational therapist friend of mine claims that there will be a serious reduction in repetitive strain injuries if we could just get everyone full-body gesturing rather than sitting tapping on keyboards with our heads staring at screens. It made me smile to think about the transformation cube-land offices would undergo if we redesigned them to allow employees to physically engage with digital data though full-body motion. At the same time, it perturbed me that I may have to do a series of yoga sun salutations to find my files or deftly execute a ‘downward facing dog’ pose to send an email. In any case, watching my friends prance around with their Wiis and Kinects gives me pause and makes me think we are still some way away from anything that is not repetitive strain injury inducing; we are, I fear, far from something of which my friend would truly approve.

From a broader social perspective, even the way we gesture is socially prescribed and sanctioned. It’s not just that you need to have a gesture be performed well enough for others to recognize it. How you gesture or gesticulate is socially grounded; we learn what are appropriate and inappropriate ways to gesture. Often assessments of other cultures’ ways of gesturing and gesticulating are prime material for asserting moral superiority. Much work was done in the first half of the 20^th century on gestural and postural characteristics of different cultural groups. This work was inspired in part by Wilhelm Wundt’s premise in Volkerpsychologie that primordial speech was a gesture and that gesticulation was a mirror to the soul. Much earlier than this research, Erasmus’ bestseller De civilitate morum puerilium [8], published in 1530 an admonition that translates as “[Do not] shrug or wrygg thy shoulders as we see in many Italians”. Adam Smith compared the English and the French in terms of the plenitude, form and size of their gesturing. “Foreigners observe that there is no nation in the world that uses so little gesticulation in their conversation as the English. A Frenchman, in telling a story that is of no consequence to him or anyone else sill use a thousand gestures and contortions of his face, whereas a well-bred Englishman will tell you one wherein his life and fortune are concerned without altering a muscle.” [2]

Less loftily, cultural concerns for the specifics of a point were exemplified recently when I went to Disneyland. Disney docents point with two fingers, not just an outstretched index finger but both the index finger and middle finger. When asked why, I was informed that in some cultures pointing with a single index finger is considered rude. Curious, I investigated. Sure enough, a (draft) Wikipedia page on etiquette in North America states clearly “Pointing is to be avoided, unless specifically pointing to an object and not a person”. A quick bit of café-based observation suggests people are unaware of this particular gem of everyday etiquette. Possibly apocryphally, I was also told by a friend the other night when opining on this topic that people that in some Native American cultures it is considered appropriate to point with the nose. And, apparently some cultures prefer lip pointing.

So bother with this pondering on pointing? I am wondering what research lies ahead as this gestural interface revolution takes hold. What are we as designers and developers going to observe and going to create? What are we going to do to get systems learning with us as we point, gesture, gesticulate and communicate? As humans, we know that getting to know someone often involves a subtle mirroring of posture, the development of an inter-personal choreography of motion–I learn how you move and learn to move as you move, in concert with you, creating a subtle feedback loop of motion that signifies connection and intimacy. Will this happen with our technologies? And how will they manage with multiple masters and mistresses of micro-motion, of physical-emotional choreography? More prosaically, as someone out and about in the world, as digital interactions with walls and floors become commonplace, am I going to be bashed by people pointing? Am I going to abashed about their way of pointing? Julie Rice and Stephen Brewster of Glasgow University in Scotland have been doing field and survey work on just this, addressing how social setting affects the acceptability of interactional gestures. Just what would people prefer not to do in public when interacting with the digital devices, and how much difference does it make if they do or don’t know others who are present? Head nodding and nose tapping apparently are more likely to be unacceptable than wrist rotation and foot tapping [3]. And what happens when augmented reality becomes a reality and meets gestural interaction? I may not even be able to see what you are thumbing your nose at–remembering that to thumb one’s nose at someone is the highest order of rudeness and indeed the cause of many deadly fights in Shakespearean plays–and I may assume for the lack of shared referent that it is in fact me not the unseen, digital interlocuter at whom the gesture is directed. And finally, will our digital devices also develop subtle sensibilities about how a gesture is performed beyond simply system calibration? Will they ignore us if we are being culturally rude? Or will they accommodate us, just as the poor taxi driver in China did, forgiving us for being linguistically ignorant, and possibly posturally and gesturally ignorant too? I confess; I don’t know if pointing with one index finger is rude in China or not. I didn’t have the spoken or body language to find out.

NOTE: If you are as bewildered by the array of work on gesture-based interaction that has been published, it is useful to have a framework. Happily, one exists. In her PhD thesis Maria Karam [4] elaborated a taxonomy of gestures in the human computer interaction literature, summarized in a working paper written with m.c. schreafel [5]. Drawing on work by Francis Quek from 2002 and earlier work by Alan Wexelblat in the late 1990’s this taxonomy breaks research into different categories of gesture style: gesticulation, manipulations, semaphores, deictic and language gestures.

---

[1] John Underkoffler was the designer of Minority Report’s interface. The g-speak tracks hand movements and allows users to manipulate 3D objects in space. See also SixthSense, developed by Pranav Mistry at the MIT Media Lab.

[2] For more on this see A Cultural History of Gesture, Jan Bremmer and Herman Roodenburg, Polity Press, 1991

[3] Rico, J. and Brewster, S.A. Usable Gestures for Mobile Interfaces: Evaluating Social Acceptability. In Proceedings of ACM CHI 2010 (Atlanta, GA, USA), ACM Press.

[4] Karam, M. (2006) PhD Thesis: A framework for research and design of gesture-based human-computer interactions. PhD thesis, University of Southampton.

[5] Karam, M. and schraefel, m. c. (2005) A Taxonomy of Gestures in Human Computer Interactions. Technical Report ECSTR-IAM05-009, Electronics and Computer Science, University of Southampton.

Resources on Experience Design, a selection

Here are some resources/references/publications on Experience Design that were used to develop a Tutorial at CSCW 2011 in China that Elizabeth Goodman, Marco de Sa and I prepared and delivered. Thanks to all Facebook friends who offered suggestions.

CAVEAT: This is NOT an exhaustive list obviously; there are some excellent resources out there. These just happen to be some we used when creating the tutorial. We also drew on some of the content that was taught at a workshop at UX Week 2010.

TEXTS

Industry perspectives on UX design

• Peter Merholz, Todd Wilkens, Brandon Schauer, and David Verba (2008), Subject To Change: Creating Great Products & Services for an Uncertain World: Adaptive Path on Design. O’Reilly Media.
• Bill Moggridge (2006) Designing Interactions. MIT Press
• Mike Kuniavsky (2010) Smart Things: Ubiquitous Computing User Experience Design, Morgan Kaufman

Skills and techniques

• Kim Goodwin (2009) Designing for the Digital Age: How to Create Human-Centered Products and Services, Wiley Publishing

Sketching and prototyping

• Bill Buxton (2007) Sketching User Experiences, Elsevier (hovers in between academic and industry perspectives)
• Dan Roam (2009) The Back of the Napkin: Solving Problems and Selling Ideas with Pictures, Penguin Group USA

Business

• Roger Martin (2009) The Design of Business: Why Design Thinking is the Next Competitive Advantage, Harvard Business School Publishing.
• Tim Brown with Barry Katz (2009) Change by Design. How Design Thinking Transforms Organizations and Inspires Innovation, Harper Collins.
• Alan Cooper (1999) The Inmates are Running the Asylum, Macmillan Publishing Co., Inc.

Psychology and philosophy of experience

• Don Norman (1998) The Design of Everyday Things. MIT Press
• M Csikszentmihalyi (2003) Good Business: Leadership. Flow and the Making of Meaning (Viking, New York)
• Csikszentmihalyi, Mihaly (1998). Finding Flow: The Psychology of Engagement With Everyday Life. Basic Books.
• Byron Reeves and Clifford Nass (1996) The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places, University of Chicago Press.
• Peter Wright & John McCarthy (2004) Technology As Experience. MIT press
• Tor Nørretranders (1998). The User Illusion: Cutting Consciousness Down to Size. Viking.
• Merleau-Ponty, Maurice. Trans: Colin Smith. Phenomenology of Perception (London: Routledge, 2005)
• Wolfgang Iser (1980) The Act of Reading: A Theory of Aesthetic Response, John Hopkins University Press

Visual design

• Stop Stealing Sheep (type usage)
• UI Wizards.com

Storyboarding

• Scott McCloud (2006) Making Comics, Harper
• John Hart (2008) The Art of the Storyboard, Second Edition: A filmmaker’s introduction. Elsevier.
• Wendy Tumminello (2004) Exploring Storyboarding (Design Exploration). Thomson Learning.

Video

• Steven Douglas Katz (1991) Film directing shot by shot: visualizing from concept to screen, Focal Press (on movie making; good for video and transitional literacy)
• Scott Kelby (2006) The Digital Photography Book, Peachpit Press

Some other books/resources we have drawn on in the past

• Richard Saul Wurman (2000) Information Anxiety 2, Pearson Education
• Works by Edward Tufte
• Hiroshi Ishii’s work.
• Neil Gershenfeld (1999) When Things Start to Think, Holt, Henry & Company, Inc.
• Brain Rules by John Medina (see also http://brainrules.net/)
  What the Bleep Do We Know? (and ancillary pieces, such as the movie and the Web site)
• Richard Sexton (1987) American Style Classic Product Design from Airstream to Zippo

Online resources
http://www.nathan.com/ed/index.html

We also watched a number of short presentations and interviews online with Experience Design researchers, practitioners and luminaries.

Making Time

[This is a draft of a column I wrote for the ACM’s interactions magazine.  It will appear mid 2011].
______________________________________________________

One morning, as Gregor Samsa was waking up from anxious dreams, he discovered that in his bed he had been transformed into a monstrous verminous bug.

Thus begins one of my favourite novels, The Metamorphosis by Franz Kafka. What is most remarkable about Gregor’s awakening, and his discovery that he has metamorphised into a dung beetle is that, in the minutes that follow, his greatest concern is that he has missed his train.

Like Gregor, time and schedules have been much on my mind of late. Why? Well, firstly, I overslept the other day. My phone is my alarm clock. Sadly, my phone had died quietly during the night. Ergo–no alarm to awaken me. Although I did not wake up a dung beetle, I was nevertheless disoriented. Secondly, about a week ago, I missed a meeting. Well, strictly speaking, I didn’t miss it because I didn’t know I was supposed to be at it. All I can surmise is that there had been a breakdown in the complicated network of services, applications, devices and people that constitutes the sociotechnical practice of time management called “calendaring”. The meeting was clearly listed on my colleague’s calendar, but not on mine.

So, given my recent horological mishaps, I have been ruminating on the concept of time and it’s management through calendars and alerts.

Calendars reckon past and/or future time. The primary purpose of the calendar is the orientation of our bodies and minds–and those of others–in time and space. In contrast to the fluidity of experienced time, calendars create boundaries between activities. They prescribe the amount of time we should spend on something:  30 minutes with Jane talking about her project, an hour for the meeting on budget, 1 hour giving a lecture on HTML-5, thirty minutes on a mandated management course…..and of course, finally, a day of rest.

To be effective social coordinators, calendars require that we share an idea of how time is structured, how it breaks down quantitatively. My minute and yours should both be 60 seconds; thus we can pass time at the same rate quantitatively–even if, qualitatively, for me the hours have rushed by and for you they have felt like swimming in treacle. And, we should share an idea of exactly when 8pm is if we are going to meet for dinner at 8pm.

Calendars don’t just keep individuals synchronised. Calendars, so scholars like the sociologist Emile Durkheim tell us, are central to societal order. Calendars are the sentinels of ‘appropriate’ behavior. Minutes and days and hours often have activities associated with them–indications of when we should work, rest, pray and/or play. Different social values are placed on different hours of the day and on days of the week; in many calendars Saturdays and Sundays are by default given less space, reflecting social norms that separate workdays from (non-work) weekend days. Routine, calendared time is central to creating a social sense of belonging. In his 2006 article, Tim Edensor argues that structured time in the form of everyday rhythms–which he breaks down into institutionalized schedules, habitual routines, collective synchronicities and serialized time-spaces–are how a sense of national identity and belonging is sustained. One can see this play out in my neighbourhood, wherein many different immigrant cultures reside. What is considered an appropriate time for dinner differs by several hours: between 6pm and 7pm for some, between 9pm and 10pm for others.

I suspect most of us take for granted the idea that we have a shared concept of time. However, the carving up of time into seconds, minutes, hours, days, months and years is a convention, and the familiar structure of the predominant western calendar–the Gregorian calendar, which was only in introduced in 1582–differs from classical calendars like the Mayan, Aztec and Inca, and the more recent Julian calendar[1]. Notably, Russia and Greece only converted to the Gregorian calendar from the Julian calendar in the 20^th century. Further, it has not always been the case that someone in Bangalore could so easily work out what exactly time it is for me in San Francisco. It was only in the 1880’s that a uniform time was imposed in Britain; until then, time in Britain varied according to location. This local time stood in contrast to ‘ London time ’ (i.e. Greenwich Mean Time (GMT)); Oxford was five minutes behind London, while Plymouth was twenty minutes behind London[2]. In his book The Culture of Time and Space 1880-1918 Stephen Kern writes of the railroads in the US, “Around 1870 if a traveler from Washington to San Francisco set his watch in every town he passed through, he would set it over 200 times”. The railroads instituted uniform time on November 18, 1883. In 1884 Greenwich was established to be the zero meridian and the 24 time zones one hour apart were established. Countries signed up to this structuring of time one by one: Japan in 1888, Belgium and Holland in 1892, Germany, Austro-Hungary and Italy in 1893. At the International Conference on Time in 1912 the telegraph was proposed to be the method of maintaining accurate time signals and transmitting them around the world; astronomical readings were to be taken and sent to the Eiffel Tower that would relay them to eight stations spaced over the globe. This process was inaugurated on July 1st 1913 at 10am. Global time was born, and the death knell rang for the quaint custom of local time. In an odd way, we can thus trace our globally shared, personal and corporate calendars back to the railroads for instigating the rationalization of time across the globe. It’s quite fitting, therefore, that missing the train is foremost in Gregor’s mind when he wakes up.

However, while synchronised global time connects us, it is all too easy sometimes to forget that there are in fact a number of calendars in operation in parallel today–Chinese, Hebrew and Islamic are just three examples.

As I turn back to my missed meeting, I note that calendars have ecclesiastical origins; the Book of Hours structured time into routines for work and worship for monks in the Benedictine order. However, in sharp contrast to the quiet, stable regularity of the liturgical life, my calendar is a chaotic beast in constant need of maintenance and management. Meetings pop on and off like jumping beans as the hoping-to-be-assembled try to find a time that works for all concerned. Vigilence is required lest one is triply booked, and priorities are always being calculated: Is this meeting more important than that one but if so and so is there then that is a good opportunity to get things moving forward…… Oh no, now they are not going to be there after all and yet I am committed to going, how do I shift this around…… and on and on.

The root of the problem lies in the multiples–multiple calendars and multiple people on one calendar. For the first point, I have too many calendars and the effective synchronization of my calendars is not a solved problem. Ghost (long departed/deleted) meetings haunt the calendar on my computer, while my mobile phone presents a suspiciously clean blank slate. Sometimes there is little correspondence between the two, despite their notionally being jacked in to the same server. For the second point, shared calendars (such a good idea in principle) are a gargantuan, social rogue elephant. Herein lie clashes in culture, herein lie power relationships and herein lie a network of complex dependencies. Routine issues arise for me in the following forms: blank space on the calendar, the curse of durational rigidity, the clash between sociotemporal and biotemporal time, and the problem of travel time. Lets briefly review each of these…..

‘Idle’ time People routinely look at my calendar to determine when I am free to meet; they plop meetings on my calendar based on what they see as ‘free’ time. This is based on a fallacious assumption–that if there is nothing recorded there, then I am free. This is a misreading of my practice of calendar use. Booked times on my calendar are not simply islands of colour in a collaborative paint-by-numbers schematic where the blanks are inviting others to fill them in–I saw a gap so I filled it.

Of course, idle time is anathema to the shared calendar in a culture where to be not actively doing could possibly be interpreted as shirking. In my view, day of back-to-back meetings means there is too little time for creative thought or for reflection. Research indicates that time when one is doing the least, as for example when meditating, is when the most creative moments can occur[3]. The jammed calendar, continual context-switching and mad dashes from one location to another are emotionally draining, mania inducing and counter to creativity.

So I sometimes put “meetings” onto my calendar to simply block some thinking time. I feel sheepish about this. I am reminded of a friend of mine, who, when we were teenagers, used to write things like “peas and carrots for tea” in her journal. Recording peas and carrots was not because of some dietary obsession, they stood in as code for ‘held hands’ and ‘kissed’, reporting on her teenage encounters with her boyfriend; the code was invented lest her mother should read her journal and be mortified by her teenage explorations. So, it is that I transform thinking, writing and reading into ‘Strategy’ and ‘Planning’, appropriate behaviours for a corporate context. Durkheim and followers are correct: how one manages one’s time is an issue of morality and social accountability, not just temporal coordination. It’s a tricky business.

Durational rigidity For the operationally minded, a meeting that is scheduled for an hour must last an hour even when nothing is being achieved. On the other side of that, sometimes one can be just warming up, just getting to the crux of a problem and the hour is up, the meeting has to end truncating the creative process.

Travel time Another problem, and one where a simple technical solution would help out, is travel time between locations. When one works in several different office buildings that are miles apart, it takes time to get from one to the other. It would be useful if I could hook my calendar up to these locations, and have travel time calculated and reflected automatically. So if a meeting is dropped onto my calendar, travel time is automatically blocked in–in fact, I could imagine a lot of background calculating that can be done by hooking my calendar up to location and to my social services and applications[4].

Biotemporal time Working across time zones can be really hard. The cheerful calendar flattens time, it sees all times as equal. Calendars are simply tabulated time in a grid, they do not reflect lived time. Odd times for calls can sneak in there, creating social and personal dilemmas–I want to be a good citizen but I know I am going to be less than my best at that time. Sociotemporal times (as in when it is appropriate to be working and when not) clashes here with biotemporal time. Being on a meeting conference call when your body and your entire environment tells you that you should be sleeping is simply hard. Time may be global but my body is not.

None of my observations are earth-shatteringly novel. There has been a wealth of research in the HCI community from the early 1980’s and continuing now today, on life scheduling and calendaring–in collocated and in distributed workgoups, in the home, in leisure groups, within families, between families, on paper, across paper and other devices, on personal computers, using mobiles, using location services and with visual front end experiences including3D representations. Just to name a few of the research directions. There are typologies of calender user type such as that offered by Carmen Neustaedter and colleagues who call out three different types of families—assigning them to the categories monocentric, pericentric, and polycentric according to the level of family involvement in the calendaring process. Monocentric families are those where the routine is centered on a primary scheduler, pericentric families have the calendar routine centered on the primary scheduler with infrequent involvement by secondary schedulers and polycentric families are those where the calendar routine is still centered on the primary schedulers, yet secondary schedulers are now frequently involved. BUT despite all this work, there’s still plenty we can do in the world of sociotechnical design to rethink the calendar. My calendar does not feel “_centric” in any way; it feels chaotic.

“We shape our dwellings and afterward our dwellings shape us” said Winston Churchill in 1943. We could apply this observation to time; we shaped the calendar and now the calendar shapes us, it dictates how we (should) live. True to Louis Sullivan’s adage form follows function, the digital calendar wears its assumptions and its intellectual heritage on its sleeve: computer science, psychology, information architecture and the ethical structure of the approved-of day. Perhaps we need a new tack.

In Branko Lukic’s and Barry Katz’s 2011 text, Nonobject, they explore product designs that sit at the interstices of philosophy and technology. They step back from simplistic notions of form and function to shake up how we think about products, to question what is ‘normal’ or taken for granted, and to question the values that are embedded within the typical form of everyday artifacts. In a section entitled Overclocked, they explore clocks and watches, our time-keepers. Katz writes, “as our measuring devices grow ever more accurate, we find ourselves perpetually “overclocked” to use a term familiar to every computer hacker who has ratcheted up a component to run at a higher clock speed than it was intended for in order to coax higher performance out of a system. We do the same to ourselves.” A number of designs are presented: the Tick-Tock Inner Clock that taps against the skin to let someone feel the passage of time and the Clock Book where time is laid out on pages we can turn–when we want to–push. Lukic’s watches and clocks invite us to rethink we conceptualize, represent and manage time. Somewhat less extreme but nevertheless taking a playful take on clock design, Alice Wang’s 2009 suggestion for the Tyrant alarm clock is brilliant. This alarm clock calls people from your address book on your mobile phone every three minutes if you don’t get up and turn it off; with this, Wang is betting that the anxiety of broadcasting your slothful habits to anyone in your address book will propel you to get up. Wang gleefully reports that it is the social guilt that will get people moving out of bed. Social anxiety has long been a driver for action; this is I think a nice example of it, and this is a step beyond thinking instrumentally about the clock’s utility/function in isolation from the rest of one’s life.

Let’s do the same thing with calendars. Let’s take a step back. Let’s follow Lukic and take our lead from Architectura Da Carta, the Italian tradition of articulating and illustrating the unlikely, the unbuilt and the unbuildable. Let’s use art, philosophy and technological creativity to envision a better aesthetic experience, to blast the calendar apart and rebuild it; let’s be better about enabling the plularity of private and public times that humans live in parallel; let’s automate the calculation of time in motion between location(s); let’s build in time for creativity and reflection as social and moral imperative; let’s make a calendar that adapts the schedule when it realizes you have woken up having metamorphised into a sentient dung beetle.

---

[1] See Anthony Aveni Empires of Time. Calendars, Clocks and Cultures, New York: Basic Books, 1953

[2] See Journal of Design History Vol. 22 No. 2 Designing Time: The Design and Use of Nineteenth-Century Transport Timetables by Mike Esbester

[3] See for example The neuropsychological connection between creativity and meditation published in ‘Creativity Research Journal’, 2009 by Roy Horan

[4] See Lovett and colleagues on this in their Ubicomp 2010 paper: The Calendar as a Sensor: Analysis and Improvement Using Data Fusion with Social Networks and Location

Rat, Rational or Seething Cauldron of Desire: Designing the Shopper

[This is a draft of a column I wrote for the ACM’s interactions magazine. Publication details: interactions, Volume 18 Issue 1, January + February 2011, ACM New York, NY, USA]
______________________________________________________

Since the 1930s, governments have largely followed the view of John Maynard Keynes that spending, not saving, drives successful economies. We have taken this charge seriously in the U.S. If Adam Smith characterized Britain as a nation of shopkeepers, the U.S. is indubitably a nation of shoppers. No other country in the world spends as much on consumer goods as America.

Whether you love it or hate it, everyone has an opinion on shopping. It’s an expression of identity. It’s the gift I give my family. It’s a chore I have to endure. It’s sport. It’s a sign of the demise of our culture, an assault on our base ethics and morality; “Stop the Shopocalypse!” entreats the performance artist Reverend Billy and his troupe, the Church of Life after Shopping.

As the memory of Santa’s season of goodwill, joy, and multifarious forms of consumption fades and I review my credit card bill, I have been reflecting on how we shop—more specifically, on how the shopping experience is designed for us, and we are in some sense designed by it. For this reverie, I am more interested in what has been dubbed “wants based” rather than “needs based” shopping, which, as evidenced by my credit card statement, clearly makes up most of my purchases.

So, yesterday I made a diary of a shopping day, noting the things and experiences that I usually take for granted, but with a view to wondering how they came to be so. With the caveat that this is not an exhaustive diary, here’s the timeline:

6:30 a.m.: Wake up.
6:45 a.m.: Pick up mail, physical. Two catalogs, an envelope of discount coupons, and an advertisement for expensive holidays masquerading as a travel magazine.
7:00 a.m.: Scan the list of emails from services, vendors, shopping-discount sites, and so on. Canon seems to think that I could make use of yet another camera. Apple thinks I will be enchanted with some new products from them.
7:30 a.m.: Shower, using a multitude of free samples sent to me by online-shopping vendors, most of which products I feel disinclined to purchase.
8:30 a.m.: Drive to work, listen to national public radio (NPR), and hear a review of a book I will likely purchase.
11:25 a.m.: Alleviate mild distraction and boredom by reviewing what is currently available on shopping sites RueLaLa and ideeli, sites that sell designer clothes and accessories. Find names of three designers who are new to me but I will look out for in future. (Thank goodness for online shopping, which means the indignity of trying on clothes in public fitting rooms is circumvented and provides idle-time shopping in 15-minute chunks when I am bored.)
5:45 p.m.: Drive to shopping mall and locate specific department store to buy a gift for a friend, navigate lights, music, perfume, and other sensory assaults.
7:00 p.m.: Drive home from work via supermarket, pick up groceries.
7:50 p.m.: Drag loads of heavy grocery bags upstairs; curse the shopping cart that masked the weight of the load I was going to have to carry later.
9:45 p.m.: Check out the Canon offers and surf the Web, reading reviews of various computer products; cave in and buy coat from RueLaLa; download infeasibly peppy music from iTunes based purely on beats per minute, and select a book for a friend from Amazon. Watch bizarre and dubious “shopping haul” videos—teenage girls sharing shopping expertise on Youtube. Am transfixed by these videos with ghoulish fascination, and note from view counts that tens of thousands of people have watched them.

Following yesterday’s diary, today I surfed the Internet, researching how some of these experiences came to be. As you will see, aside from the Internet as a channel for learning, researching, and purchasing, many of the things I encountered yesterday and have, up until now, taken for granted, carefully were designed or created with a view of what the practice of shopping does, could and indeed should look like.

The department store as we know it derived from the entrepreneurial efforts of several key figures: the Bloomingdale brothers, John and Lyman, bankrolled by profits made from sales of the wildly successful hoop skirt; Brigham Young, who in 1868 founded Zion’s Cooperative Mercantile Institution (ZMCI) in Salt Lake City, Utah; Marshall Field in Chicago; Alexander T. Steward in New York; and last but certainly not least, John Wanamaker, who opened the Grand Depot, a round, six-story department store in Philadelphia in 1877. The key business driver for the department store was simple: Buying wholesale meant retail costs could be kept relatively low. The key experience driver was educating people as to hitherto unknown possibilities for parting with their cash. The department store’s development from convenient goods provision to center of cultural production is wonderfully described in Jan Whitaker’s book Service and Style: How the American Department Store Fashioned the Middle Class.

On a larger scale, the first shopping mall, the Country Club Plaza, was founded by the J.C. Nichols Company and opened near Kansas City, Missouri, in 1922. While the Country Club Plaza was open to the sky, the first enclosed shopping mall was opened in Edina, Minnesota (near Minneapolis), in 1956. The first shopping mall, a collection of stores, was designed to make shopping more convenient by placing lots of shops in one location. The West Edmonton Mall opened in Alberta in 1981, housing 800 stores, a hotel, an amusement park, a miniature-golf course, a church, a “water park” for sunbathing and surfing, a zoo, and a 438-foot-long lake. Park the family while you shop. Or better still, shop-play-shop.

While the shopping mall as we know it is so familiar, much of what we take for granted now was invented by Victor Gruen, an architect from Vienna. Gruen and associates designed the Northland Shopping Center in Detroit, in 1954, the Southdale Shopping Center near Minneapolis in 1956, and the Midtown Plaza in Rochester, N.Y., which opened in 1962. The innovations they designed revolutionised shopping. Innovations developed over the years: 1939, Gruen was invited to design a leather-goods store for a friend in New York. Gruen’s inspiration was to create an entrance to the store that was intriguing and enticing to the eye. Colours, lighting and reflections were used in combination with fake marble, and green corrugated glass on the ceiling to create a “customer trap” designed to lure customers into the store regardless of their initial purchase intent or their interest in the products on display (so much for the shopping list). Gruen went on to create the multi-level shopping facility with escalators between the levels and multiple entry ways from multi-level carparks. Gruen was also the genius who initiated blank wall exterior for stores; prior to that shows focused on window displays.

The shopping cart was invented by Sylvan Goldman, owner of a chain of Oklahoma City grocery stores called Standard/Piggly-Wiggly, in 1936. The original shopping cart was like all prototypes, a humble contraption. It involved two wire baskets, a folding chair and some wheels. Anyone who has wheeled themselves to the checkout, waited the allotted time, and then experienced a wave of shock at the bill followed by a panicked glance to assess what could possibly have ended up in the cart, knows how much easier it is to purchase more than you intended with a cart. The best “forcing function” to reduced purchasing is carrying the stuff you intend to buy; as your arms get wrenched from their sockets, you are more sanguine about that extra, just-in-case bottle of lotion or packet of batteries. And thank goodness for Dee Horton and Lew Hewitt’s 1954 invention, the sliding automatic door–turning handles and pushing doors while holding laden shopping bags is never a graceful exercise.

Store layout. It should be noted that stores themselves are scripted spaces, laid out to maximize our likelihood of purchasing certain items. The most famous researcher on the topic of store layout and its impact on purchasing behavior is Paco Underhill. An environmental psychologist and author of Why We Buy: The Science of Shopping and Call of the Mall: The Geography of Shopping, Underhill sets forth fairly straightforward ideas—how you structure the environment leads to different purchasing behavior, and our in-store experience is as much a part of why we purchase something as the carefully constructed shopping list.

Coupons. Introduced in 1895 as the brainchild of Asa Candler, coupons were placed in the newspaper for a free Coke from any fountain as a way of promoting the soft drink. I lost count of how many physical and digital coupons I encountered yesterday. But they are clearly part of my life, even though I use perhaps one in 100.

The mail-order catalog was created by Aaron Montgomery Ward in 1872. As I look at the two I received in my paper recycling pile and review the vendor sites I visited when surfing the Internet, I am comforted to know that many generations before me also preferred shopping from the couch to running around a mall.

Price tags, sales, in-store restaurants, money-back guarantees, and newspaper ads. All of these were created by John Wanamaker, who I think has to get the most points as creator of the contemporary shopping experience. It never occurred to me that someone actually had to invent the price tag!

As I researched these different inventions and designed experiences, I realized that I was also participating in a designed abstraction, playing my part with gusto. I have been trundling down the “purchase funnel,” a concept developed in 1898 by St. Elmo Lewis, who mapped a theoretical customer journey from the moment a brand or product attracts your attention to the moment you actually buy it. To advertisers, marketers, and merchandisers, AIDA is not an opera by Giuseppe Verdi; rather, it reflects points along the purchase funnel: attention, interest, desire, and action. Oddly, Gruen’s “customer trap”, physically luring people into stores, almost seems to have been captured in the funnel metaphor. Draw people in, and help them move from unexpected awareness to desire to intent and actual purchase.

I learned about a host of new products from the catalogs, the online surfing, the coupons, the radio, and the products on display. I find myself tenderized by the marketers and advertisers and am somewhere along the “purchase funnel” for a number of upcoming purchases. I also realize that designers of all sorts (advertisers, print layout, store layout, online stores, architects, entrepreneurs, engineers) have other abstractions that are being played out in their designs: an abstraction of me as a human, as a shopper and what motivates me to shop in the first place, what gets me peering into the funnel.

In researching this column, I also discovered that, as a shopper, that I am variously cast as a rat, as a rational person, as a seething cauldron of desire, or as a primeval Homo sapien, hell-bent on foraging for my kin. The model of me as rat derives from behaviorist psychology, emphasizing that I have been “conditioned” through reward (nice treats, Jimmy Choo shoes) and punishment (nasty things like hunger, cold, and no Jimmy Choo shoes) to respond to certain cues that drive us to shop. The second character, the rational shopper in me, is Homo economicus—rationally making choices on the basis of utility with some cost. This model is based on Lionel Robbins’s rational choice theory. While I recognize some of the behaviors, in general Homo economicus does not hang out with me. My third potential shopping mantle is a seething, bubbling cauldron of desire based on psychoanalytic theory. My ego, my superego and my id–the three-ring circus of conflicting urges that underlie whatever identity issues I may have–all held in molten, psychic, tension, stabilized momentarily by the acquisition of stuff. That is, I buy things so I can express my true inner self, whoever that might be today. Finally, the hunter-gatherer model emphasizes that we shop because it is innately human to shop—it is wired into our primal brain to rush out and hunt for things. Of all the models I read today, I confess the last leaves me most cold; it’s about as interesting as saying that shopping involves human beings.

Reflecting on my own experience of shopping, however, these theories have (at least) two problems. First, they are overly focussed a lot on individual shopping, when shopping is clearly social. Artist Barbara Krueger coined the phrase “I shop therefore I am”; I say, “We shop therefore we are.” Shopping is something we learn to do; we learn culturally how to shop and what to shop for. Pierre Bourdieu, the French philosopher, talks about “taste communities”; you are not just what you yourself consume but also what your friends consume. You Blahnik, therefore I Blahnik. The existence of the shopping-haul videos would not have shocked Bourdieu. Those young women are teaching others what to shop for, why the products matter, where to acquire them and what to do with them once they are acquired. Second, what constitutes a rational decision depends on how you scope the reasoning process. What looks entirely reasonable in the moment (tired, hungry, desperate for something for Uncle John so I don’t look like a cheapskate at Christmas), in retrospect looks daft (when he opens his package to find his 23rd pair of socks). The bounds of rational reasoning shift. A product or service I would not have looked at yesterday somehow becomes absolutely essential today. Grant McCracken, the anthropologist and commentator on contemporary culture, talks about the “Diderot effect,” invoking the philosopher Diderot, whose apartment remodel was kicked off by the receipt of a luxurious dressing gown, in the shadow of which the rest of his apartment looked shabby.

If we consider the trajectory of department store to mall to Internet, our choices of what to buy and how to “upgrade” our lives are increasing at a ridiculous pace. I wonder, what’s going to be the next revolution in the shopping experience? I also note it took me significantly less time and energy to part with my cash during the last Holiday season than it would have 20 years ago. This reflects the fact that, in tandem with the choice explosion, the time and energy taken to potentially acquire things is decreasing markedly; easy-to-use, one-click and delivery-to-door are in cahoots, encouraging the impulse buy, enticing the rat and irrational id, evolving the sloth model that has not yet been articulated fully, and muscling Homo economicus out the door. So, next time you’re shopping, online or off, think about how that experience is designed and in what ways it appeals to you the shopper–as rat, rational, id, hunter … or sloth?

Enticing engagement

[This is a draft of a column I wrote for the ACM’s interactions magazine. It appeared in Volume 17 Issue 3, May + June 2010].
______________________________________________________

Human engagement
A: Do you love me?
B: Yes
A: Will you marry me?
B: Yes.

Internet engagement
A: Do you love us?
B: click click click

A: Do you love us now?
B: click click click

Internet disengagement
A: Do you love us?
B: click click click
A: Do you love us now?
B:

A: What about now?
B:
A: Hello? Where did you go?

For me, long gone are the days when the word “engagement” conjured up diamonds, parties, and champagne. These days engagement is all about divining how much love your users or the audience have for product(s) and/or application(s). In the Internet world, the word “engagement” is intimately associated with measurement, metrics, and monetization; it is all about clicks and conversion, visitors, page views and duration. This is the world of media marketing, and of Internet success and failure.

What is Engagement?
Before I dig into Internet engagement measurement, I want to step back and think about what engagement means to me. When first asked about engagement by a colleague at work, I spouted all I knew about the experience of engagement as I understood it from a psychologist’s worldview. Flow. Immersion. Fascination. The swift passing of time as attention is held—tempus fugit. Engagement is the absolute opposite of restless boredom; boredom is the experience of time expanding painfully, a feeling of being itchy and twitchy and ready for any external stimulus beyond what you are doing now.

From a psychologist’s point of view, engagement sits within a framework of arousal states—boredom at one part of the spectrum, engagement at the other. Altertness, forced focus and stress all sit somewhere in between. Perhaps the most famous approach to thinking about engagement is the concept of flow, proposed by Mihály Csíkszentmihályi. Flow is the state of full immersion in what one is doing: a feeling of energized focus, total involvement, and success in the process of the activity. Flow is a positive feeling, reflecting an utterly focused concentration and alignment with what one is doing. In some conceptions, flow is associated with feelings of joy and rapture, an exclusion of all but the central activity. Engagement may not be this rapturous positive feeling; people can be negatively engaged if something is standing between them and their desired state or goal, and for many people, military engagement denotes a negative form of entanglement.

But central to the idea of engagement is the triggering and capturing of attention, of being beguiled and perhaps focused to the exclusion of everything else. Further, lasting engagement can manifest in episodes of focus; perhaps someone will come back for more, again and again and again. This is where engagement in my early training intersects with the world of marketing and media engagement, the focus to which I now turn.

Creating Engagement
“Engagement” is a favorite word for media marketers and for the designers of products, services, applications. Take a look: The Web is brimming with smart people opining on the topic. In this world, marketers, data miners, statisticians, designers, and a host of other professionals come together. “Engagement is the most essential measure of the success of any medium. Engagement speaks to passion, preference, and habit,” said Charles Buchwalter, vice president of analytics at Nielsen NetRatings, in a 2005 white paper, “Integrated Interactive Marketing: Quantifying the Evolution of Online Engagement.”

In this world, the point is to stimulate engagement, to create it and make it last for whatever period is appropriate for your offering. In most cases, the goal of creating or “driving” engagement is to provide a platform for advertising products or services.

Models of how to engage consumers make assumptions about the level of active involvement of which the consumer is likely to be capable or willing to offer. At one end consumers are seen as being passively engaged with the content; think of the stereotypical couch potato glued to the screen while shows and advertising flicker across the TV screen. At the other end of the spectrum consumers are seen as actively seeking out content, developing their own understandings of that content, and then perhaps sharing with others. The active consumer concept is at the center of viral or word-of-mouth advertising and product engagement. Another example on the active consumer end of the spectrum is engagement marketing (which is also variously called experiential marketing, event marketing, live marketing, or participation marketing).

I like the idea that persuasion architects, who are responsible for creating campaigns to get people actively involved in advertising and engagement, are not completely in control of how the recipient perceives the message or whether or not they are likely to be engaged. I like to bestow upon the reader some agency in their own engagement. Borrowing from Tony Schwarz, a ’60s political advertising guru, who coined the idea of successful campaigns as those that invite people to “participate in their own manipulation,” I call this new world of Internet and cross-channel media play a form of “partipulation.” Perhaps we need “vongagement” to stand in for voluntary engagement? We need a word implies, entails and insists upon reciprocation, a word that acknowledges that the engagement is essentially about a relationship between a person and an activity or an entity.

Measuring Engagement
Measurement and evaluation are key in good design research; the same is true of good audience studies—it is a good thing to be able to evaluate how successful you are at engaging your audience.

In the media world, there have been many scales and measures of engagement, most of them developed for TV viewing. Examples include Q Scores from Marketing Evaluations that measured “likeability,” launched in 1963, and Jack Myers’s Emotional Connections model of viewer engagement, launched in 1999. Since then scales have measured relevance, affinity, comfort and resonance, impression, opinions, recall, awareness of brands and products, memory for program details, emotional involvement, and, of course, moment-by-moment measurements of physiological and neurological effects. Most of these studies are with representative panels of people sampled from the population of viewers as a whole.

In the world of the Internet, measures of engagement focus on features, applications, sites, services, and brands. Here, the Holy Grail is generating engagement measures based on the activity of the whole audience, not a sample; in theory, as all actions take place online, the entire system can be instrumented for activity and therefore measured. The metrics and measures even here, however, are constantly in negotiation and transition. Leaders in the discussions surrounding measurement for engagement are data services like comScore and Neilsen, who work with massive data sets as well as with panels of participants to generate measures and metrics.

A central measure of engagement used to be page views—how many times a webpage was visited. In 2007 the “visits” metric—defined as the number of times a unique person accesses content within a Web entity with breaks between access of at least 30 minutes—was introduced as a way of measuring the frequency with which a person views content, thereby illustrating a key component of user engagement. To determine engagement one looks at total visits, average minutes per visit, average visits per visitor, and average visits per usage day. One of the reasons for changing measures is technological. “With advances like AJAX changing the Internet landscape, certain measures of engagement, such as page views, are diminishing in significance for many Web properties,” said Jack Flanagan, executive Vice President of comScore Media Metrix. “The introduction of these new metrics based on ‘visits’ provides an alternative for measuring user engagement that tells us how frequently visitors are actually returning to the site to view more content.”

Things are also changing as sites become more focused on the social Internet. A more extensive list, based on my review of about six sites dedicated to Web analytics between 2007 and 2009, is this one: There are “root” metrics that appear in all discussions of engagement and Internet use. Some of these are relevant for all Internet sites, and some are relevant only for some sites, e.g., sales apply only to sites that are actively selling something or sites that contain advertising that is intended to lead to a purchase. These “root” metrics are:

• number of unique users
• number of sessions a unique user had with the site/page/application
• duration of visit
• frequency of visit (returning to the site directly—through a URL or bookmark—or indirectly)
• percentage of visits that can be determined to be repeat visits for an identifiable single goal
• recentness of visit
• depth of visit (percent of site visited)
• click-through rate
• “bounce measure” (determined by number of visits that were less than a set number of seconds, which implies the person arrived at the site by mistake or arrived, found nothing they wanted, and left)
• subscriptions, and conversion from free to subscriber service
• sales of product(s) through site and/or referrals that resulted in partner sales
• lifetime value – the last being probably the hardest to evaluate

There are also action metrics, and social-engagement metrics based on elements that are key to social interaction design:

• content contribution (e.g., text, uploads – images, video)
• RSS feed subscriptions
• bookmarks, tags, ratings
• viewing of high-value or medium-value content (as valued from the organization’s point of view—note that depth of visit can be combined with this variable)
• inquiries
• providing personal information, such as filling out profiles
• downloads, content resyndication, and customer reviews
• comments (with some modulation depending on their quality, and some analysis of the ratio between posts and comments plus trackbacks)

Recent applications like Tynt’s Tracer tracks people’s copy-and-paste actions to see what you liked enough to copy and paste it, and many services track site recommendations.

Problems with Internet Engagement Measurement
The artful combination of measures with a clear view of what one is trying to achieve is key. Avinash Kaushik, an analytics practitioner and author of Web Analytics 2.0, calls for good design practice around Web analytics on the issue of engagement. His steps are:

1. Define why your website/application exists, asking, “If there were one thing your website would do, what would that one thing be?”
2. Determine the critical metrics (three or fewer) that will identify exactly how you can measure if your website is successful at delivering against its purpose.
3. Decide what engagement means to you in your context.
4. Don’t call that metric “engagement.”

He urges us to call a spade a spade, saying, “Think very carefully about what you are measuring if you do measure engagement. If engagement to you is repeat visitors by visitors then call it Visit Frequency, don’t call it engagement.” It is crucial to determine ahead of time what your goals are and what constitutes success. This is akin to working on a product and being clear as to what the primary goal of the product is. So many great ideas and great products are hampered by a version of “creeping featurism” that could be called “excitable expansionism.” Whenever I hear someone say “Yes, and we could do this as well!”, I cringe and try to drive the conversation back to the core issue we believe we are addressing—the core goal, need, or hedonistic channel we believe the product should exist to serve.

Engagement cannot be a one-size-fits-all calculation. It varies according to industry, organization, business goals, and so on. Along with this tailoring, one needs to be clear that the relative weighting you give any action depends on your goal. Your metrics and measurement research program depend on whether you are assessing a feature, an application, a site, or a business/brand; whether you are assessing the value of the product in itself for a single user or as a broker between social entities. There are (single) user-engagement measures, but these may not be the same as those that are appropriate for social media. A useful single-user metric might be clicks and content (i.e., how many features are clicked on, how much information and content is uploaded), but a useful measure for social applications may be reciprocity between the people using the application (i.e., how many links do you send me in response to what I send you—how conversational is the application, and how much sharing does it encourage?)

A Broader Yet Still Measurable Notion of Engagement
Before closing, I would like to broaden the idea of engagement still further. Taking my cue from media studies, I am inspired by the “consumer expressions” model and the “connectedness scale” model, both models applied to media like TV and film, but nevertheless fully within the broader and braver new world of partipulation. Or was it vongagement/reciprocative assessment?

The consumer expressions model, devised by researchers from MIT in 2002, suggested the development of a metric that combines qualitative information (the nature of the content, the media consumption environment) with quantitative information (time spent, satisfaction). In 1999 the connectedness model proposed by Cristel Russell and Christopher Puto focused on use of media including self-definition, creative contribution, social interaction around the show, ritualization (organizing and maintaining specific social functions of viewing and using), and the purchase of show-related paraphernalia. This was broadened in 2004, when researchers Russell, Norman, and Heckler conducted a validation of the connectedness scale that examined the extended contribution around a series of constructs—attitude (degree of favor or disfavor), involvement (mental state during viewing), and overall viewing (time spent overall in the medium). Their model, albeit particularly focused on TV and media events, offers six overarching dimensions, some of which perhaps could be extended to the world of digital and physical products and especially to media products:

• escape (immersion into the event)
• fashion (the extent to which the viewer is influenced by the characters’ appearance)
• imitation (the inclination to imitate the characters’ behavior or speech patterns)
• modeling (the degree to which individuals relate their lives to the lives of characters
• aspiration (aspiring to actually be on the show or meet the characters)
• paraphernalia (the degrees to which people collect items to bring the show into their real world)

As connectedness increases, so does frequency of show-related social interaction with others, the development of relationships within the community of co-viewers, and the size of the viewers’ social network of co-consumers. People also have a greater memory for the characters and storylines, and surely this kind of memory is key to long-term engagement as well as in-the-moment engagement. So a deeper sense of overall engagement, and a measure that moves beyond immediate activities into activities beyond the screen, mouse, click and brand awareness into behavior and identity. Indeed, consumption, participation, and identification are key. When I think of a social Internet site/service like Flickr, I immediately see how this level of analysis is useful as a complement beyond assessing clicks and content. The impact of a site or service like Flickr beyond the screen and beyond the session is what makes Flickr a place to see and be seen, not just a place to visit now and then. Avid Flickr users identify with Flickr; they don’t just see it as a place to view images.

My main point is that more integration between different measurement forms is needed to get a clear picture of overall engagement. This is systems thinking; it necessitates measurement perspectives from clicks to brand to identity; it requires a mixed-methods approach and the insight to see how a survey can relate to a field study can relate to a programmatic series of principled data collections … and so on.

In sum, there is so much more to engagement than immediate actions captured by a simplistic action-analytics model or reflected feelings on a survey. Engagement is more than immediate instrumental value and, as a topic, engagement is in need of a broader perspective. Apparently, engagement is itself quite engaging.

Socializing at Cross Purposes

[This is a draft of a column I wrote for the ACM’s interactions magazine. It can out in Volume 17 Issue 1, January + February 2010. The full, final, text with glorious imagery can be found here.]

______________________________________________________

Indulge me for a moment. I have a series of jokes I want to tell you:

• How many social scientists does it take to screw in a lightbulb? None. They do not change lightbulbs; they search for the root cause of why the last one went out.
• How many simulationists does it take to replace a lightbulb? There’s no finite number. Each one builds a fully validated model, but the light never actually goes on.
• How many statisticians does it take to screw in a lightbulb? We really don’t know yet. Our entire sample was skewed to the left.

So what’s with the (not particularly funny) jokes? The point is that they play off particular ways of thinking. In doing so, they show us how different the world can appear, depending on the perspective.

This was evident in a recent meeting. Reminiscent of another set of common (and usually also not funny) jokes involving different nationalities walking into a bar, there were six people in a room: an interaction designer, a statistician with an interest in behavioral modeling, a social scientist, a computer scientist, an self-described “back end with a touch of front end” engineer, and a business executive. We were brainstorming about the potential for accessing social Web applications from personal mobile devices.

Two minutes into our conversation, I said, “We should start with some sound social principles.” This was my bland opening gambit, a preface. Or so I thought. Once I paused for a fraction of a second, everyone started talking at once—like whippets after the faux rabbit at a dog race, the conversation was off. Then it stopped, followed by blank looks. To mix my metaphors horribly: The conversation plumed, spiraled, and evaporated like the contrails of the Blue Angels on July 4th.

The problem was the word “social.”

A quick perusal of the dictionary yielded these definitions of social: relating to human society and its members, living together or enjoying life in communities or organized groups, tending to move or live together in groups or colonies of the same kind, and living or liking to live with others, disposed to friendly intercourse. Etymologically, the word derives from the Latin socialis, meaning “united,” “living with others,” and sequi, meaning “follower,” which should make contemporary social Web application designers happy.

The famous 17th-century philosopher John Locke spoke of “social” as meaning “pertaining to society as a natural condition of human life.” And as an adjective, “social” appears as: “social climber” (starting in 1926); “social work” (1890); “social worker” (1904); “social drink(ing)” (1976); “social studies” as an inclusive term for history, geography, economics (1938); and a concept close to our hearts in these hard times, “social security” as a “system of state support for needy citizens” (1908). That is the backdrop to the conversation I thought I was starting. However…

To the interaction designer, “social” invoked “social Web applications” and all that it means for human interaction with voting (thumbs up and down), favoriting (stars), contact lists and buddy lists, followers, avatars and profiles, chat threading, commenting, recommendations, and view counts. It meant a discussion of icons that suggested (or were derivative of) those on successful social media sites and multimedia content upload and sharing. Talk poured forth about social games and questionnaires, pokes and winks and friending. Let me be clear about my position: I love thinking about these issues, and have recently reviewed drafts for two excellent books in this area—Building Social Web Applications by Gavin Bell and Designing Social Interfaces by Christian Crumlish and Erin Malone. But for the purposes of this meeting, tossing out all of these concepts was great, but it was also putting the cart before the horse. We’d get there, but not yet.

To the computer scientist, “social” sparked sweet, seductive imaginings of the social graph. Wikipedia defines a social graph by explaining that “a social network is a social structure made of individuals (or organizations) called ‘nodes,’ which are tied (connected) by one or more specific types of interdependency, such as friendship, kinship, financial exchange, dislike, sexual relationships, or relationships of beliefs, knowledge or prestige.” The entry continues: “Social network analysis views social relationships in terms of network theory about nodes and ties. Nodes are the individual actors within the networks, and ties are the relationships between the actors. The resulting graph-based structures are often very complex.” No kidding. I love the complexity and curiosity of my species—human beings—and these ways of conceiving social relationships often strike me as dismayingly reductive. They’re very useful within their bounds, but they are summary abstractions of the lyrical complexity of everyday social life. We had a very fruitful foray at this meeting into social recommendations and boundaries—the complexity of “friend relations” and “access control privileges”; the connections between objects via hash tables; and connections between people, their stuff, and other people’s stuff. We discussed these things as inadequate approximations for supporting the negotiated and fluid natures of social trust relationships and the subtle boundaries we negotiate with others.

Somewhat related, my colleague with statistical training was excited to introduce aggregate behavioral models from activity data and collective intelligence from explicit data in our discussion about contemporary notions of “harnessing the hive.” We pressed through issues in database design and the potential for data mining, as well as the relevance, recommendation, and algorithms for automatically inferring buzz, interest, and so on. Here, “social” was to be found in the shadows cast by humans clicking, clacking, typing, uploading across the interconnected networks of the Internet, and making connections betwixt and between things that were heretofore not there to be connected, or at least not visibly so. We discussed how we sometimes derive models—hypotheses, really—about behavior from these obscured traces and how we are sometimes fooled into seeing patterns where there in fact are none.

I won’t enumerate all views expressed, but surely you get the point. I also don’t want to pretend I was seeing the whole picture as this conversation was unfolding. I was as seduced by all these viewpoints, just as my colleagues were entranced by possibilities for implementation and creation. But later—and it was some time later—when I pondered how this short conversation had played out, I realized that we had all collectively engaged in a “we could build/implement/create/design/make” discussion. I had intended to have a conversation at a higher level—one that addressed what people really need, or what would be really helpful and valuable to people. Stepping back even further in the ideation process, I would have liked a conversation about outlining which spaces to probe to see what people need.

Instead, we were enacting the classic “ready, fire, aim!” approach to design that has been a parody of innovation for many years now—design it because you can, stitch together what you already know how to do, throw it out and see what sticks. This is perhaps a natural human tendency because really creative thinking is hard. In a 1986 paper entitled “No Silver Bullet,” Fred Brooks differentiates software design and development processes; he calls them “essential” and “accidental.” “Essential” refers to the difficult part of understanding the domain for which we’re building software and the determination of what software to build in that domain—what activities to inspire, improve, or transform. “Accidental” is the programming and process that has to follow to implement the solution that has been devised. Brooks aptly points out that we have become quite good at training people and designing tools for the accidental part of software development, but the ideation part continues to loom. He succinctly states, “The hardest single part of software development [remains] deciding precisely what to build.”

So what I meant to inspire when I dropped my word-bomb was a discussion of the role that such a device or application could play in everyday life. I was talking about how whatever we propose fits into people’s social context, into how they manage their everyday doings. I was talking about people, about relationships and friendships, and about the contexts that we inhabit and co-create as we move through daily life. I was hoping to focus on the social settings that would afford, support, and allow a technology to be used. I was talking about delivering value in situ without disrupting the situ—or at least giving some thought to designing ethically; to considering the positive versus negative knock-on effects—any disruptions to the social status quo we wanted to inspire and those we did not. I was talking about social norms in the places people frequent. Whether or not, for example, it is socially acceptable to whip out said devices and applications in public. I was not meaning to talk about ‘like’ buttons and ‘share’ buttons and diffusion models. I was talking about the whys and wherefores, not the hows, whats and whens.

I was silently channeling ergonomists and social scientists who address the physical and/or social constraints of the situation, which can in fact mean that an attention-grabbing application would be dropped in favor of physical and social comfort. I was talking about the potential contexts of use for any application we would build.

I am not saying we should have indulged in an infinite regress into consequences of every design decision, but I am saying it is worth engaging in chain-reaction thinking. I was, however, inviting us to understand what the problem was, where there would be something useful for people, before we start coming up with solutions. It seemed to me that we needed to understand what we were designing for before we starting designing.

I just started the conversation poorly.

I am trying to make a bigger point here, beyond the natural human tendency to trot out well-worn, already known processes, and avoid thinking about hard things. I am also highlighting different ways of seeing the world. I am not a social linguist, but in my gathering there was a clash of understandings. Even in a group that believes it is aiming for the same goal, words mean different things to different people and provoke different semantic associations. This has consequences for collaborative, multidisciplinary design teams. People from different disciplines come from different “epistemic cultures”, a concept I am borrowing (and hopefuly not mangling too much) from Karin Knorr Cetina’s book on the ways in knowledge is created in scientific endeavours. Epistemic cultures are shaped by affinity and necessity; by the ways in which things that arise are constructed as issues or problems or growth opportunities; and also by historical coincidence. An epistemic culture determines how we know what we know. Think of places you have worked: how different organizational cultures think about and institute incentives, employee evaluation, teamwork, work-life balance, and so on. In the same way, “social” may be measured, used, and applied differently in different epistemic communities or cultures. We simply need to be alert to that fact. What this means practically is that we must be aware of the need for translation and persuasion for human-centered design in contexts where multiple parties are present—brokering between different epistemic communities and different constituencies in the design space. It also means that if one is to carry a vision from ideation to implementation and release, one had better be present for all the “decision” gates—be careful about how ideas are being understood as an application or the artifact goes from inspiration to innovation. Because all along the way, the initial vision will slippy-slide away into the details, and those details may consume the whole. Rube Goldberg and W. Heath Robinson come to mind—creators of fantastic contraptions that maybe, perhaps, get the job done, but in the most circuitous manner possible, honoring the engineering over and above utility and aesthetics.

Before I take my leave, let me ask you, as an illustration of all the different perspectives on the simplest of technologies, how many designers does it take to change a lightbulb?

Well, normally one…but:

+1 because we need to user-test the lightbulb
+1 because marketing wants to build the box and brand the bulb
+1 because sales wants to go global with the lightbulb
+1 because engineering wants details specs
+1 since we need to evangelize about it
+1 since we need to build a user community around the lightbulb
+1 since we need to explore different directions we can take the lightbulb

The golden age of newsprint collides with the gilt age of internet news

[This is an early draft of a column I wrote for ACM’s interactions magazine. It appeared here and the final version is available from here. It appeared in Volume 16 Issue 4, July + August 2009 of the magazine].

_______________________________________________________

Sitting in the Economy Class seat on a United Airlines flight, I ducked for the third time as the gentleman next to me struggled to turn the page of his broadsheet newspaper.

While he was assimilating what was happening in the world, I was contemplating the unfortunate juxtaposition of two iconic forms – the over-sized broadsheet newspaper and the undersized airline seat – and the current state of two industries that find themselves in deep financial trouble.

News stories. Crosswords. Horoscopes. Book reviews. Political cartoons. Recipes. Print-dirtied fingers. Papier mache. Stuffing sodden shoes. Wrapping fish and chips. Ad hoc packing materials. Fire kindling. These are things that I think about when I think of newspapers. And despite the fact that I could never quite physically control a broadsheet without the aid of a table, I cannot believe that this everyday artifact may go away. But according to my friends here in the digiphilic environment of San Francisco it is inevitable – you can’t walk into a coffee shop, never mind turn on a TV or the radio without hearing someone opine about economic crisis that newspapers are facing and the likely disappearance of the daily rag. I am as shocked and mortified by this as I was by the 2003 news story that bananas may be extinct by 2013.

Newspapers have a long history. The first printed forerunners of the newspaper appeared in Germany in the late 1400’s in the form of news pamphlets or broadsides, often highly sensationalized in content. In Renaissance Europe handwritten newsletters circulated privately among merchants, passing along information about everything from wars and economic conditions to social customs and “human interest” features. In 1556 the Venetian government published Notizie scritte, for which readers paid a small coin, or “gazetta”. The earliest predecessors of the newspaper, the corantos, were small news pamphlets that were produced only when some event worthy of notice occurred. In the first half of the 17th century, newspapers began to appear as regular and frequent publications. The first modern newspapers were products of western European countries like Germany (publishing Relation in 1605), France (Gazette in 1631), Belgium (Nieuwe Tijdingen in 1616) and England (the London Gazette, founded in 1665, is still published as a court journal). These periodicals consisted mainly of news items from Europe, and occasionally included information from America or Asia. They rarely covered domestic issues; instead English papers reported on French military blunders while French papers covered the latest British royal scandal. Newspaper content began to shift toward more local issues in the latter half of the 17th century. Still, censorship was widespread and newspapers were rarely permitted to discuss events that might incite citizens to opposition. Sweden was the first country to pass a law protecting press freedom in 1766. Timeliness was always an issue; news could take months to reach audiences. The invention of the telegraph in 1844 transformed print media. Now information could be transferred within a matter of minutes, allowing for more timely, relevant reporting, and newspapers appeared societies around the world. This was truly a revolution.

The Internet is bringing about an even bigger revolution: timeliness, open rather than controlled information sharing and easy access. This shake-up is bigger than any other that has been faced in the last 100 years from the likes of radio and television. Broadcast radio in the 1920’s was low-cost with broad distribution, and content delivery was often more timely. The newspapers responded by adding content that was not so easily represented through audio waves, providing more in-depth and visually vivid coverage of key stories. As the 1940’s and 1950’s came around, television appeared as the main challenger. Newspapers again responded, taking from television the short, pithy story format. Newspapers like USA Today responded with graphics and colour imagery. More generally, news publications started diversifying their content, mixing human interest stories with puzzles, crosswords, book reviews, cartoons, cooking recipes and all the good stuff we have grown to love. Newspapers became about browsing, grazing, sharing, surfing, with content that satisfied immediate information needs and longer-term general interests.

Despite radio and television, newspapers managed to retain their position in the information value chain. Not so anymore. There are three interrelated causes for this shift in the information ecosphere: internet-related innovations in news production and news dissemination; the impact of new digital devices that are changing the ways in which content is consumed; and a no longer viable business model.

Lets quickly look at these in turn. It is obvious that the Internet has revolutionized news production and dissemination. Speedy transmission of information around the globe means news can reach us as events are unfolding – hot off the keyboard rather than the press with images and video for that “being there” feeling. “Citizen journalists” give us the lay-person’s perspective on events that journalists cannot or have not yet reached. Indeed, reports from various disasters from the fires in California to the shootings in Mumbai came for many people first from Twitter, the micoblogging service that is currently the darling of the media and blogs from Iraq told us much more that we could possibly find out from our daily newspapers. The efficiency and effectiveness of this interconnected internet world cannot be denied. Production and consumption of news has also been transformed by the explosion of lightweight, wireless, internet-enabled recording and reading devices, plus the proliferation of computers in the home and in offices. Finally, the old business model is failing. The newspaper industry in the US has been generating most of its revenue from advertising for decades. The global recession and the resulting decline in advertizing revenues has dealt a possibly fatal blow; the Newspaper Association of America reports that in 2008 the total advertising revenues declined 16.6 percent to $37.85 billion, representing a $7.5 billion reduction on numbers for 2007. Proposals on the table for saving the industry now include micropayment schemes plus bailout and/or government subsidies.

I don’t feel qualified to assess the likelihood of success for the various bailout schemes from micropayments to government bailouts, and for the purposes of this column, I will not go into the importance of ensuring we don’t lose good journalistic practice. But I am really worried about what is happening in the world of news, because I am screaming for a better news reading experience on my desktop and mobile devices. What the news industry at its best did really well is missing from the online reading experience: easy navigation of well filtered content plus effective selection and segmentation of content plus a clear voice/view of the publication. Can we take the best of what we had in newsprint and create a good digital news reading experience? Here are some basics I would like to work on:

(1) information quality- can we provide better tools for the collection and management of information gathered on the ground that would aid with quality and provide guidelines for the coupling of different media types (text, imagery, video) to avoid gratuitous visuals? Let’s be active in designing better technologies for production of the news by citizen and professional journalists and editors.

(2) information architecture design: can we design better relational models so we can surface relationships between stories that are actually meaningful instead of the ‘also see’ hyperlink that takes me to a story from 5 years ago that somehow got linked to the current one? Can we design better tools for following story developments, for enabling the creation of narrative by producers and consumers?

(3) can we improve the representation of information – graphics, fonts, layouts – so that it is possible to skim more effectively?

(4) can we design for reading the news – what is next after the Kindle? Is electronic paper or Xerox’s promised reprintable paper going to be a reality so I can have the large gesture, embodied experience of the broadsheet back and decent screen real estate for laying out content?

(5) can we design anything better than the crass, ugly and inconvenient model of url bookmarking to support different temporalities of information usefulness and different consumption paces, and for slow-burn stories to persist while fast-burn stories are updated with new content?

In sum, we should more deeply address the practices of news readership. We should design for convenience and skimming. We should design filters and surfacers of quirky items or items that for some reason search algorithms find unpalatable. We should develop better editorial tools than we currently have.

I am not alone in wanting some good design heads on these problems. Addressing people’s everyday news consumption practices, a 2008 Associated Press ethnographic study cited email and internet-based sources as a mainstay many young people’s experience of the news. However, these interviewees, plus interviewees in a study I am currently running in the Bay Area all talk about the “work” of reading the news online and that “news fatigue” is increasing. What this seems to boil down to is that there are plenty of places to find news on the internet, but in all this bacchanalian information glut the shallow story dominates, often it is hard to find the follow-up to a reported news item, and there is a lot of repetition. To the last point, the Project for Excellence in Journalism observed in their 2006 State of the News Media report that 14,000 unique stories were found on an internet news aggregator site in one 24 hour period, there were in fact only 24 discrete news events. There is vastly more content available of course, and things have improved somewhat since 2006, but that other content is relatively speaking hard to find. And online content usually does not offer the structured, well-designed experience that its printed counterpart does. Ethan Zuckerman of the Berkman Institute blogs about his experience of a national newspaper’s online presence “…counting possible links (using a search for anchor tags in the source HTML), there are 423 other webpages linked from the front page. A more careful count, ignoring ads, links to RSS feeds and links to account tools for online readers, gives 315 content links, possible stories or sections a reader could explore from the front page. While there are almost 14 times as many pages for a reader to explore, they’ve got much less information on what links to follow: while twelve stories have text hooks, the wordcount ranges between 10 and 26 words. While there’s a good chance one of those stories might convince you to click on it, you won’t start reading it on the front page, the way you might with the 200-400 word stories in the paper edition.”

I just replicated his analysis by looking at three online papers. He’s right.

But let’s not get ahead of ourselves or judge too early. We are only at the beginning. Right now we are at the stage the car was in the last 1800s and early 1900s – in shape and form reproducing the horse drawn carriage, not yet having found its own aesthetic reflective of its infrastructure and capability.

Newspaper companies are on board with enlisting others to aid in the design of the next generation of news forms. In early 2009, the New York Times Developer Network hosted its first API seminar so we can start designing and building new forms of content provision. The aim is to make the entire newspaper “programmable”. Programmers will be able to mash up the paper’s structured content — reviews, event listings, recipes, and so on. This is a great opportunity for those immersed in information and experience design.

Some are for example pushing on really nice “read later” and bookmarking facilities. This is a good start. But you still have to know how to search for content, and spend time doing that. In my opinion, today’s aggregators/algorithms and automatically updated webpages are in no way shape or form replacing the work done by a good editor and a good layout designer.

As I have been thinking about news, I conducted an informal review of a couple of local and national newspapers. Bearing in mind that I am only am interloper who is curious because reading the news is integral to my identity, I took a quick review of the differences between online and print versions. I could not discern any consistent re-representations between online and print. I wonder: Are there standard reformulations and standard channel “jumps” for different content types? Who is making those decisions and how? How is the “shelf-life” or temporal relevance affected by the channel or the medium? And…… what of the services that were once the purview of the local newspaper have not been reformulated or replicated? What has happened to the content that once constituted the local daily paper? How has it morphed and reformed? Finally, and of course, I note that I have not seen many people argue for the fact that the very form of the newspaper, it’s affordances (size not withstanding), may be missed. No indeed, the print form of the newspaper still has affordances that cannot be matched by the digital medium. These are, to me at least, complementary forms, but perhaps I am in the minority; I still print things out rather than reading them on a screen.

And, there is another affordance about paper. It gets left lying around, apparently discarded but ripe for re-reading. That is not the case with contemporary digital devices, although who knows where the future may take us as device manufacturing gets cheaper. Information left lying around for others to consume is important. It allows others who are idling to encounter than which they may not have otherwise; to be literal, if I sit on a subway train and out of curiosity read the newspaper that has been discarded on the seat next to me, I am encountering something I did not choose, that was not filtered for me. And just perhaps, I will learn something unexpected.

Addressing issues in the creation and dissemination of news is important; it is not enough to say that late adopters or those who not actively seek the news as opposed to having it literally pushed through the letter box should catch up with us digerati and get those phone applications downloaded. Making it harder to get access to information affects civic engagement. Following the closure of the Cincinnati Post in late 2007, Princeton University economists Sam Schulhofer-Wohl and Miguel Garrido showed a decline in people voting in elections, fewer candidates running in opposition to the incumbents, and less knowledge of and debate around issues and policies supported by the incumbents. If democracy in some sense depends on an informed electorate, then making it harder for people to easily find digestible but detailed and well-balanced arguments is a serious problem. Even if you don’t agree with everything you read in a newspaper, encountering things that you have not actively selected broadens your outlook, the flip-side of how filtering and narrowing can save time. Specifically, filtering saves time, but also shuts down challenges to assumptions, and it is those challenges that help us grow and create the debate of a functioning, democratic society. Newspapers can of course do the opposite; they can function to bring a community together in a shared narrative (whether they agree with the narrative or not will drive whether then accept or fight it). Moving to purely digital forms, some say could increase ‘discovery’ problems for the non-digerati, and thus the number of ‘news dropouts’.

So, I confess that personally, I love the materiality of a good broadsheet newspaper and of the magazines that I read. And it annoys me just a little that, thanks to my beloved Kindle electronic reading device, I don’t have newspaper lying around the house to stuff my rain sodden shoes. But I am also looking forward to a world with better designed digital news formats. What we need is some technical savvy, a design sensibility and a deeper human-centered understanding the Gestalt of news consumption in practice between and across representational forms. We need something more that the current state of the art, which offers us only the most superficial, easy to implement, of technical convergences. We need more than the horseless carriage of digital news.

Open, closed, or ajar? Content access and interactions magazine

[This is a draft of a column I co-authored for the ACM’s interactions magazine with Mark Vanderbeeken of Experientia.  It appeared in September 2008 — interactions, ACM Press, Volume 15, Issue 5 (2008)].
______________________________________________________

Harvard’s Law School announced it would open access to the intellectual content created by its faculty members. This means content that is produced by faculty at Harvard will be available for us all to read. A driving issue behind this turn of events is the sheer cost of academic journals. For most individuals, certainly, the costs are prohibitive. But it’s sobering to note that fewer and fewer libraries can afford to stock titles that are directly relevant to academic courses.

The university’s blog post states, “the faculty voted to make each faculty member’s scholarly articles available online for free, making HLS the first law school to commit to a mandatory open access policy. Under the new policy, HLS will make articles authored by faculty members available in an online repository, whose contents would be searchable and available to other services such as Google Scholar.” Contrary to some of the negative rhetoric around openness, “open” does not necessarily mean losing control of ownership altogether; publications will be made available with copy/share-friendly licenses. The blog goes on to say “Authors can also legally distribute the articles on their own websites, and educators here and elsewhere can freely provide the articles to students, so long as the materials are not used for profit.”

Appropriately, this announcement rattled swiftly around the blogosphere. For many it represents a significant step, a step toward the vision for democratic access to information on the Internet. Commentators and open-access campaigners were articulate on what this means for individuals and for the broader intellectual community. Peter Suber, a research professor of philosophy at Earlham College, and John Palfrey, of Harvard’s Law School (also the executive director of the Berkman Center for Internet and Society and a principal investigator of the OpenNet Initiative) are both long-term open-access campaigners involved in the motion at Harvard, and both blogged the event. Writer Cory Doctorow, who has benefited from publishing novels online before moving to print, reported the event with enthusiasm on the technology and Internet “pulse” blog, BoingBoing.

Even for those not immersed in debates around open content, it is easy to see that this is an important event. From its inception, one of the founding concepts of what sociologist Patrice Fiichy refers to as the “Internet imaginaire” has been democratic access to content. Many have argued that closed knowledge bases limit innovation, stunt business potential, and reduce creative growth potential for business globally; personally, I recommend Eric von Hippel’s 2006 book, Open Innovation, and the writing of authors like Dominique Foray and Steven Weber  on open and closed innovation and the economics of knowledge.

How does all of this relate to us, the readers and writers of interactions? Well, directly. Recent debates around interactions illustrate how discussions of openness and open content are challenging traditional views of publication, content distribution and dissemination, and indeed the economics of idea circulation. This issue was foregrounded when the interactions website went live earlier this year, with a number of people expressing surprise that only the first paragraphs of articles were available for download unless one had a subscription to ACM’s digital library. Currently, two pieces in each issue are available in their entirety on the interactions site; they’re pieces that promote the magazine as a whole and the editors’ vision for the magazine. But a subscription to the magazine is required for viewing the rest of the articles.

At CHI 2008 in Florence, a discussion arose on whether or not interactions should publish more articles, and possibly the magazine in its entirety, for open download, online (acm.org/interactions). Mark Vanderbeeken, of the experience design consultancy Experientia, and I spoke on the “Interactions Magazine Comes Alive” panel. We discussed some of our personal thoughts and experiences of open content, and discussed our perspectives about making the content of interactions freely available on the Web. Some of the points we brought up there reappear here.

First and foremost, paraphrasing John Thackara, quality that is not communicated is simply not quality. To put it crudely, who cares how great the ideas are if we make barriers to hearing/reading them so high that the ideas reach only a small in-group? Closed content is restricted content, and restricted content shared among the few is likely to have limited impact.

Second, the Internet has been called a disruptive technology for the publishing industry in general. Disruptive technologies shake things up. Disruptive technologies lead to the creation of a new industry or transform an existing one. They can add value and shift market position. They can destroy existing competencies and drive/enable change in a value network. And most strategists agree that the only losers are those ossified in an outmoded business model. That said, readership of print magazines in the United States remains stable; around 85 percent of adults read consumer magazines, and this figure has not changed since 2003. So far, it looks like digital publication of articles is simply not a replacement for a carefully designed printed artifact.

Why? Well, for starters, print and digital artifacts simply have very different properties. They invite a different interaction, and the experience of the content is radically different. For example, I am a big fan of O’Reilly’s Craft magazine. I have given copies as gifts to a number of people. I like to annotate my own copies with scribbles and notes. I share my annotated copies with friends. Copies sit around the house visibly waiting – plaintively calling out – to be read. And, crucially, my magazines can be read in the bath and can get just a little soggy without catastrophic effect.

The layout of a magazine differs greatly from the layout of content on the Web. Flicking through a magazine and stopping when something catches your eye is different from browsing through windows, pointing and clicking. Tearing a page out feels different from bookmarking or printing a page. In short, the Web page and the magazine have different informational, tangible and aesthetic properties and ‘affordances’. And in my opinion, the core product, the print magazine, is not going away.

Digital artifacts should invite the reader to desire the physical artifact. And vice versa: I go online to sites like ravelry.com, to instuctables.com, and to YouTube.com to find other crafters and to see videos of how to do something—some skills, especially motor skills, just don’t get communicated as well in static print. I believe that medium matters, and we need to take seriously the careful design of a complementary relationship between the two.

Third, there are many kinds of value, aside from charging hard currency, for content. Value may be purely nonmonetary. For individuals who do not charge for their labor to produce content, the value may be personal satisfaction, or reputation and contribution to the community. For companies who produce materials, like print magazines or web sites and blogs, the value the artifact provides may come from a subscription, it may come from where the content points people to perhaps spend later. The artifact may be a link in a chain of value but not the point of monetary exchange.

Mark’s report of Experientia’s strategy on content sharing renders abstract assertions around value generation concrete. Experientia has demonstrated that the paradigm of company information as proprietary, protected at all cost, is now completely obsolete. Rather, an alternative approach is being taken there: Everything not protected by NDA or of strategic value (e.g., the markets they plan to address in the next four months), should be open to all. All important content and ideas are published on the company blog, Putting People First. The blog started out as something internal. However, it was not protected, and before long, it was getting more and external visitors. The team decided that was not such a bad thing. Many visitors now come to the blog from major international companies like Yahoo!, Intel, Microsoft, Motorola, Nokia, Philips, and Samsung.

The company has directly experienced several benefits of this approach:

1. A channel—Putting People First provides an easy-to-handle communications channel (no email newsletters, no expensive advertising campaigns);
2. An audience—usually about 6,000 professionals a day
3. A reputation—a company that has its finger on the pulse of what is going on;
4. A brand—substantial brand recognition for a company that is not even three years old;
5. Loyalty—readers feel involved in the content that is given for “free” (Putting People First has become a professional research tool for many, thanks also to its categories and effective search engine), and this openness creates loyalty;
6. Regularity—the blog is updated nearly every day, so to some extent Experientia is constantly in the minds of its readers;
7. Relations—people contact the company regularly based on the blog; the blog offers a social nexus;
8. Jobs—Experientia even got a few new clients, although that is not the main reason for the firm’s doing this; if anything, PPF confirms the firm’s reputation rather than landing new clients out of the blue;
9. PR opportunities—Experientia staff is regularly invited to conferences and offered writing assignments based on their perceived qualifications;
10. Dialogue—reactions and reflections on what is going on, either informally or publicly, either directly, or because people link and relink to the site

As the Experientia example suggests, the value-add of the open content is the ripple effect—the other things that become known, which do generate monetary reward. In the case of scholarly journals and magazines like interactions, much of the labor of content production is volunteered, not for monetary gain. But the labor fits within a system in which the rewards are very real—promotion of ideas, of products, of companies, of self, personal satisfaction, growth of future opportunities.

Printing a magazine has a price tag of course. The costs of magazine production include editing, illustrating, layout, printing, distribution, and archiving. Online distribution does not erase operating costs; funds are needed to cover platform and interface development and maintenance, promoting, and archiving. The revenue model currently in place to cover these costs is subscription, or what has been called “reader page charges.” Other models that we can start playing with are:

• Free access after an embargo period—charge those who want content immediately, but after a while the content can become freely available; this is one of a number of possible tiered revenue models
• Author page charges—charge authors for the content
• Institutional, governmental, and vested agency payment—with open content in the academic domain, many argue that taxpayers have already paid for government-funded work through taxes, so the results should be freely available
• Advertising—arguably, the model that drives much of the Internet
• Sponsorship is another possibility—this could be issue-based sponsorship or section-based sponsorship

Mark and I argued on the panel that there is a great opportunity with interactions to generate interest and gain momentum around important sociotechnical design issues through a thoughtful, well-written, and well-edited magazine very much appreciated by the professional user-experience community. interactions provides—as named above—a great entrée point: to ideas, people, and potentially, to our funder, the ACM.

We sparked debate from the panel with ideas that simply scratch the surface on the issues involved. What are your views? Are you someone who would/do pay for the subscription, who would pay to download the articles? Do you have artful suggestions for business models not explored in this brief article? So the question is, what does open content mean for a magazine like interactions? Please share your thoughts on the interactions site.

Keep your hair on: Designed and emergent interactions for graphical virtual worlds

[This is a draft I wrote for my column, P’s&Q’s in ACMs interactions magazine. For it, I interviewed Bob Moore who was at the time with now defunct Multiverse. It came out in Volume 15 Issue 3, May + June 2008. The final version can be found here.]

______________________________________________________

Chatting with virtual world researchers Jeffrey Bardzell and Shaowen Bardzell, I found out that a seriously desired artifact in Second Life, not unlike in First Life, is hair. Swishy, shiny hair. But there is one problem with this real-dreamy hair: Such hair is computationally costly compared to render in comparison to your average avatar body. And so, sadly, your avatar body arrives before your hair. For a matter of moments, no matter how fashionable the outfits, everyone is bald as coots.

And it’s worse than that. Often from other players’ perspectives, your hair has failed to “rez,” but not from your perspective. So you think you look hot-REALLY hot–until, that is, some newbie says, “Why are you bald?”

There are now apparently a lot of people routinely spending time hanging out in virtual worlds. MMO Crunch reported 36 million regularly active MMORPG players in August 2007. And they are getting more popular.

“Virtual worlds are fundamentally a medium for social interaction. One that takes the face-to-face conversation as its metaphor. As such it leverages users’ common sense knowledge: I see a humanoid avatar and I know that if I want to talk to that player, I should approach his or her avatar with my own” says Bob Moore interaction researcher and designer at Multiverse. As with all metaphors, however, there are fractures in the metaphor. Which matter and which don’t? When do we happily immerse and feel like we are really there and when do we get amusingly or annoyingly jolted by something that doesn’t work? What are key things designers are thinking about right now? Bob and I chatted for a while, reflecting on the work that has been previously done on the research areas of graphical worlds, collaborative virtual environments (CVEs) and online gaming environments. Given my own background in researching text-based virtual worlds and animated interface characters, I was intrigued to her Bob’s perspective. Here are some of the main points we covered in our conversation:

(1) For virtual worlds, user interface design is also social interaction design

Bob argues that one thing UI designers for virtual worlds have not yet fully grasped is that when users are interacting with the system, they are often at the same time interacting with other users. When standing avatar-to-avatar, if you ask me, “Did you get the sword?” and I promptly open my inventory, that UI action is a relevant part of the interactional context. Or if I say, “Let’s go” and you promptly open your map, that’s a relevant next action that I should know about. In almost all current virtual worlds, opening your inventory or your map triggers no public cues, only private cues for the individual user. So you not only need to give the individual user feedback about what the system is doing, also need to give the other players feedback about what the individual user is doing. In other words, users’ interactions with the system should be made public.

(2) Where avatar bodies are not like physical bodies

Unlike real life, most people tend to play virtual worlds with their camera view zoomed back so they can see their avatars rather than in true first-person view where you can only occasionally see your hands or legs. There are a couple of good reasons for this. Computer screens don’t allow for peripheral vision. But pulling back the camera can help mitigate this limitation by widening your field of view. Similarly, as Bob points out, avatars don’t allow for proprioception, or our awareness of the positions of our body. Zooming back the camera also helps players deal with this fact. I may think my avatar should be waving like the Queen of England or winking flirtatiously at someone else because I typed /wave or /wink, but it’s hard to be sure if I can’t see my avatar. Maybe I mistyped the commands or maybe the animation associated with the command actually looks more like a New Yorker hailing a taxi than Her Royal Highness.

(3) When the conversation lags behind the avatars

Despite a lot of the interesting audio experiments that were being conducted over 15 years ago–see Benford, S. D. and Fahlén, L. E., Awareness, Focus, Nimbus and Aura – A Spatial Model of Interaction in Virtual Worlds, Proc. HCI International ’93, Orlando, Florida, 1993 as a good example–most virtual world conversations take place through chat. The avatars may be wandering about gesturing and wiggling their hips, but chat does not come out in audio from the mouth of the avatar. One problem this causes is the discontinuity, the lack of congruence, between action and uttered words. Bob points out that typing a chat message is another kind of action that other players should know about, and he recounts a case in which a team of players is about to attack a group of “mobs,” or computer-controlled opponents. While one player is composing a question about how the team might change its tactics, a fellow player initiates combat. The tactical question then publicly appears too late. The pseudo-synchronous chat lags behind the synchronous avatars. Bob, who was trained in Conversation Analysis, explains that a key feature of real-life conversation is that you can hear a turn unfolding in real-time. This enables you to do things like determine who should speak next, anticipate precisely when the turn will end so you can start your next turn with minimal gap and overlap, and even preempt the completion of the current speaker’s turn if you don’t like the direction its going. In other words, the ability to monitor other people’s turns-in-progress is a requirement for tight coordination in conversation. Most virtual worlds (with the exception of There) use IRC- or IM-style chat systems, and therefore, do not allow players to achieve this tight coordination among their turns-at-chat and avatar actions. The result is an interactional experience that feels very unnatural (at first) and which motivates players to invent workarounds to the system.

(4) Perennials of place

One of the amazing things about virtual worlds is how quickly we get a sense of being co-present in a place with other people, even though it may be an image on a screen, a world into which we are kind of peering. And, as in the real world, ambience is created by building and room size and scale in relation to crowd size. In my research on MUDs/MOOs with Jeni Tennison and then later work with Sara Bly, we found that even very simple text exchanges in textually described “rooms” can make dyads and groups feel co-present, imersively in the virtual world together. Humans tend to get engaged with each other as long as there is some consistent chain of action-reaction. Indeed some analysts would argue that turn-taking in conversation is the fundamental unit of human communication and connectedness.

As we explored these concepts, Bob described a comparative ethnographic study he’d done of bars and dance clubs in multiple virtual worlds. He hung out in these social public spaces and analyzed features of the design that impacted the success of the space as a social environment. One key feature of club design is size. Bob discovered that, while construction in these worlds is cheap compared to real life, it is more difficult to fill these spaces with people than it is to fill the real-life urban centers that researcher William H. Whyte examined. As a result, the dance club in City of Heroes and the majority of player-built clubs in Second Life are simply too large. They feel like an airport terminal or concert hall rather than a corner pub.

So in order to achieve the kind of social density necessary for a vibrant social space, or “third place” as academic Ray Oldenburg would call it, designers should make virtual bars and clubs much smaller than they currently do. The most successful virtual third place that Bob discovered was a Second Life bar that was intentionally tiny. In order to get into the place, you had to “rub elbows” with other patrons. The place felt “busy” with only five players and “hoppin’” with twenty. And everyone was within everyone else’s chat radius, which facilitated the public conversation. In other words, lessons from real-life urban design appear to apply in several ways to the design of virtual public places.

(5) A range of skill sets and a modifiable world.

There are challenges in designing a world where “newbies” or newcomers can learn the ropes. If you want to learn about interacting in one, you simply have to get off (or rather for most people onto) the sofa and get in there. Go in-world. It is much easier to learn in-game than to learn out of game, just like learning to play golf requires you take up a golf club and try it. You just can’t learn by watching someone else. Bob says, “I’d recommend getting into a pick-up group and go and doing some adventuring”. There are plenty of folks in-world who are willing to help out, to show off their knowledge. Bob agrees that not everybody likes to help, and admits that whether you are more of less likely to be helped depends on some other factors……you guessed it-having an attractive female avatar means you are more likely to get help.

There are other things to learn aside from interactions and activities. Many virtual worlds allow people to buy, build and exchange things. Second Life is perhaps the primo example, it is a sandbox with a constructive geometry, which enables them to stream your data and create your objects on the fly. But that also means people arrive and build stuff and leave it behind. There is a certain  “I just learned to build today” look that Bob identifies as one of the main scars on the aesthetics of the virtual world. Despite the parallels that some people make between the real live music/art/dance festival known as the Burning Man Project and playful interactions and explorations in Second Life, there is no motto inviting us to “leave no trace” in Second Life. Of course the ecological consequences are somewhat different but the visual aesthetic of clutter and detritis is experienced as the same for Second Life afficionados. So, there is a tension between giving people freedom to build anything they want and making sure the world doesn’t end up looking like the aftermath of an afternoon in a Montessori School for Gremlins. Beyond clean-up though, there is another point. At this juncture the building tools are much better than they were but they still leave a lot to be desired. I say: If we want Gaudi not Brutalism, we need to provide better tools to scaffold the building endeavours of the folks in-world.

To get an understanding of issues like these above and push on understanding how people really experience these virtual/online places, Bob advocates a close and detailed analysis of what is actually going on as it unfolds in real time, looking at patterns of action and interaction, how those patterns develop and are understood, learned and evolved, and identifying patterns that are persistent and prevalent. Too many people have theories about what is going on that are based on something completely external to the situation. “You need to get close to the phenomenon/experience” says Bob. And we need to “be concerned both with the in-world simulation of face-to-face interaction AND the usability of the interface for puppeteering the avatars and interacting with the system.”

By looking at the challenges in interaction that people routinely encounter and work around, it is possible to ask how important – or disruptive to interaction in-world – those challenges are, and propose ways to address them through interaction, interface and system (re)design. I could not agree with him more.

And on that note, it’s time for me to get my hair on and go build a shack.

Reference: Bardzell, S., & Bardzell, J. (2007). Docile avatars: Aesthetics, experience, and sexual interaction in Second Life. Proceedings of British HCI 2007. Lancaster, UK