More than a feeling

This is a draft of a column I wrote for the ACM’s interactions magazine. It will appear late 2011.


Mildly irritated. Frustrated. Somewhat annoyed. Profoundly agitated.

This was the trajectory of feeling I experienced this afternoon as I ascended and descended a customer service phone tree in the hopes of reaching a human who could (or would) answer my question. As I reached a flushed state of agitation, I had a meta-moment. I was catapulted out of my immediate experience into a view onto self from a distance. You are seeing red . Your reaction is unreasonable. Stop. Get a grip. Take a deep breath. Hang up the phone. [1]

As a result of this incident, in the longstanding tradition of armchair philosophy I started to ponder: What are feeling, affect, emotion? Was I really being unreasonable having an emotional reaction, by feeling agitated? What are reasonable and unreasonable in this context? Would others have been swept away with agitation and simply shouted at the person who finally answered? Would I have felt better if I had done that? Aside from hiring more people, what, if anything, could the company have done to prevent this ascension to annoyance? What was clear to me was that a tinny rendition of Antonio Vivaldi’s Four Seasons was not cutting it in the keep-Elizabeth-calm department.

For decades human computer interaction professionals have been thinking about cognition and how to present choices in ways that are intuitively obvious, and we’ve worried about how to measure emotions like frustration for almost as long. We know that perceived options change and that people make different choices among the same set of options depending on their emotional state.

Well, I pondered, perhaps we can detect subtle cues about someone’s state, and based on that, present more copacetic options and different interaction experiences. It is postulated that people have visceral, pre-cognitive positive or negative reactions to things [2]. Perhaps someone’s state can be detected before the person themself realizes how they feel? Perhaps, there could be sensors in my phone handset that detect my emotional state and initiate cheer-up, calm-down or fall in love sequences? Evidence suggests that visceral responses aren’t just in response to an artifact or situation; they can be modulated by social factors. “Affective priming” studies suggest that others’ emotional states can affect our decision making even if we are unaware of it: people take more risks after being subliminally exposed to smiling faces than to frowning ones [3]. If the service I was calling had subliminally played laughter down the phone line, just maybe my ascension to agitation would have been prevented.

As I idly entertained myself with these technospeculative reveries, I came across a discussion of innovations in sensor technologies which can reliably tell us what people are feeling, can accurately map their emotional state. Frankly, I had cast aside my reveries as overly deterministic and somewhat creepy, the stuff of potions and hexes, and way too much of a wicked design/engineering problem [4]. Such a scenario, I reasoned, requires accurate detection, a good model of emotion and mood and it’s impact on thought and action, and an understanding of how the person’s reactions and their surrounding context are going to interfere with any initiated interactive sequence. I decided to turn to research to see if there exist grounds for these optimistic, techno-detection-of-emotion, media narratives.

The plethora of disciplines interested in emotion (for example, biology, linguistics, psychology, sociology, cultural studies, anthropology, design, literary studies, performing arts…..) suggest many have pondered: What is the relationship between emotion, thought and action? How can we detect and measure emotion? How can we assess the influence of emotion on thinking and action? Some conclude that emotions are an annoying flaw in an otherwise perfect reasoning system. Others assert that emotion/feeling and reason are intertwined, that it is futile to assert emotion-free cognition is even possible. In 1994, António Damásio argued from neuroscientific evidence that emotions play a critical role in cognition and provide the scaffolding for the construction of social cognition and underlie all human consciousness [5]. Armed with Damásio’s perspective, I delved deeper into theories on emotion, hoping to find more evidence as the nature and measurement of emotional influence on human behavior. The theories can be crudely bucketed into three types:

• naturalistic theories, which maintain that emotions are products of natural processes which are independent of social norms and conscious interpretation; they result from hormones, neuro-muscular feedback from facial expressions, and genetic mechanisms,

• interactionist theories, which acknowledge culture’s importance yet suggest that enduring and universal biological mechanisms provide for core emotional features (social aspects are cast as derivative, contingent, and variable)


• social constructionist theories, which maintain that emotions depend on a social consciousness concerning when, where, and what to feel as well as when, where, and how to act.

There are more or less extreme versions of each; the more extreme flaunt clear ideological stances. For example, naturalistic theorists like Robert Zajonc associate emotions with spontaneity, creativity, sociability and passion, they are more authentic and are preferred over trained, ‘cold’ reason [6]. A whole research agenda exists to illustrate that we process emotional cues (preferences) separately from making reason-driven decisions (inferences) . Much of the interactionist perspective is rooted in the work of Charles Darwin, the father of evolutionary theory, whose view on emotions is that they are gestures that hark back to basic human behaviours—love is the vestige of copulation, angry teeth-baring a vestige of biting. Returning to my own telephonic experience, I confess I did not bare my teeth at the handset. However, had I done so, my Darwininan ancestral counterpart would simply have bitten it.

Other researchers focus a great deal on responses in the sympathetic nervous system; elevated activity is known as arousal, and arousal is deemed positive or negative. Reading articles in this tradition, I have to say there seems to be a conflation of body arousal with emotion and feeling. In my view, arousal tells you something happened, but not how the individual felt, what emotion they experienced nor how those around behaved in response to the same incident; these factors are likely to have a major influence on the emotion as felt and whether it persists or not. Moving from detection of usually non-visible sensory nervous system arousal to visually available body reactions, Paul Ekman’s work on emotion and facial expression is perhaps the most famous work from the interactionist camp. Ekman suggests there are a few core or “basic”, universal, biologically generated, emotion-revealing facial expressions—these reflect anger, disgust, fear, sadness, surprise and joy [7]. Ekman acknowledges that most of the elicitors and expressions (“display rules”) of emotions are socially learned and can thus vary.

Ekman’s allusion to ‘variance’ is also interesting, of course. People modulate their behaviors quite substantially depending on social context—raising children is largely about trying to infuse a framework for socially appropriate emotional displays given different social contexts. Modification of spontaneous physical expressions is part of enculturation, and cultures vary. Anthropologists like Dewight Middleton propose different cultures have ‘emotional styles’, which not only guide what we consider to be reasonable feelings for any situation but also how to enact that experienced feeling [8]. These ways of enacting feeling are part of what are called the ‘techniques of the body’, a concept introduced by Marcel Mauss in 1973 to describe highly developed body actions that embody aspects of a given culture—how one walks, eats and emotes are learned through explicit instruction, through subtle approval/disapproval cues and through postural mimicry, and they can reflect one’s gender and class. Our physical expression of emotion including our facial expression is modulated by the company we are in, a problem for systems that claim to detect anything but the most extreme of emotional displays with any degree of universal reliability [9].

Mapping the physiological to the psychological to the social and cultural is important; however, to replace biological determinism with cultural determinism is not satisfactory either. Social constructionists assert that there are two kinds or classes of human emotions; one class is universal and has analogues in animals and can be seen in human infants (e.g., joy, sadness, fear), and the other is adult human emotions which are culturally variable. Through enculturation emotions lose any ‘natural’ or ‘spontaneous’ quality because they are mediated by culturally located social consciousness. Arlie Russell Hochschild’s work is a good illustration of this [10]. Her position is that society and culture provide “feeling rules” and prototypes of feeling that focus on particular kinds of emotion . In the West we feel grief at funerals but we’re supposed to look happy at parties no matter how dull they are; of course, we are perfectly able to embrace other ways of behaving at funerals and parties should we choose to do so. In addition to modeling appropriate postures and behaviors as discussed above, people verbally communicate culturally sanctioned or appropriate reactions in saying things like “You must be delighted”, “Don’t fret over that” or “You should be absolutely incensed” when others narrate incidents. Adult emotions, Hochschild says, involve personal appraisal of things like attachment (to things, situations, people and outcomes), agency and approval/disapproval. Such appraisal involves other entities: for example, envy is a name for noting an attachment or desire for something, and noting that another has it; sadness focuses on something or someone liked/loved that is not available. Hochchild addresses a range of complex emotional states including sadness and grief, nostalgia, depression, frustration, anger, fear, indignation, disgust, contempt, guilt, anguish, love, envy, jealousy, shame and anxiety. Familiar to most adults, these emotions do not map easily to reductive concepts like arousal or positive/negative affect. Rather, they require that we consider the personal, social and cultural locatedness of feeling and emotion. Hochschild also introduces the very useful concept of “emotional labor”—the effort to try to actually feel the “right” feeling for a situation, and to try to induce the “right” feeling in others [11]. Appraisal and emotional labor are not just about the current situation or the past; they also involve imagined futures. Personally, my breaking point on the phone tree debarcle was when I realized I could not be relied upon to engage in the emotional labour of masking my irritation. As a result, my concern was that I’d say something I’d regret when a person finally came on the phone. Imagining the future-self being embarrassed is a great way to restrain the inner jackass.

Walter Benjamin in Illuminations wrote “In every case the storyteller is a man who has counsel for his readers.” It’s true. My main message is not, however, ‘avoid phone trees at all costs’ ; that’s my secondary message [12]. Rather, as content and communication tools proliferate and we design people’s experiences with and through those tools, it behooves us to know why and how people react emotionally and express, perceive and communicate their feelings, not just verbally but also pre- or para-verbally. We need to be sensitive to norms in different cultures, and what happens when we design across them. And, while automated detection of arousal and affect are a very promising for connecting with people who have little insight into their own feelings or cannot communicate their emotions easily, let’s please be wary of reducing the entirety of human emotional experience to simplistic, deterministic, electrochemical models.

Cycling back, to solve the phone-tree problem, my solution would not be to save up for an fmri scanner, nor to measure my galvanic skin response, nor have a webcam trained to my face to detect my expression. Without claiming any universality but knowing myself pretty well, I’d hook something up to listen to my voice, my intonation [13]. Following the old saying, “It’s not what you said, it’s how you said it”, tone of voice includes the timbre, rhythm, loudness, breathiness and hoarseness of how something is uttered. Personally, I seem to follow the general rules: softer tones and pitches are associated with friendliness, higher tones and pitches signal upbeat and happy, and clipped and louder tones signal irritation. Let’s suffice to say when I finally hung up the phone, I was not intoning in a kittenesque fashion, nor in a light, cheerful tone. I was gearing up to bark.


[1] In their 1980 book Metaphors We Live By, they suggest the body is directly reflected in metaphors we use. They suggest that “our concept of anger is embodied via the autonomic nervous system and that the conceptual metaphors and metonymy used in understanding and are by no means arbitrary: instead they are motivated by a physiology”. Feeling anger is “seeing red” because it reflects bodily reactions—when people get angry, as a result of soaring cortisol levels, they heat up and often faces flush red.

[2] See Don Norman’s Emotional Design for a lovely exposition on this.

[3] People who are asked to make more complex financial decisions—for example, whether to gamble $1 for $.50 chance of winning $2.50—are primed with subliminal happy faces or sad faces. Those shown happy faces were likely to choose the investment than people primed with angry faces.

[4] ““Wicked problem” is a phrase originally used in social planning to describe a problem that is difficult or impossible to solve because of incomplete, contradictory, and changing requirements that are often difficult to recognize. Moreover, because of complex interdependencies, the effort to solve one aspect of a wicked problem may reveal or create other problems.” See

[5] Antonio Demasio (1994) Descartes’ Error: Emotion, Reason, and the Human Brain

[6] In his 1980 American Psychologist paper, “Feeling and Thinking: Preferences need no Inferences”, Zajonc asserts “affect dominates social interaction, and it is the major currency in which social intercourse is transacted.”

[7] Neither Darwin nor Ekman nor their followers, I note, have managed to give us any advice on emotion recognition in the world of cosmetic modification. The American Society for Aesthetic Plastic Surgery reports over 2 million procedures in 2010 using Botulinum Toxin Type A (the active ingredient of products like Botox) to reduce facial lines. Eyebrow raising and frowning, clear signals of interest and affect, are not what they used to be when this research thread was first hatched.

[8] Dewight Middleton, Emotional Style: The Cultural Ordering of Emotions, 2009, Ethos, American Anthropological Association

[9] I also note that, to date, the most successful biofeedback systems present motivated people with data that allow them annotate and moderate their own behaviours; they don’t presume to mimic the careful collaborative choreography of emotions as we might achieve with a sensitive human friend. See research into the Quantified Self.

[10] The Managed Heart: The Commercialization of Human Feeling (1982) Berkeley: The University of California Press.

[11] See also

[12] If you are going to use one, refer to first; it’ll tell you what to do to get to a human operator in the most expeditious of ways.

[13] I note that well-publicised affect detecting companies like Affectiva (whose technology is based on research conducted in Rosalind Picard’s Affective Computing Lab at MIT) do not use voice, but rather focus on the reactions of the sympathetic nervous system and facial expressions. This is, in large part, because they technologies were originally designed for people for whom speech is difficult; e,g, those who suffer from autism or babies. See

Missing the point in gesture-based interaction

This is a draft of a column I wrote for the ACM’s interactions magazine. It will appear mid 2011.


“Zhège” she said, pointing emphatically at the top right of her iPhone screen.  She leaned further into the gap between the passenger and driver seat of the taxi. Then, lifting her head, she pointed forward through the windscreen in a direction that, I assumed, was where we were hoping soon to be headed.

The taxi driver looked at her quizzically.

Undeterred, she repeated the motion, accompanied by a slower, more carefully enunciated rendition of the word: “zhège”. This time she added a new motion. She pointed at the bottom left of her iPhone screen, at herself, at the taxi driver himself, and then at the ground below us. Balletic though this motion was, it did not reduce the look of confusion on the driver’s face.

Gently taking the device from her hand, he studied the screen. A moment later, his expression changed. He smiled and nodded. He stretched out the index finger on his right hand, pointed to the location on the screen she had isolated, and said “zhège”. He handed the device back to her, flipped on the meter, and grasped the steering wheel. A second later we accelerated out of the taxi rank. He had understood the point of her point(s).

My traveling partner, Shelly, and I know precisely 6 words of Chinese. ‘Zhège’ is one of them. We cannot actually pronounce any of the words we know with any consistency. Sometimes, people nod in understanding. Mostly they don’t. However, the scenario I painted above is how we navigated two weeks in China. The word ‘navigated’ is intentional–it is about the physical and conceptual traversal of options. We navigated space and location. We navigated food. We navigated products. We navigated shopping locations, shopping possibilities and shopping traps (always a concern for tourists, wherever they may be). We did all this navigation speechless; owing to our linguistic ignorance, we accomplished it by pointing. We pointed at menus. We pointed at paper and digital maps. We pointed at applications on our phone screens. We pointed at ourselves. We pointed at desired products. We pointed in space toward unknown distant locations… Basically, we pointed our way to just about all we needed and/or wanted, and we got our way around Beijing with surprisingly few troubles.

Pointing of this kind is a deictic gesture. The Wikipedia definition for ‘deixis’ is the “phenomenon wherein understanding the meaning of certain words and phrases in an utterance requires contextual information. Words are deictic if their semantic meaning is fixed but their denotational meaning varies depending on time and/or place.” [1] In simpler language, if you point and say “this”, what “this” refers to is fixed to the thing at which you are pointing. In the scenario above, it was the location on a map where we wanted to go. Linguists, anthropologists, psychologists and computer scientists have chewed deixis over for decades, examining when the words “this” and “that” are uttered, how they function in effective communication and what happens when misunderstandings occur. In his book Lectures on Deixis, Charles Fillmore describes deixis as “lexical items and grammatical forms which can be interpreted only when the sentences in which they occur are understood as being anchored in some social context, that context defined in such a way as to identify the participants in the communication act, their location in space, and the time during which the communication act is performed”. Stephen Levinsohn in his 1983 book, Pragmatics, states that deixis is “the single most obvious way in which the relationship between language and context is reflected”.

Pointing does not necessitate an index finger. If conversants are savvy to each other’s body movements–that is, their body ‘language’–it is possible to point with a minute flicker of the eyes. A twitch can be an indicator of where to look for those who are tuned in to the signals. Arguably, the better you know someone, the more likely you will pick up on subtle cues because of well-trodden interactional synchrony. But even with unfamiliar others, where there is no shared culture or shared experience, human beings as a species are surprisingly good at seeing what others are orienting toward, even when the gesture is not as obvious as an index finger jabbing the air. Perhaps it is because we are a fundamentally social species with all the nosiness that entails; we love to observe what others are up to, including what they are turning their attention toward. Try it out sometime, stop in the street and just point. See how many people stop and look in the direction at which you are pointing.

Within the field of human-computer interaction–HCI–much of the research on pointing has been done in the context of remote collaboration and telematics. However, pointing has been grabbing my interest of late as a result of a flurry of recent conversations where it has been suggested that we are on the brink of a gestural revolution in HCI. In human-device/application interaction, deictic pointing establishes the identity and/or location of an object within an application domain. Pointing may be used in conjunction with speech input–but not necessarily. Pointing does not necessarily imply touch, although touch-based gestural interaction is increasingly familiar to us as we swipe, shake, slide, pinch and poke our way around our applications. Pointing can be a touch-less, directive gesture, where what is denoted is determined through use of cameras and/or sensors. Most people’s first exposure to this kind of touch-less gesture-based was when Tom Cruise swatted information around by swiping his arms through space in the 2002 film Minority Report. However, while science fiction interfaces often inspire innovations in technology–it is well worth watching presentations by Nathan Shedroff and Chris Noessel and by Mark Coleran on the relationship between science fiction and the design of non-fiction interfaces, devices and systems–there really wasn’t anything innovative in the 2002 Minority Report cinematic rendition of gesture-based interaction, nor in John Underkoffler’s [1] presentation of the non-fiction version of it, g-speak, in a TED Talk in 2010. Long before this TED talk, Richard Bolt created the ”Put that there” system in 1980 (demoed at the CHI conference in 1984). In 1983 Gary Grimes at Bell Laboratories patented the first glove that recognized gestures, the “Digital Data Entry Glove”. Pierre Wellner’s work in the early 1990’s explored desktop based gesture based interaction and Thomas Zimmerman and colleagues used gestures to identify objects in virtual worlds using the VPL DataGlove in the mid 1980’s.

This is not to undermine the importance of Underkoffler’s demonstration; gesture-based interfaces are now more affordable and more robust than these early laboratory prototypes. Indeed, consumers are experiencing the possibilities everyday. Devices like the Nintendo Wii and the Kinect for Xbox 360 system from Microsoft are driving consumer exuberance and enthusiasm for the idea that digital information swatting by arm swinging is around the corner. Anecdotally, an evening stroll around my neighbourhood over a holiday weekend will reveal that a lot of people are spending their evenings jumping around gesticulating and gesturing wildly at large TV screens, trying to beat their friends at flailing.

There is still much research to be done here, however. The technologies, their usability but also the conceptual design space needs exploration. For example, current informal narratives around gesture-based computing regularly suggest that gesture-based interactions are more “natural” than other input methods. But, I wonder, what is “natural”? When I ask people this, I usually I get two answers: better for the body and/or simpler to learn and use. One could call these physical and cognitive ergonomics. Frankly, I am not sure I buy either of these yet for the landscape of current technologies. I still feel constrained and find myself repeating micro actions with current gesture-based interfaces. Flicking the wrist to control the Wii does not feel “natural” to me, neither in terms of my body nor in terms of the simulated activity in which I am engaged. Getting the exact motion on any of these systems feels like cognitive work too. We may indeed have species specific and genetic predispositions to being able to pick up certain movements more easily than others, but that doesn’t make most physical skills “natural” as in “effortless”. Actually, with the exception of lying on my couch gorging on chocolate biscuits, I am not sure anything feels very natural to me. I used to be pretty good at the movements for DDR (Dance Dance Revolution) but I would not claim these are movements in any sense natural, and these skills were hard won with hours of practice. It took hours of stomping in place before stomping felt “natural”. Postures and motions that some of my more nimble friends call “simple” and “natural” require focused concentration for me.  “Natural” also sometimes gets used to imply physical skill transfer from one context of execution to another. Not so. Although there is a metaphoric or inspired-by relationship to the ‘real’ physical work counterparts, with the Wii, I know I can win a marimba dancing competition by sitting on the sofa twitching and I can scuba-dive around reefs while lying on the floor more or less motionless, twitching my wrist.

An occupational therapist friend of mine claims that there will be a serious reduction in repetitive strain injuries if we could just get everyone full-body gesturing rather than sitting tapping on keyboards with our heads staring at screens. It made me smile to think about the transformation cube-land offices would undergo if we redesigned them to allow employees to physically engage with digital data though full-body motion. At the same time, it perturbed me that I may have to do a series of yoga sun salutations to find my files or deftly execute a ‘downward facing dog’ pose to send an email. In any case, watching my friends prance around with their Wiis and Kinects gives me pause and makes me think we are still some way away from anything that is not repetitive strain injury inducing; we are, I fear, far from something of which my friend would truly approve.

From a broader social perspective, even the way we gesture is socially prescribed and sanctioned. It’s not just that you need to have a gesture be performed well enough for others to recognize it. How you gesture or gesticulate is socially grounded; we learn what are appropriate and inappropriate ways to gesture. Often assessments of other cultures’ ways of gesturing and gesticulating are prime material for asserting moral superiority. Much work was done in the first half of the 20th century on gestural and postural characteristics of different cultural groups. This work was inspired in part by Wilhelm Wundt’s premise in Volkerpsychologie that primordial speech was a gesture and that gesticulation was a mirror to the soul. Much earlier than this research, Erasmus’ bestseller De civilitate morum puerilium [8], published in 1530 an admonition that translates as “[Do not] shrug or wrygg thy shoulders as we see in many Italians”. Adam Smith compared the English and the French in terms of the plenitude, form and size of their gesturing. “Foreigners observe that there is no nation in the world that uses so little gesticulation in their conversation as the English. A Frenchman, in telling a story that is of no consequence to him or anyone else sill use a thousand gestures and contortions of his face, whereas a well-bred Englishman will tell you one wherein his life and fortune are concerned without altering a muscle.” [2]

Less loftily, cultural concerns for the specifics of a point were exemplified recently when I went to Disneyland. Disney docents point with two fingers, not just an outstretched index finger but both the index finger and middle finger. When asked why, I was informed that in some cultures pointing with a single index finger is considered rude. Curious, I investigated. Sure enough, a (draft) Wikipedia page on etiquette in North America states clearly “Pointing is to be avoided, unless specifically pointing to an object and not a person”. A quick bit of café-based observation suggests people are unaware of this particular gem of everyday etiquette. Possibly apocryphally, I was also told by a friend the other night when opining on this topic that people that in some Native American cultures it is considered appropriate to point with the nose. And, apparently some cultures prefer lip pointing.

So bother with this pondering on pointing? I am wondering what research lies ahead as this gestural interface revolution takes hold. What are we as designers and developers going to observe and going to create? What are we going to do to get systems learning with us as we point, gesture, gesticulate and communicate? As humans, we know that getting to know someone often involves a subtle mirroring of posture, the development of an inter-personal choreography of motion–I learn how you move and learn to move as you move, in concert with you, creating a subtle feedback loop of motion that signifies connection and intimacy. Will this happen with our technologies? And how will they manage with multiple masters and mistresses of micro-motion, of physical-emotional choreography? More prosaically, as someone out and about in the world, as digital interactions with walls and floors become commonplace, am I going to be bashed by people pointing? Am I going to abashed about their way of pointing? Julie Rice and Stephen Brewster of Glasgow University in Scotland have been doing field and survey work on just this, addressing how social setting affects the acceptability of interactional gestures. Just what would people prefer not to do in public when interacting with the digital devices, and how much difference does it make if they do or don’t know others who are present? Head nodding and nose tapping apparently are more likely to be unacceptable than wrist rotation and foot tapping [3]. And what happens when augmented reality becomes a reality and meets gestural interaction? I may not even be able to see what you are thumbing your nose at–remembering that to thumb one’s nose at someone is the highest order of rudeness and indeed the cause of many deadly fights in Shakespearean plays–and I may assume for the lack of shared referent that it is in fact me not the unseen, digital interlocuter at whom the gesture is directed. And finally, will our digital devices also develop subtle sensibilities about how a gesture is performed beyond simply system calibration? Will they ignore us if we are being culturally rude? Or will they accommodate us, just as the poor taxi driver in China did, forgiving us for being linguistically ignorant, and possibly posturally and gesturally ignorant too? I confess; I don’t know if pointing with one index finger is rude in China or not. I didn’t have the spoken or body language to find out.

NOTE: If you are as bewildered by the array of work on gesture-based interaction that has been published, it is useful to have a framework. Happily, one exists. In her PhD thesis Maria Karam [4] elaborated a taxonomy of gestures in the human computer interaction literature, summarized in a working paper written with m.c. schreafel [5]. Drawing on work by Francis Quek from 2002 and earlier work by Alan Wexelblat in the late 1990’s this taxonomy breaks research into different categories of gesture style: gesticulation, manipulations, semaphores, deictic and language gestures.

[1] John Underkoffler was the designer of Minority Report’s interface. The g-speak tracks hand movements and allows users to manipulate 3D objects in space. See also SixthSense, developed by Pranav Mistry at the MIT Media Lab.
[2] For more on this see A Cultural History of Gesture, Jan Bremmer and Herman Roodenburg, Polity Press, 1991
[3] Rico, J. and Brewster, S.A. Usable Gestures for Mobile Interfaces: Evaluating Social Acceptability. In Proceedings of ACM CHI 2010 (Atlanta, GA, USA), ACM Press.

[4] Karam, M. (2006) PhD Thesis: A framework for research and design of gesture-based human-computer interactions. PhD thesis, University of Southampton.

[5] Karam, M. and schraefel, m. c. (2005) A Taxonomy of Gestures in Human Computer Interactions. Technical Report ECSTR-IAM05-009, Electronics and Computer Science, University of Southampton.

Resources on Experience Design, a selection

Here are some resources/references/publications on Experience Design that were used to develop a Tutorial at CSCW 2011 in China that Elizabeth Goodman, Marco de Sa and I prepared and delivered. Thanks to all Facebook friends who offered suggestions.

CAVEAT: This is NOT an exhaustive list obviously; there are some excellent resources out there. These just happen to be some we used when creating the tutorial. We also drew on some of the content that was taught at a workshop at UX Week 2010.


Industry perspectives on UX design

  • Peter Merholz, Todd Wilkens, Brandon Schauer, and David Verba (2008), Subject To Change: Creating Great Products & Services for an Uncertain World: Adaptive Path on Design. O’Reilly Media.
  • Bill Moggridge (2006) Designing Interactions. MIT Press
  • Mike Kuniavsky (2010) Smart Things: Ubiquitous Computing User Experience Design, Morgan Kaufman

Skills and techniques

  • Kim Goodwin (2009) Designing for the Digital Age: How to Create Human-Centered Products and Services, Wiley Publishing

Sketching and prototyping

  • Bill Buxton (2007) Sketching User Experiences, Elsevier (hovers in between academic and industry perspectives)
  • Dan Roam (2009) The Back of the Napkin: Solving Problems and Selling Ideas with Pictures, Penguin Group USA


  • Roger Martin (2009) The Design of Business: Why Design Thinking is the Next Competitive Advantage, Harvard Business School Publishing.
  • Tim Brown with Barry Katz (2009) Change by Design. How Design Thinking Transforms Organizations and Inspires Innovation, Harper Collins.
  • Alan Cooper (1999) The Inmates are Running the Asylum, Macmillan Publishing Co., Inc.

Psychology and philosophy of experience

  • Don Norman (1998) The Design of Everyday Things. MIT Press
  • M Csikszentmihalyi (2003) Good Business: Leadership. Flow and the Making of Meaning (Viking, New York)
  • Csikszentmihalyi, Mihaly (1998). Finding Flow: The Psychology of Engagement With Everyday Life. Basic Books.
  • Byron Reeves and Clifford Nass (1996) The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places, University of Chicago Press.
  • Peter Wright & John McCarthy (2004) Technology As Experience. MIT press
  • Tor Nørretranders (1998). The User Illusion: Cutting Consciousness Down to Size. Viking.
  • Merleau-Ponty, Maurice. Trans: Colin Smith. Phenomenology of Perception (London: Routledge, 2005)
  • Wolfgang Iser (1980) The Act of Reading: A Theory of Aesthetic Response, John Hopkins University Press

Visual design

  • Stop Stealing Sheep (type usage)
  • UI


  • Scott McCloud (2006) Making Comics, Harper
  • John Hart (2008) The Art of the Storyboard, Second Edition: A filmmaker’s introduction. Elsevier.
  • Wendy Tumminello (2004) Exploring Storyboarding (Design Exploration). Thomson Learning.


  • Steven Douglas Katz (1991) Film directing shot by shot: visualizing from concept to screen, Focal Press (on movie making; good for video and transitional literacy)
  • Scott Kelby (2006) The Digital Photography Book, Peachpit Press

Some other books/resources we have drawn on in the past

  • Richard Saul Wurman (2000) Information Anxiety 2, Pearson Education
  • Works by Edward Tufte
  • Hiroshi Ishii’s work.
  • Neil Gershenfeld (1999) When Things Start to Think, Holt, Henry & Company, Inc.
  • Brain Rules by John Medina (see also
    What the Bleep Do We Know? (and ancillary pieces, such as the movie and the Web site)
  • Richard Sexton (1987) American Style Classic Product Design from Airstream to Zippo

Online resources

We also watched a number of short presentations and interviews online with Experience Design researchers, practitioners and luminaries.

Making Time

[This is a draft of a column I wrote for the ACM’s interactions magazine.  It will appear mid 2011].

One morning, as Gregor Samsa was waking up from anxious dreams, he discovered that in his bed he had been transformed into a monstrous verminous bug.

Thus begins one of my favourite novels, The Metamorphosis by Franz Kafka. What is most remarkable about Gregor’s awakening, and his discovery that he has metamorphised into a dung beetle is that, in the minutes that follow, his greatest concern is that he has missed his train.

Like Gregor, time and schedules have been much on my mind of late. Why? Well, firstly, I overslept the other day. My phone is my alarm clock. Sadly, my phone had died quietly during the night. Ergo–no alarm to awaken me. Although I did not wake up a dung beetle, I was nevertheless disoriented. Secondly, about a week ago, I missed a meeting. Well, strictly speaking, I didn’t miss it because I didn’t know I was supposed to be at it. All I can surmise is that there had been a breakdown in the complicated network of services, applications, devices and people that constitutes the sociotechnical practice of time management called “calendaring”. The meeting was clearly listed on my colleague’s calendar, but not on mine.

So, given my recent horological mishaps, I have been ruminating on the concept of time and it’s management through calendars and alerts.

Calendars reckon past and/or future time. The primary purpose of the calendar is the orientation of our bodies and minds–and those of others–in time and space. In contrast to the fluidity of experienced time, calendars create boundaries between activities. They prescribe the amount of time we should spend on something:  30 minutes with Jane talking about her project, an hour for the meeting on budget, 1 hour giving a lecture on HTML-5, thirty minutes on a mandated management course…..and of course, finally, a day of rest.

To be effective social coordinators, calendars require that we share an idea of how time is structured, how it breaks down quantitatively. My minute and yours should both be 60 seconds; thus we can pass time at the same rate quantitatively–even if, qualitatively, for me the hours have rushed by and for you they have felt like swimming in treacle. And, we should share an idea of exactly when 8pm is if we are going to meet for dinner at 8pm.

Calendars don’t just keep individuals synchronised. Calendars, so scholars like the sociologist Emile Durkheim tell us, are central to societal order. Calendars are the sentinels of ‘appropriate’ behavior. Minutes and days and hours often have activities associated with them–indications of when we should work, rest, pray and/or play. Different social values are placed on different hours of the day and on days of the week; in many calendars Saturdays and Sundays are by default given less space, reflecting social norms that separate workdays from (non-work) weekend days. Routine, calendared time is central to creating a social sense of belonging. In his 2006 article, Tim Edensor argues that structured time in the form of everyday rhythms–which he breaks down into institutionalized schedules, habitual routines, collective synchronicities and serialized time-spaces–are how a sense of national identity and belonging is sustained. One can see this play out in my neighbourhood, wherein many different immigrant cultures reside. What is considered an appropriate time for dinner differs by several hours: between 6pm and 7pm for some, between 9pm and 10pm for others.

I suspect most of us take for granted the idea that we have a shared concept of time. However, the carving up of time into seconds, minutes, hours, days, months and years is a convention, and the familiar structure of the predominant western calendar–the Gregorian calendar, which was only in introduced in 1582–differs from classical calendars like the Mayan, Aztec and Inca, and the more recent Julian calendar[1]. Notably, Russia and Greece only converted to the Gregorian calendar from the Julian calendar in the 20th century. Further, it has not always been the case that someone in Bangalore could so easily work out what exactly time it is for me in San Francisco. It was only in the 1880’s that a uniform time was imposed in Britain; until then, time in Britain varied according to location. This local time stood in contrast to ‘ London time ’ (i.e. Greenwich Mean Time (GMT)); Oxford was five minutes behind London, while Plymouth was twenty minutes behind London[2]. In his book The Culture of Time and Space 1880-1918 Stephen Kern writes of the railroads in the US, “Around 1870 if a traveler from Washington to San Francisco set his watch in every town he passed through, he would set it over 200 times”. The railroads instituted uniform time on November 18, 1883. In 1884 Greenwich was established to be the zero meridian and the 24 time zones one hour apart were established. Countries signed up to this structuring of time one by one: Japan in 1888, Belgium and Holland in 1892, Germany, Austro-Hungary and Italy in 1893. At the International Conference on Time in 1912 the telegraph was proposed to be the method of maintaining accurate time signals and transmitting them around the world; astronomical readings were to be taken and sent to the Eiffel Tower that would relay them to eight stations spaced over the globe. This process was inaugurated on July 1st 1913 at 10am. Global time was born, and the death knell rang for the quaint custom of local time. In an odd way, we can thus trace our globally shared, personal and corporate calendars back to the railroads for instigating the rationalization of time across the globe. It’s quite fitting, therefore, that missing the train is foremost in Gregor’s mind when he wakes up.

However, while synchronised global time connects us, it is all too easy sometimes to forget that there are in fact a number of calendars in operation in parallel today–Chinese, Hebrew and Islamic are just three examples.

As I turn back to my missed meeting, I note that calendars have ecclesiastical origins; the Book of Hours structured time into routines for work and worship for monks in the Benedictine order. However, in sharp contrast to the quiet, stable regularity of the liturgical life, my calendar is a chaotic beast in constant need of maintenance and management. Meetings pop on and off like jumping beans as the hoping-to-be-assembled try to find a time that works for all concerned. Vigilence is required lest one is triply booked, and priorities are always being calculated: Is this meeting more important than that one but if so and so is there then that is a good opportunity to get things moving forward…… Oh no, now they are not going to be there after all and yet I am committed to going, how do I shift this around…… and on and on.

The root of the problem lies in the multiples–multiple calendars and multiple people on one calendar. For the first point, I have too many calendars and the effective synchronization of my calendars is not a solved problem. Ghost (long departed/deleted) meetings haunt the calendar on my computer, while my mobile phone presents a suspiciously clean blank slate. Sometimes there is little correspondence between the two, despite their notionally being jacked in to the same server. For the second point, shared calendars (such a good idea in principle) are a gargantuan, social rogue elephant. Herein lie clashes in culture, herein lie power relationships and herein lie a network of complex dependencies. Routine issues arise for me in the following forms: blank space on the calendar, the curse of durational rigidity, the clash between sociotemporal and biotemporal time, and the problem of travel time. Lets briefly review each of these…..

Idle’ time People routinely look at my calendar to determine when I am free to meet; they plop meetings on my calendar based on what they see as ‘free’ time. This is based on a fallacious assumption–that if there is nothing recorded there, then I am free. This is a misreading of my practice of calendar use. Booked times on my calendar are not simply islands of colour in a collaborative paint-by-numbers schematic where the blanks are inviting others to fill them in–I saw a gap so I filled it.

Of course, idle time is anathema to the shared calendar in a culture where to be not actively doing could possibly be interpreted as shirking. In my view, day of back-to-back meetings means there is too little time for creative thought or for reflection. Research indicates that time when one is doing the least, as for example when meditating, is when the most creative moments can occur[3]. The jammed calendar, continual context-switching and mad dashes from one location to another are emotionally draining, mania inducing and counter to creativity.

So I sometimes put “meetings” onto my calendar to simply block some thinking time. I feel sheepish about this. I am reminded of a friend of mine, who, when we were teenagers, used to write things like “peas and carrots for tea” in her journal. Recording peas and carrots was not because of some dietary obsession, they stood in as code for ‘held hands’ and ‘kissed’, reporting on her teenage encounters with her boyfriend; the code was invented lest her mother should read her journal and be mortified by her teenage explorations. So, it is that I transform thinking, writing and reading into ‘Strategy’ and ‘Planning’, appropriate behaviours for a corporate context. Durkheim and followers are correct: how one manages one’s time is an issue of morality and social accountability, not just temporal coordination. It’s a tricky business.

Durational rigidity For the operationally minded, a meeting that is scheduled for an hour must last an hour even when nothing is being achieved. On the other side of that, sometimes one can be just warming up, just getting to the crux of a problem and the hour is up, the meeting has to end truncating the creative process.

Travel time Another problem, and one where a simple technical solution would help out, is travel time between locations. When one works in several different office buildings that are miles apart, it takes time to get from one to the other. It would be useful if I could hook my calendar up to these locations, and have travel time calculated and reflected automatically. So if a meeting is dropped onto my calendar, travel time is automatically blocked in–in fact, I could imagine a lot of background calculating that can be done by hooking my calendar up to location and to my social services and applications[4].

Biotemporal time Working across time zones can be really hard. The cheerful calendar flattens time, it sees all times as equal. Calendars are simply tabulated time in a grid, they do not reflect lived time. Odd times for calls can sneak in there, creating social and personal dilemmas–I want to be a good citizen but I know I am going to be less than my best at that time. Sociotemporal times (as in when it is appropriate to be working and when not) clashes here with biotemporal time. Being on a meeting conference call when your body and your entire environment tells you that you should be sleeping is simply hard. Time may be global but my body is not.

None of my observations are earth-shatteringly novel. There has been a wealth of research in the HCI community from the early 1980’s and continuing now today, on life scheduling and calendaring–in collocated and in distributed workgoups, in the home, in leisure groups, within families, between families, on paper, across paper and other devices, on personal computers, using mobiles, using location services and with visual front end experiences including3D representations. Just to name a few of the research directions. There are typologies of calender user type such as that offered by Carmen Neustaedter and colleagues who call out three different types of families—assigning them to the categories monocentric, pericentric, and polycentric according to the level of family involvement in the calendaring process. Monocentric families are those where the routine is centered on a primary scheduler, pericentric families have the calendar routine centered on the primary scheduler with infrequent involvement by secondary schedulers and polycentric families are those where the calendar routine is still centered on the primary schedulers, yet secondary schedulers are now frequently involved. BUT despite all this work, there’s still plenty we can do in the world of sociotechnical design to rethink the calendar. My calendar does not feel “_centric” in any way; it feels chaotic.

“We shape our dwellings and afterward our dwellings shape us” said Winston Churchill in 1943. We could apply this observation to time; we shaped the calendar and now the calendar shapes us, it dictates how we (should) live. True to Louis Sullivan’s adage form follows function, the digital calendar wears its assumptions and its intellectual heritage on its sleeve: computer science, psychology, information architecture and the ethical structure of the approved-of day. Perhaps we need a new tack.

In Branko Lukic’s and Barry Katz’s 2011 text, Nonobject, they explore product designs that sit at the interstices of philosophy and technology. They step back from simplistic notions of form and function to shake up how we think about products, to question what is ‘normal’ or taken for granted, and to question the values that are embedded within the typical form of everyday artifacts. In a section entitled Overclocked, they explore clocks and watches, our time-keepers. Katz writes, “as our measuring devices grow ever more accurate, we find ourselves perpetually “overclocked” to use a term familiar to every computer hacker who has ratcheted up a component to run at a higher clock speed than it was intended for in order to coax higher performance out of a system. We do the same to ourselves.” A number of designs are presented: the Tick-Tock Inner Clock that taps against the skin to let someone feel the passage of time and the Clock Book where time is laid out on pages we can turn–when we want to–push. Lukic’s watches and clocks invite us to rethink we conceptualize, represent and manage time. Somewhat less extreme but nevertheless taking a playful take on clock design, Alice Wang’s 2009 suggestion for the Tyrant alarm clock is brilliant. This alarm clock calls people from your address book on your mobile phone every three minutes if you don’t get up and turn it off; with this, Wang is betting that the anxiety of broadcasting your slothful habits to anyone in your address book will propel you to get up. Wang gleefully reports that it is the social guilt that will get people moving out of bed. Social anxiety has long been a driver for action; this is I think a nice example of it, and this is a step beyond thinking instrumentally about the clock’s utility/function in isolation from the rest of one’s life.

Let’s do the same thing with calendars. Let’s take a step back. Let’s follow Lukic and take our lead from Architectura Da Carta, the Italian tradition of articulating and illustrating the unlikely, the unbuilt and the unbuildable. Let’s use art, philosophy and technological creativity to envision a better aesthetic experience, to blast the calendar apart and rebuild it; let’s be better about enabling the plularity of private and public times that humans live in parallel; let’s automate the calculation of time in motion between location(s); let’s build in time for creativity and reflection as social and moral imperative; let’s make a calendar that adapts the schedule when it realizes you have woken up having metamorphised into a sentient dung beetle.

[1] See Anthony Aveni Empires of Time. Calendars, Clocks and Cultures, New York: Basic Books, 1953

[2] See Journal of Design History Vol. 22 No. 2 Designing Time: The Design and Use of Nineteenth-Century Transport Timetables by Mike Esbester

[3] See for example The neuropsychological connection between creativity and meditation published in ‘Creativity Research Journal’, 2009 by Roy Horan

[4] See Lovett and colleagues on this in their Ubicomp 2010 paper: The Calendar as a Sensor: Analysis and Improvement Using Data Fusion with Social Networks and Location

Rat, Rational or Seething Cauldron of Desire: Designing the Shopper

[This is a draft of a column I wrote for the ACM’s interactions magazine. Publication details: interactions, Volume 18 Issue 1, January + February 2011, ACM New York, NY, USA]

Since the 1930s, governments have largely followed the view of John Maynard Keynes that spending, not saving, drives successful economies. We have taken this charge seriously in the U.S. If Adam Smith characterized Britain as a nation of shopkeepers, the U.S. is indubitably a nation of shoppers. No other country in the world spends as much on consumer goods as America.

Whether you love it or hate it, everyone has an opinion on shopping. It’s an expression of identity. It’s the gift I give my family. It’s a chore I have to endure. It’s sport. It’s a sign of the demise of our culture, an assault on our base ethics and morality; “Stop the Shopocalypse!” entreats the performance artist Reverend Billy and his troupe, the Church of Life after Shopping.

As the memory of Santa’s season of goodwill, joy, and multifarious forms of consumption fades and I review my credit card bill, I have been reflecting on how we shop—more specifically, on how the shopping experience is designed for us, and we are in some sense designed by it. For this reverie, I am more interested in what has been dubbed “wants based” rather than “needs based” shopping, which, as evidenced by my credit card statement, clearly makes up most of my purchases.

So, yesterday I made a diary of a shopping day, noting the things and experiences that I usually take for granted, but with a view to wondering how they came to be so. With the caveat that this is not an exhaustive diary, here’s the timeline:

6:30 a.m.: Wake up.
6:45 a.m.: Pick up mail, physical. Two catalogs, an envelope of discount coupons, and an advertisement for expensive holidays masquerading as a travel magazine.
7:00 a.m.: Scan the list of emails from services, vendors, shopping-discount sites, and so on. Canon seems to think that I could make use of yet another camera. Apple thinks I will be enchanted with some new products from them.
7:30 a.m.: Shower, using a multitude of free samples sent to me by online-shopping vendors, most of which products I feel disinclined to purchase.
8:30 a.m.: Drive to work, listen to national public radio (NPR), and hear a review of a book I will likely purchase.
11:25 a.m.: Alleviate mild distraction and boredom by reviewing what is currently available on shopping sites RueLaLa and ideeli, sites that sell designer clothes and accessories. Find names of three designers who are new to me but I will look out for in future. (Thank goodness for online shopping, which means the indignity of trying on clothes in public fitting rooms is circumvented and provides idle-time shopping in 15-minute chunks when I am bored.)
5:45 p.m.: Drive to shopping mall and locate specific department store to buy a gift for a friend, navigate lights, music, perfume, and other sensory assaults.
7:00 p.m.: Drive home from work via supermarket, pick up groceries.
7:50 p.m.: Drag loads of heavy grocery bags upstairs; curse the shopping cart that masked the weight of the load I was going to have to carry later.
9:45 p.m.: Check out the Canon offers and surf the Web, reading reviews of various computer products; cave in and buy coat from RueLaLa; download infeasibly peppy music from iTunes based purely on beats per minute, and select a book for a friend from Amazon. Watch bizarre and dubious “shopping haul” videos—teenage girls sharing shopping expertise on Youtube. Am transfixed by these videos with ghoulish fascination, and note from view counts that tens of thousands of people have watched them.

Following yesterday’s diary, today I surfed the Internet, researching how some of these experiences came to be. As you will see, aside from the Internet as a channel for learning, researching, and purchasing, many of the things I encountered yesterday and have, up until now, taken for granted, carefully were designed or created with a view of what the practice of shopping does, could and indeed should look like.

The department store as we know it derived from the entrepreneurial efforts of several key figures: the Bloomingdale brothers, John and Lyman, bankrolled by profits made from sales of the wildly successful hoop skirt; Brigham Young, who in 1868 founded Zion’s Cooperative Mercantile Institution (ZMCI) in Salt Lake City, Utah; Marshall Field in Chicago; Alexander T. Steward in New York; and last but certainly not least, John Wanamaker, who opened the Grand Depot, a round, six-story department store in Philadelphia in 1877. The key business driver for the department store was simple: Buying wholesale meant retail costs could be kept relatively low. The key experience driver was educating people as to hitherto unknown possibilities for parting with their cash. The department store’s development from convenient goods provision to center of cultural production is wonderfully described in Jan Whitaker’s book Service and Style: How the American Department Store Fashioned the Middle Class.

On a larger scale, the first shopping mall, the Country Club Plaza, was founded by the J.C. Nichols Company and opened near Kansas City, Missouri, in 1922. While the Country Club Plaza was open to the sky, the first enclosed shopping mall was opened in Edina, Minnesota (near Minneapolis), in 1956. The first shopping mall, a collection of stores, was designed to make shopping more convenient by placing lots of shops in one location. The West Edmonton Mall opened in Alberta in 1981, housing 800 stores, a hotel, an amusement park, a miniature-golf course, a church, a “water park” for sunbathing and surfing, a zoo, and a 438-foot-long lake. Park the family while you shop. Or better still, shop-play-shop.

While the shopping mall as we know it is so familiar, much of what we take for granted now was invented by Victor Gruen, an architect from Vienna. Gruen and associates designed the Northland Shopping Center in Detroit, in 1954, the Southdale Shopping Center near Minneapolis in 1956, and the Midtown Plaza in Rochester, N.Y., which opened in 1962. The innovations they designed revolutionised shopping. Innovations developed over the years: 1939, Gruen was invited to design a leather-goods store for a friend in New York. Gruen’s inspiration was to create an entrance to the store that was intriguing and enticing to the eye. Colours, lighting and reflections were used in combination with fake marble, and green corrugated glass on the ceiling to create a “customer trap” designed to lure customers into the store regardless of their initial purchase intent or their interest in the products on display (so much for the shopping list). Gruen went on to create the multi-level shopping facility with escalators between the levels and multiple entry ways from multi-level carparks. Gruen was also the genius who initiated blank wall exterior for stores; prior to that shows focused on window displays.

The shopping cart was invented by Sylvan Goldman, owner of a chain of Oklahoma City grocery stores called Standard/Piggly-Wiggly, in 1936. The original shopping cart was like all prototypes, a humble contraption. It involved two wire baskets, a folding chair and some wheels. Anyone who has wheeled themselves to the checkout, waited the allotted time, and then experienced a wave of shock at the bill followed by a panicked glance to assess what could possibly have ended up in the cart, knows how much easier it is to purchase more than you intended with a cart. The best “forcing function” to reduced purchasing is carrying the stuff you intend to buy; as your arms get wrenched from their sockets, you are more sanguine about that extra, just-in-case bottle of lotion or packet of batteries. And thank goodness for Dee Horton and Lew Hewitt’s 1954 invention, the sliding automatic door–turning handles and pushing doors while holding laden shopping bags is never a graceful exercise.

Store layout. It should be noted that stores themselves are scripted spaces, laid out to maximize our likelihood of purchasing certain items. The most famous researcher on the topic of store layout and its impact on purchasing behavior is Paco Underhill. An environmental psychologist and author of Why We Buy: The Science of Shopping and Call of the Mall: The Geography of Shopping, Underhill sets forth fairly straightforward ideas—how you structure the environment leads to different purchasing behavior, and our in-store experience is as much a part of why we purchase something as the carefully constructed shopping list.

Coupons. Introduced in 1895 as the brainchild of Asa Candler, coupons were placed in the newspaper for a free Coke from any fountain as a way of promoting the soft drink. I lost count of how many physical and digital coupons I encountered yesterday. But they are clearly part of my life, even though I use perhaps one in 100.

The mail-order catalog was created by Aaron Montgomery Ward in 1872. As I look at the two I received in my paper recycling pile and review the vendor sites I visited when surfing the Internet, I am comforted to know that many generations before me also preferred shopping from the couch to running around a mall.

Price tags, sales, in-store restaurants, money-back guarantees, and newspaper ads. All of these were created by John Wanamaker, who I think has to get the most points as creator of the contemporary shopping experience. It never occurred to me that someone actually had to invent the price tag!

As I researched these different inventions and designed experiences, I realized that I was also participating in a designed abstraction, playing my part with gusto. I have been trundling down the “purchase funnel,” a concept developed in 1898 by St. Elmo Lewis, who mapped a theoretical customer journey from the moment a brand or product attracts your attention to the moment you actually buy it. To advertisers, marketers, and merchandisers, AIDA is not an opera by Giuseppe Verdi; rather, it reflects points along the purchase funnel: attention, interest, desire, and action. Oddly, Gruen’s “customer trap”, physically luring people into stores, almost seems to have been captured in the funnel metaphor. Draw people in, and help them move from unexpected awareness to desire to intent and actual purchase.

I learned about a host of new products from the catalogs, the online surfing, the coupons, the radio, and the products on display. I find myself tenderized by the marketers and advertisers and am somewhere along the “purchase funnel” for a number of upcoming purchases. I also realize that designers of all sorts (advertisers, print layout, store layout, online stores, architects, entrepreneurs, engineers) have other abstractions that are being played out in their designs: an abstraction of me as a human, as a shopper and what motivates me to shop in the first place, what gets me peering into the funnel.

In researching this column, I also discovered that, as a shopper, that I am variously cast as a rat, as a rational person, as a seething cauldron of desire, or as a primeval Homo sapien, hell-bent on foraging for my kin. The model of me as rat derives from behaviorist psychology, emphasizing that I have been “conditioned” through reward (nice treats, Jimmy Choo shoes) and punishment (nasty things like hunger, cold, and no Jimmy Choo shoes) to respond to certain cues that drive us to shop. The second character, the rational shopper in me, is Homo economicus—rationally making choices on the basis of utility with some cost. This model is based on Lionel Robbins’s rational choice theory. While I recognize some of the behaviors, in general Homo economicus does not hang out with me. My third potential shopping mantle is a seething, bubbling cauldron of desire based on psychoanalytic theory. My ego, my superego and my id–the three-ring circus of conflicting urges that underlie whatever identity issues I may have–all held in molten, psychic, tension, stabilized momentarily by the acquisition of stuff. That is, I buy things so I can express my true inner self, whoever that might be today. Finally, the hunter-gatherer model emphasizes that we shop because it is innately human to shop—it is wired into our primal brain to rush out and hunt for things. Of all the models I read today, I confess the last leaves me most cold; it’s about as interesting as saying that shopping involves human beings.

Reflecting on my own experience of shopping, however, these theories have (at least) two problems. First, they are overly focussed a lot on individual shopping, when shopping is clearly social. Artist Barbara Krueger coined the phrase “I shop therefore I am”; I say, “We shop therefore we are.” Shopping is something we learn to do; we learn culturally how to shop and what to shop for. Pierre Bourdieu, the French philosopher, talks about “taste communities”; you are not just what you yourself consume but also what your friends consume. You Blahnik, therefore I Blahnik. The existence of the shopping-haul videos would not have shocked Bourdieu. Those young women are teaching others what to shop for, why the products matter, where to acquire them and what to do with them once they are acquired. Second, what constitutes a rational decision depends on how you scope the reasoning process. What looks entirely reasonable in the moment (tired, hungry, desperate for something for Uncle John so I don’t look like a cheapskate at Christmas), in retrospect looks daft (when he opens his package to find his 23rd pair of socks). The bounds of rational reasoning shift. A product or service I would not have looked at yesterday somehow becomes absolutely essential today. Grant McCracken, the anthropologist and commentator on contemporary culture, talks about the “Diderot effect,” invoking the philosopher Diderot, whose apartment remodel was kicked off by the receipt of a luxurious dressing gown, in the shadow of which the rest of his apartment looked shabby.

If we consider the trajectory of department store to mall to Internet, our choices of what to buy and how to “upgrade” our lives are increasing at a ridiculous pace. I wonder, what’s going to be the next revolution in the shopping experience? I also note it took me significantly less time and energy to part with my cash during the last Holiday season than it would have 20 years ago. This reflects the fact that, in tandem with the choice explosion, the time and energy taken to potentially acquire things is decreasing markedly; easy-to-use, one-click and delivery-to-door are in cahoots, encouraging the impulse buy, enticing the rat and irrational id, evolving the sloth model that has not yet been articulated fully, and muscling Homo economicus out the door. So, next time you’re shopping, online or off, think about how that experience is designed and in what ways it appeals to you the shopper–as rat, rational, id, hunter … or sloth?

Enticing engagement

[This is a draft of a column I wrote for the ACM’s interactions magazine. It appeared in Volume 17 Issue 3, May + June 2010].

Human engagement
A: Do you love me?
B: Yes
A: Will you marry me?
B: Yes.

Internet engagement
A: Do you love us?
B: click click click

A: Do you love us now?
B: click click click

Internet disengagement
A: Do you love us?
B: click click click
A: Do you love us now?

A: What about now?
A: Hello? Where did you go?

For me, long gone are the days when the word “engagement” conjured up diamonds, parties, and champagne. These days engagement is all about divining how much love your users or the audience have for product(s) and/or application(s). In the Internet world, the word “engagement” is intimately associated with measurement, metrics, and monetization; it is all about clicks and conversion, visitors, page views and duration. This is the world of media marketing, and of Internet success and failure.

What is Engagement?
Before I dig into Internet engagement measurement, I want to step back and think about what engagement means to me. When first asked about engagement by a colleague at work, I spouted all I knew about the experience of engagement as I understood it from a psychologist’s worldview. Flow. Immersion. Fascination. The swift passing of time as attention is held—tempus fugit. Engagement is the absolute opposite of restless boredom; boredom is the experience of time expanding painfully, a feeling of being itchy and twitchy and ready for any external stimulus beyond what you are doing now.

From a psychologist’s point of view, engagement sits within a framework of arousal states—boredom at one part of the spectrum, engagement at the other. Altertness, forced focus and stress all sit somewhere in between. Perhaps the most famous approach to thinking about engagement is the concept of flow, proposed by Mihály Csíkszentmihályi. Flow is the state of full immersion in what one is doing: a feeling of energized focus, total involvement, and success in the process of the activity. Flow is a positive feeling, reflecting an utterly focused concentration and alignment with what one is doing. In some conceptions, flow is associated with feelings of joy and rapture, an exclusion of all but the central activity. Engagement may not be this rapturous positive feeling; people can be negatively engaged if something is standing between them and their desired state or goal, and for many people, military engagement denotes a negative form of entanglement.

But central to the idea of engagement is the triggering and capturing of attention, of being beguiled and perhaps focused to the exclusion of everything else. Further, lasting engagement can manifest in episodes of focus; perhaps someone will come back for more, again and again and again. This is where engagement in my early training intersects with the world of marketing and media engagement, the focus to which I now turn.

Creating Engagement
“Engagement” is a favorite word for media marketers and for the designers of products, services, applications. Take a look: The Web is brimming with smart people opining on the topic. In this world, marketers, data miners, statisticians, designers, and a host of other professionals come together. “Engagement is the most essential measure of the success of any medium. Engagement speaks to passion, preference, and habit,” said Charles Buchwalter, vice president of analytics at Nielsen NetRatings, in a 2005 white paper, “Integrated Interactive Marketing: Quantifying the Evolution of Online Engagement.”

In this world, the point is to stimulate engagement, to create it and make it last for whatever period is appropriate for your offering. In most cases, the goal of creating or “driving” engagement is to provide a platform for advertising products or services.

Models of how to engage consumers make assumptions about the level of active involvement of which the consumer is likely to be capable or willing to offer. At one end consumers are seen as being passively engaged with the content; think of the stereotypical couch potato glued to the screen while shows and advertising flicker across the TV screen. At the other end of the spectrum consumers are seen as actively seeking out content, developing their own understandings of that content, and then perhaps sharing with others. The active consumer concept is at the center of viral or word-of-mouth advertising and product engagement. Another example on the active consumer end of the spectrum is engagement marketing (which is also variously called experiential marketing, event marketing, live marketing, or participation marketing).

I like the idea that persuasion architects, who are responsible for creating campaigns to get people actively involved in advertising and engagement, are not completely in control of how the recipient perceives the message or whether or not they are likely to be engaged. I like to bestow upon the reader some agency in their own engagement. Borrowing from Tony Schwarz, a ’60s political advertising guru, who coined the idea of successful campaigns as those that invite people to “participate in their own manipulation,” I call this new world of Internet and cross-channel media play a form of “partipulation.” Perhaps we need “vongagement” to stand in for voluntary engagement? We need a word implies, entails and insists upon reciprocation, a word that acknowledges that the engagement is essentially about a relationship between a person and an activity or an entity.

Measuring Engagement
Measurement and evaluation are key in good design research; the same is true of good audience studies—it is a good thing to be able to evaluate how successful you are at engaging your audience.

In the media world, there have been many scales and measures of engagement, most of them developed for TV viewing. Examples include Q Scores from Marketing Evaluations that measured “likeability,” launched in 1963, and Jack Myers’s Emotional Connections model of viewer engagement, launched in 1999. Since then scales have measured relevance, affinity, comfort and resonance, impression, opinions, recall, awareness of brands and products, memory for program details, emotional involvement, and, of course, moment-by-moment measurements of physiological and neurological effects. Most of these studies are with representative panels of people sampled from the population of viewers as a whole.

In the world of the Internet, measures of engagement focus on features, applications, sites, services, and brands. Here, the Holy Grail is generating engagement measures based on the activity of the whole audience, not a sample; in theory, as all actions take place online, the entire system can be instrumented for activity and therefore measured. The metrics and measures even here, however, are constantly in negotiation and transition. Leaders in the discussions surrounding measurement for engagement are data services like comScore and Neilsen, who work with massive data sets as well as with panels of participants to generate measures and metrics.

A central measure of engagement used to be page views—how many times a webpage was visited. In 2007 the “visits” metric—defined as the number of times a unique person accesses content within a Web entity with breaks between access of at least 30 minutes—was introduced as a way of measuring the frequency with which a person views content, thereby illustrating a key component of user engagement. To determine engagement one looks at total visits, average minutes per visit, average visits per visitor, and average visits per usage day. One of the reasons for changing measures is technological. “With advances like AJAX changing the Internet landscape, certain measures of engagement, such as page views, are diminishing in significance for many Web properties,” said Jack Flanagan, executive Vice President of comScore Media Metrix. “The introduction of these new metrics based on ‘visits’ provides an alternative for measuring user engagement that tells us how frequently visitors are actually returning to the site to view more content.”

Things are also changing as sites become more focused on the social Internet. A more extensive list, based on my review of about six sites dedicated to Web analytics between 2007 and 2009, is this one: There are “root” metrics that appear in all discussions of engagement and Internet use. Some of these are relevant for all Internet sites, and some are relevant only for some sites, e.g., sales apply only to sites that are actively selling something or sites that contain advertising that is intended to lead to a purchase. These “root” metrics are:

• number of unique users
• number of sessions a unique user had with the site/page/application
• duration of visit
• frequency of visit (returning to the site directly—through a URL or bookmark—or indirectly)
• percentage of visits that can be determined to be repeat visits for an identifiable single goal
• recentness of visit
• depth of visit (percent of site visited)
• click-through rate
• “bounce measure” (determined by number of visits that were less than a set number of seconds, which implies the person arrived at the site by mistake or arrived, found nothing they wanted, and left)
• subscriptions, and conversion from free to subscriber service
• sales of product(s) through site and/or referrals that resulted in partner sales
• lifetime value – the last being probably the hardest to evaluate

There are also action metrics, and social-engagement metrics based on elements that are key to social interaction design:

• content contribution (e.g., text, uploads – images, video)
• RSS feed subscriptions
• bookmarks, tags, ratings
• viewing of high-value or medium-value content (as valued from the organization’s point of view—note that depth of visit can be combined with this variable)
• inquiries
• providing personal information, such as filling out profiles
• downloads, content resyndication, and customer reviews
• comments (with some modulation depending on their quality, and some analysis of the ratio between posts and comments plus trackbacks)

Recent applications like Tynt’s Tracer tracks people’s copy-and-paste actions to see what you liked enough to copy and paste it, and many services track site recommendations.

Problems with Internet Engagement Measurement
The artful combination of measures with a clear view of what one is trying to achieve is key. Avinash Kaushik, an analytics practitioner and author of Web Analytics 2.0, calls for good design practice around Web analytics on the issue of engagement. His steps are:

1. Define why your website/application exists, asking, “If there were one thing your website would do, what would that one thing be?”
2. Determine the critical metrics (three or fewer) that will identify exactly how you can measure if your website is successful at delivering against its purpose.
3. Decide what engagement means to you in your context.
4. Don’t call that metric “engagement.”

He urges us to call a spade a spade, saying, “Think very carefully about what you are measuring if you do measure engagement. If engagement to you is repeat visitors by visitors then call it Visit Frequency, don’t call it engagement.” It is crucial to determine ahead of time what your goals are and what constitutes success. This is akin to working on a product and being clear as to what the primary goal of the product is. So many great ideas and great products are hampered by a version of “creeping featurism” that could be called “excitable expansionism.” Whenever I hear someone say “Yes, and we could do this as well!”, I cringe and try to drive the conversation back to the core issue we believe we are addressing—the core goal, need, or hedonistic channel we believe the product should exist to serve.

Engagement cannot be a one-size-fits-all calculation. It varies according to industry, organization, business goals, and so on. Along with this tailoring, one needs to be clear that the relative weighting you give any action depends on your goal. Your metrics and measurement research program depend on whether you are assessing a feature, an application, a site, or a business/brand; whether you are assessing the value of the product in itself for a single user or as a broker between social entities. There are (single) user-engagement measures, but these may not be the same as those that are appropriate for social media. A useful single-user metric might be clicks and content (i.e., how many features are clicked on, how much information and content is uploaded), but a useful measure for social applications may be reciprocity between the people using the application (i.e., how many links do you send me in response to what I send you—how conversational is the application, and how much sharing does it encourage?)

A Broader Yet Still Measurable Notion of Engagement
Before closing, I would like to broaden the idea of engagement still further. Taking my cue from media studies, I am inspired by the “consumer expressions” model and the “connectedness scale” model, both models applied to media like TV and film, but nevertheless fully within the broader and braver new world of partipulation. Or was it vongagement/reciprocative assessment?

The consumer expressions model, devised by researchers from MIT in 2002, suggested the development of a metric that combines qualitative information (the nature of the content, the media consumption environment) with quantitative information (time spent, satisfaction). In 1999 the connectedness model proposed by Cristel Russell and Christopher Puto focused on use of media including self-definition, creative contribution, social interaction around the show, ritualization (organizing and maintaining specific social functions of viewing and using), and the purchase of show-related paraphernalia. This was broadened in 2004, when researchers Russell, Norman, and Heckler conducted a validation of the connectedness scale that examined the extended contribution around a series of constructs—attitude (degree of favor or disfavor), involvement (mental state during viewing), and overall viewing (time spent overall in the medium). Their model, albeit particularly focused on TV and media events, offers six overarching dimensions, some of which perhaps could be extended to the world of digital and physical products and especially to media products:

• escape (immersion into the event)
• fashion (the extent to which the viewer is influenced by the characters’ appearance)
• imitation (the inclination to imitate the characters’ behavior or speech patterns)
• modeling (the degree to which individuals relate their lives to the lives of characters
• aspiration (aspiring to actually be on the show or meet the characters)
• paraphernalia (the degrees to which people collect items to bring the show into their real world)

As connectedness increases, so does frequency of show-related social interaction with others, the development of relationships within the community of co-viewers, and the size of the viewers’ social network of co-consumers. People also have a greater memory for the characters and storylines, and surely this kind of memory is key to long-term engagement as well as in-the-moment engagement. So a deeper sense of overall engagement, and a measure that moves beyond immediate activities into activities beyond the screen, mouse, click and brand awareness into behavior and identity. Indeed, consumption, participation, and identification are key. When I think of a social Internet site/service like Flickr, I immediately see how this level of analysis is useful as a complement beyond assessing clicks and content. The impact of a site or service like Flickr beyond the screen and beyond the session is what makes Flickr a place to see and be seen, not just a place to visit now and then. Avid Flickr users identify with Flickr; they don’t just see it as a place to view images.

My main point is that more integration between different measurement forms is needed to get a clear picture of overall engagement. This is systems thinking; it necessitates measurement perspectives from clicks to brand to identity; it requires a mixed-methods approach and the insight to see how a survey can relate to a field study can relate to a programmatic series of principled data collections … and so on.

In sum, there is so much more to engagement than immediate actions captured by a simplistic action-analytics model or reflected feelings on a survey. Engagement is more than immediate instrumental value and, as a topic, engagement is in need of a broader perspective. Apparently, engagement is itself quite engaging.

Socializing at Cross Purposes

[This is a draft of a column I wrote for the ACM’s interactions magazine. It can out in Volume 17 Issue 1, January + February 2010. The full, final, text with glorious imagery can be found here.]


Indulge me for a moment. I have a series of jokes I want to tell you:

  • How many social scientists does it take to screw in a lightbulb? None. They do not change lightbulbs; they search for the root cause of why the last one went out.
  • How many simulationists does it take to replace a lightbulb? There’s no finite number. Each one builds a fully validated model, but the light never actually goes on.
  • How many statisticians does it take to screw in a lightbulb? We really don’t know yet. Our entire sample was skewed to the left.

So what’s with the (not particularly funny) jokes? The point is that they play off particular ways of thinking. In doing so, they show us how different the world can appear, depending on the perspective.

This was evident in a recent meeting. Reminiscent of another set of common (and usually also not funny) jokes involving different nationalities walking into a bar, there were six people in a room: an interaction designer, a statistician with an interest in behavioral modeling, a social scientist, a computer scientist, an self-described “back end with a touch of front end” engineer, and a business executive. We were brainstorming about the potential for accessing social Web applications from personal mobile devices.

Two minutes into our conversation, I said, “We should start with some sound social principles.” This was my bland opening gambit, a preface. Or so I thought. Once I paused for a fraction of a second, everyone started talking at once—like whippets after the faux rabbit at a dog race, the conversation was off. Then it stopped, followed by blank looks. To mix my metaphors horribly: The conversation plumed, spiraled, and evaporated like the contrails of the Blue Angels on July 4th.

The problem was the word “social.”

A quick perusal of the dictionary yielded these definitions of social: relating to human society and its members, living together or enjoying life in communities or organized groups, tending to move or live together in groups or colonies of the same kind, and living or liking to live with others, disposed to friendly intercourse. Etymologically, the word derives from the Latin socialis, meaning “united,” “living with others,” and sequi, meaning “follower,” which should make contemporary social Web application designers happy.

The famous 17th-century philosopher John Locke spoke of “social” as meaning “pertaining to society as a natural condition of human life.” And as an adjective, “social” appears as: “social climber” (starting in 1926); “social work” (1890); “social worker” (1904); “social drink(ing)” (1976); “social studies” as an inclusive term for history, geography, economics (1938); and a concept close to our hearts in these hard times, “social security” as a “system of state support for needy citizens” (1908). That is the backdrop to the conversation I thought I was starting. However…

To the interaction designer, “social” invoked “social Web applications” and all that it means for human interaction with voting (thumbs up and down), favoriting (stars), contact lists and buddy lists, followers, avatars and profiles, chat threading, commenting, recommendations, and view counts. It meant a discussion of icons that suggested (or were derivative of) those on successful social media sites and multimedia content upload and sharing. Talk poured forth about social games and questionnaires, pokes and winks and friending. Let me be clear about my position: I love thinking about these issues, and have recently reviewed drafts for two excellent books in this area—Building Social Web Applications by Gavin Bell and Designing Social Interfaces by Christian Crumlish and Erin Malone. But for the purposes of this meeting, tossing out all of these concepts was great, but it was also putting the cart before the horse. We’d get there, but not yet.

To the computer scientist, “social” sparked sweet, seductive imaginings of the social graph. Wikipedia defines a social graph by explaining that “a social network is a social structure made of individuals (or organizations) called ‘nodes,’ which are tied (connected) by one or more specific types of interdependency, such as friendship, kinship, financial exchange, dislike, sexual relationships, or relationships of beliefs, knowledge or prestige.” The entry continues: “Social network analysis views social relationships in terms of network theory about nodes and ties. Nodes are the individual actors within the networks, and ties are the relationships between the actors. The resulting graph-based structures are often very complex.” No kidding. I love the complexity and curiosity of my species—human beings—and these ways of conceiving social relationships often strike me as dismayingly reductive. They’re very useful within their bounds, but they are summary abstractions of the lyrical complexity of everyday social life. We had a very fruitful foray at this meeting into social recommendations and boundaries—the complexity of “friend relations” and “access control privileges”; the connections between objects via hash tables; and connections between people, their stuff, and other people’s stuff. We discussed these things as inadequate approximations for supporting the negotiated and fluid natures of social trust relationships and the subtle boundaries we negotiate with others.

Somewhat related, my colleague with statistical training was excited to introduce aggregate behavioral models from activity data and collective intelligence from explicit data in our discussion about contemporary notions of “harnessing the hive.” We pressed through issues in database design and the potential for data mining, as well as the relevance, recommendation, and algorithms for automatically inferring buzz, interest, and so on. Here, “social” was to be found in the shadows cast by humans clicking, clacking, typing, uploading across the interconnected networks of the Internet, and making connections betwixt and between things that were heretofore not there to be connected, or at least not visibly so. We discussed how we sometimes derive models—hypotheses, really—about behavior from these obscured traces and how we are sometimes fooled into seeing patterns where there in fact are none.

I won’t enumerate all views expressed, but surely you get the point. I also don’t want to pretend I was seeing the whole picture as this conversation was unfolding. I was as seduced by all these viewpoints, just as my colleagues were entranced by possibilities for implementation and creation. But later—and it was some time later—when I pondered how this short conversation had played out, I realized that we had all collectively engaged in a “we could build/implement/create/design/make” discussion. I had intended to have a conversation at a higher level—one that addressed what people really need, or what would be really helpful and valuable to people. Stepping back even further in the ideation process, I would have liked a conversation about outlining which spaces to probe to see what people need.

Instead, we were enacting the classic “ready, fire, aim!” approach to design that has been a parody of innovation for many years now—design it because you can, stitch together what you already know how to do, throw it out and see what sticks. This is perhaps a natural human tendency because really creative thinking is hard. In a 1986 paper entitled “No Silver Bullet,” Fred Brooks differentiates software design and development processes; he calls them “essential” and “accidental.” “Essential” refers to the difficult part of understanding the domain for which we’re building software and the determination of what software to build in that domain—what activities to inspire, improve, or transform. “Accidental” is the programming and process that has to follow to implement the solution that has been devised. Brooks aptly points out that we have become quite good at training people and designing tools for the accidental part of software development, but the ideation part continues to loom. He succinctly states, “The hardest single part of software development [remains] deciding precisely what to build.”

So what I meant to inspire when I dropped my word-bomb was a discussion of the role that such a device or application could play in everyday life. I was talking about how whatever we propose fits into people’s social context, into how they manage their everyday doings. I was talking about people, about relationships and friendships, and about the contexts that we inhabit and co-create as we move through daily life. I was hoping to focus on the social settings that would afford, support, and allow a technology to be used. I was talking about delivering value in situ without disrupting the situ—or at least giving some thought to designing ethically; to considering the positive versus negative knock-on effects—any disruptions to the social status quo we wanted to inspire and those we did not. I was talking about social norms in the places people frequent. Whether or not, for example, it is socially acceptable to whip out said devices and applications in public. I was not meaning to talk about ‘like’ buttons and ‘share’ buttons and diffusion models. I was talking about the whys and wherefores, not the hows, whats and whens.

I was silently channeling ergonomists and social scientists who address the physical and/or social constraints of the situation, which can in fact mean that an attention-grabbing application would be dropped in favor of physical and social comfort. I was talking about the potential contexts of use for any application we would build.

I am not saying we should have indulged in an infinite regress into consequences of every design decision, but I am saying it is worth engaging in chain-reaction thinking. I was, however, inviting us to understand what the problem was, where there would be something useful for people, before we start coming up with solutions. It seemed to me that we needed to understand what we were designing for before we starting designing.

I just started the conversation poorly.

I am trying to make a bigger point here, beyond the natural human tendency to trot out well-worn, already known processes, and avoid thinking about hard things. I am also highlighting different ways of seeing the world. I am not a social linguist, but in my gathering there was a clash of understandings. Even in a group that believes it is aiming for the same goal, words mean different things to different people and provoke different semantic associations. This has consequences for collaborative, multidisciplinary design teams. People from different disciplines come from different “epistemic cultures”, a concept I am borrowing (and hopefuly not mangling too much) from Karin Knorr Cetina’s book on the ways in knowledge is created in scientific endeavours. Epistemic cultures are shaped by affinity and necessity; by the ways in which things that arise are constructed as issues or problems or growth opportunities; and also by historical coincidence. An epistemic culture determines how we know what we know. Think of places you have worked: how different organizational cultures think about and institute incentives, employee evaluation, teamwork, work-life balance, and so on. In the same way, “social” may be measured, used, and applied differently in different epistemic communities or cultures. We simply need to be alert to that fact. What this means practically is that we must be aware of the need for translation and persuasion for human-centered design in contexts where multiple parties are present—brokering between different epistemic communities and different constituencies in the design space. It also means that if one is to carry a vision from ideation to implementation and release, one had better be present for all the “decision” gates—be careful about how ideas are being understood as an application or the artifact goes from inspiration to innovation. Because all along the way, the initial vision will slippy-slide away into the details, and those details may consume the whole. Rube Goldberg and W. Heath Robinson come to mind—creators of fantastic contraptions that maybe, perhaps, get the job done, but in the most circuitous manner possible, honoring the engineering over and above utility and aesthetics.

Before I take my leave, let me ask you, as an illustration of all the different perspectives on the simplest of technologies, how many designers does it take to change a lightbulb?

Well, normally one…but:

+1 because we need to user-test the lightbulb
+1 because marketing wants to build the box and brand the bulb
+1 because sales wants to go global with the lightbulb
+1 because engineering wants details specs
+1 since we need to evangelize about it
+1 since we need to build a user community around the lightbulb
+1 since we need to explore different directions we can take the lightbulb