A month or so ago, Nicolas Nova asked me to write an essay for a project he was doing. He said they'd been working on a catalogue of novel "gestures, postures and digital rituals" and could I write about that, please, as a kind of foreword.
"Curious Rituals is a research project conducted at Art Center College of Design (Pasadena) in July-August 2012 by Nicolas Nova (The Near Future Laboratory / HEAD-Genève), Katherine Miyake, Nancy Kwon and Walton Chiu from the media design program. (It) is about gestures, postures and digital rituals that typically emerged with the use of digital technologies…"
It so happens that, for some years now, I'd been collating a catalogue of novel gestures and interactions myself and had never got round to doing anything with it. So Nicolas's kind request gave me the chance to get it off my computer and onto the internet. Thanks Nicolas!
One outcome of their project is a rather nice short film; another is their book, here. My "preface" is reproduced below. Please note: as you'll see, this is not intended to be thorough, complete or even coherent—but you'll get the gist. I would guess many of us have such a list. It's probably more about noticing as a kind of practice, as a curious ritual of its own, rather than the particular gestures themselves. Even though I spent a little too much time trying to get the hands right. (Only to get the arms all wrong.)
21st Century Gestures Clip Art Collection
1. A text file
For some years I’ve been collating a list in a text file, which has the banal filename “21st_century_gestures.txt”. These are a set of gestures, spatial patterns and physical, often bodily, interactions that seemed to me to be entirely novel. They all concern our interactions with The Network, and reflect how a particular Networked development, and its affordances, actually results in intriguing physical interactions. The intriguing aspect is that most of the gestures and movements here are undesigned, inadvertent, unintended, the accidental offcuts of design processes and technological development that are either forced upon the body, or adopted by bodies.
For a while, I have been “a camera with its shutter open, quite passive, recording, not thinking”, simply compiling the list. As a list, it’s entirely subjective, incomplete, and essentially pointless. But I kept coming back to it. Some gestures, interactions or behaviours eventually disappeared from the text file; others were reinforced.
It turned out that Nicolas Nova and collaborators were pursuing a similar idea and asked, by chance, if I might write something about their project.
Rewriting the brief somewhat, I have instead used this essay to put my list to bed.
Walking around “eating the world with your eyes”, as the fictional design tutor in Chip Kidd’s novel The Cheese Monkeys puts it, you can’t help but observe the influence of The Network on our world. Yet The Network is often still spoken about as if it were somehow something separate to Us, as if it were an ethereal plane hovering above us, or perhaps something we might be increasingly immersed in but still separate to our bodies, to our selves. This doesn’t feel accurate now. There is no separate world, and this list indicates how we are even changing what our bodies do in entirely emergent, or at least unplanned, everyday fashion, in response to The Network’s demands.
Yet this isn’t a list of weak signals or extreme and extremely unnecessary positions—such as embedding an RFID chip in your arm, Kevin Warwick—but entirely vernacular conditions, performed by everyday people and created by everyday people.
Working on various projects for the State Library of Queensland in Brisbane, I had begun to informally assess how wifi users were inhabiting the space, not simply in terms of the locations they would flock to, or their patterns of social grouping, but how they draped themselves over furniture in new ways, found small pockets of space to hole up in, while cradling or propping up a laptop, or the newer netbooks that had emerged at that point. There were no iPads back then, but there was clearly a new physical relationship around computing that suddenly left the Library looking askance at its “computer labs” and “drop-in-centres” of “PC towers” secreted under rows of desks.
I drew sketches of the library users co-opting the spaces and objects, ascribing a name to each type as if they were yoga positions. Two laptop users sitting with their backs to each other was “Reverse Battleships”; two sitting facing each other was “Battleships”; one, watching a DVD whilst lying flat on her stomach, was “Front Crawl”; another, astride a bench, perpendicular to the designed stance, was “The Horse”, while swinging one’s legs over one side of the bench but twisting the torso to remain perpendicular would be “Side Saddle”, and so on. I also plotted where people were sitting in relation to wifi signals and building architecture, and discovered clear correlations between people, space, devices and Network.
I made a rough 3D model of the wifi itself, as if it were a physical phenomenon that we could more easily understand structurally, rather than simply connect to. (AHO’s “Immaterials: Light Painting WiFi” later articulated a similar idea in a far more sophisticated fashion.) These were all attempts to understand how The Network could be perceived in civic space. Yet in the yoga position sketches, I was also interested in how people’s movements changed.
The few examples collated here are a development of those sketches, and range from small gestures, to those involving the whole body, to those concerning bodies moving through spaces. Each example has a name and a sketch (below). The list in my text file has more examples than are listed here, but these should be enough to convey the idea. It’s interesting to compare with the much larger list independently collected by Nova and his colleagues. We are looking at the same thing, yet at different things also. What follows is not really a complete list or clip art collection at all—more a sketch of what one might be, in order to provoke more interesting questions.
Size S
In terms of small gestures, one which already seems to be dying out is the Google Map Smear.
In which a user momentarily activates international roaming data and caches adjacent map tiles by smearing the index finger around the user’s intended destination.
This you see when people are suffering from “roaming data” allowances condition on their smartphone’s payment plan, and so are anxious about expensive data transfer. When in a foreign place, the user will temporarily turn data roaming on and load up the cache in their phone’s memory by scooping up the immediately adjacent map tiles, downloading enough map to show them the way, and then quickly turn off the data. This scooping is performed in a series of “smearing” circular movements, as if scrubbing the map tiles into life, rubbing a magic lamp to conjure up some locally useful geography. It’s unlikely this tiny interaction will persist, as maps cease to be produced via tiling, and roaming data becomes less expensive.
In which a user must approach a sleeping PC in both an absent-minded and mildly aggressive fashion, and drum on the keyboard whilst rattling the mouse.
The Wake-Up Waggle, in which a user approaches a sleeping computer and attempts to rattle it into life by hammering the keyboard or aggressively waggling the mouse is at first so small as to be insignificant, but what kind of previous object would we interact with in this way? Perhaps kicking a lawnmower into life, or nudging a sleeping dog with your foot, but it feels disturbingly tied to the way we feel about computers; the mild frustration with which we often approach the device, as if it should by now be guessing when we’re about to use it. There’s a muted irritation here I find perversely appealing.
Note, I haven’t included the wipes, pinches, taps and double-taps common to gestures for touch screens. I’m not quite sure why. They are novel and inventive, clearly, but perhaps also deliberately familiar—they don’t feel odd at all, and instead mimic our previous interactions with sliders, paper or fabrics. They are also clearly designed.
Size M
In terms of bolder gestures and actions which require most of the body, we have the iPad Photographer, the Security Pass Hip Bump and the iPhone Compass Calibrator.
The iPad is held in both hands a little awkwardly and at arm’s length until the Camera app focuses on the intended subject, at which point a finger or thumb is bent onto the touch screen in order to attempt to hit the “shutter release” button.
The iPad Photographer is really a variant on existing ways of taking pictures, but feels awkward and transitional. While there needn’t be anything particularly odd about taking a picture with a largeish rectangular-shaped device, it does look and feel very peculiar indeed. The form factor of cameras has hovered around the hand for centuries, even when embedded in cellphones, yet this is a new body shape, requiring both hands holding the device at arm’s length while one thumb or finger gropes awkwardly for the in-software shutter button.
The Security Pass Hip Bump, which I first noticed a former client doing repeatedly in the library building she worked in, is particularly enjoyable. It occurs when someone carries their RFID-enabled security pass in their bag, and approaching a sensor, lifts the hip to angle the bag towards the sensor, creating a hands-free connection and activating the lock (the hands are often full of paper files, ironically enough.) It’s clearly an odd thing to do, when considered like this—to insouciantly cock your hips towards a small black rectangle on the wall as a form of greeting and personal identification—but it can be carried off with a certain panache. Try it.
In which a user opts to keep her RFID security pass in her bag and gains entrance to her office nonetheless, through lifting and angling the hip up towards the wall-mounted sensor, which unlocks the door upon successful “handshaking”.
When the same security passes are on extensible key fobs, they are articulated as if they were keys; when they are worn on lanyards, it’s as if they were simply identity cards from an earlier age, merely “shown” to a sensor rather than a security guard. Similarly, even something like retinal scans are a long-lost descendent of a family of gestures that might include the act of accessing a medieval walled city—put your face up to the grate for inspection—just as a spoken passphrase also has an ancient history. The Hip-Bump is different. It relies on a loose, instinctive, trial-and-errored understanding of the range of the radio waves involved, and the materials involved, and again feels like a form of interaction entirely unforeseen by the designers of security systems. (Note that Nova and team’s “Bag Swiping” is a close relation.)
Most famously, perhaps, is the iPhone Compass Calibrator, which perhaps shouldn’t be included as it clearly is designed. Yet the performance, often public, that it entails is appealing enough to warrant inclusion. To suddenly stand stock-still in the street and rotate the little rectangle through an arm’s length figure-8 feels almost like a physical incantation, the kind of activity that might lead you direct to the ducking stool were it 17th century Massachussets. One never stops to consider how it might calibrate the phone’s compass; we are simply following orders. What else would we do if Apple told us to? People have a frozen, absent, pretty vacant look on their face as their arm movements mechanically ape the figure-8 described on their iPhone—although some seem to realise the peculiarity of the gesture and offer up a wry smile. (You’ll sometimes also see iPhone users revolving in a circle as their compass finds its bearings, doing a little waggle-dance with the electronics.)
Upon attempting to use Maps app, user is required to move the iPhone through a figure-8 pattern within a vertical plane in order to locate the compass.
Size L
Then there are the larger spatial conditions, in which a person interacts with others, and/or The Network, moving through a space.
In which user moves through busy street focusing solely on smartphone in hand, either reading or typing or both, such that the oncoming pedestrians are forced to grudgingly flow either side.
Cellphone Wake is probably not that novel. Walking down a city street reading a book or newspaper would have had the same effect on a moving crowd a century earlier, but perhaps few would have done it. Now, it occurs frequently, and due to the deeper cognitive load involved in reading, typing and walking simultaneously, we might expect the ballet of the sidewalk to now rely on a heightened sense of “civic proprioception”, an awareness of one’s body in a bounded space of constantly moving objects, constantly yet subconsciously scanning for collision detection. The agent-based pedestrian simulations used to model subways and streets on urban planning projects are only just beginning to incorporate a sense of these interlocking wakes.
While Cellphone Wake is irritating to observe, never mind get caught up in, when a Meeting Room Wake-Up Call occurs, it’s a delicious sight to behold. One of my perennial concerns is in designing systems that enable active citizens. Many technology-led “smart building” visions actually tend towards creating passive citizens, who outsource decision-making about their environment to software, with simple data derived from simple sensors. In this way, passive citizens also abdicate their conscious participation within an environmental system. Passivity of this type cannot create a “smart building” almost by definition, nor will it reverse the irresponsible decision-making culture that created unsustainable automated buildings and spaces in the first place.
. During business meetings, preferably serious ones, a moment’s stillness in the room can trigger over-active motion sensor meaning room is plunged into darkness. Meeting participants must re-activate motion sensor through sudden and impromptu performance of manic arm-waving.
So there’s a secret delight in seeing people interact with technology like this, in seeing the sheer physicality of their interactions—seeing a Serious Meeting of Businesspeople suddenly waving their hands in the arm like they just don’t care, in order to wake up a trigger-happy motion sensor. While it’s not particularly to do with The Network, this performance, when thought of more broadly, effortlessly reveals the absurdity in much of the “smart building” idea. (For more such absurdity, see the related toilet-based “Waving at Sensors” tactic in the Curious Rituals list.)
. In order to achieve maximum throughput over a wifi network, user meanders around a space, holding tablet or laptop out in front of them, occasionally watching the signal strength indicator, until opti- mum compromise of signal and space is located.
I first consciously noticed W-ifi Dowsing during that post-occupancy evaluation of the public wifi at the State Library of Queensland. As one section of the library closed for the day, crowds of visitors would wander out with their laptops still open in front of them, staying connected and looking for the strongest wifi signal. Ultimately, users gathered near largely anonymous wireless access points; located through trial and error, and watching their devices, they manoeuvred themselves towards the best position in terms of The Network rather than the best physical space (or at least, some good compromise of both.) This seemed redolent of the ancient act of dowsing, or doodlebugging, for water, a form of divining trying to perceive and locate a hidden flow—although with rather better results than that entirely spurious practice. Neither public spaces or laptops are designed with this movement in mind.
2. Clip art sociology
I’ve drawn these activities in a kind of 1950s newspaper illustration style, often seen in classified ads at the time, and now used as clip art (they’re actually adapted from scanned originals found on Flickr.) Those ads, and indeed clip art, often work as some kind of subconscious “training” for modern life. They prime what typical behaviours, are, or should be, given clip art’s often aspirational primary uses.
Clip art such as this usually describes a form of physical interaction, in space, bereft of backstory. They are simply “man sitting at desk”, “woman, smiling, answer phone”, “people dancing”, “office party”, “shop interaction”. They describe context but not content, so that they might be freely used to underscore numerous different narratives. They are a frozen moment of interaction, with no before or after.
So perhaps we might read clip art as an unwitting description and guide to the stereotypical environments and interactions of our world.
This is why David Rees’s legendary comic strip “My New Filing Technique Is Unstoppable” is quite so funny; in its crude clip-art bastardisations, it highlights the mismatch between an age lived on The Network (us) and the office culture of the near past (them). It freeze-frames people (mis)filing paper into filing cabinets and playfully and profanely deadpans them into an absurd alternate universe, which tells us as much about contemporary corporate-speke and the thrill of The Network as it does about the ‘80s or early ‘90s office environments its characters seem to be drawn from.
I transcribed some of the emerging behaviours of our modern world into this language of clip art to similarly freeze, distill, highlight and to provoke a discussion as to whether they are simply peculiar blips or transitional hints as to where we’re going next, taking an everyday performance and stripping it of its context, pinning the butterfly under the glass.
The sports jackets can only help in this regard.
3. Could have been contenders
We can argue about these particular examples all day. Some entrants on the list get removed upon consideration. A man stooping and pointing his phone to capture a QR code at the bottom of some advertising hoarding, as I saw in the street in Helsinki the other day, is really only a variation on a photo-taking pose, simply held a little longer due to machine vision’s poor acuity.
Certainly, the act of taking photographs has changed shape due to digital technology. Where once the camera was generally brought up to the eye, now it is more often active at arm’s length, thanks to LCD screens on the back, and the “fire and forget” sensibility enabled by cheap storage, or increasingly a cloud-based upload meaning storage isn’t even an afterthought. (Those of a certain age will remember how precious each of your 24 or 36 shots on the roll of film used to be.) The freedom of movement these developments have delivered have been exploited by professional cinematographers and the average punter alike. But again, it is only a variation on a previous mode, and equally, many photographers will have shot at arm’s length for as long as cameras have been light enough to do so. There is nothing to do with The Network about this performance. (How might the physicality of taking a photograph change due to The Network is a good question, however.)
People certainly move in a distinctive fashion when they’re talking on the phone. I observed our two year-old daughter talking into her plastic Hello Kitty pretend-phone and airily meandering around the apartment, brushing her fingers absent-mindedly over surfaces, just as her mother does, as she talked to her mythical mystery companion. You might argue the Bluetooth headset or the iPhone’s earbud-based mic has freed up one hand, but the essential stance of Being On The Phone hasn’t changed as a result of The Network.
Even a intrinsically contemporary product like the Nike Fuelband is interacted with like a wristwatch, just as the laptop is essentially operated like a typewriter. Even what the body does in an achingly new game like “Tearaway” on the PS Vita, which features a riot of interaction design breakthroughs, would essentially be familiar to a 1950s child’s imagination—blowing on characters or tilting platforms to knock them over, or using fingers as puppets, for example. The novelty is in the fact that the characters are digital and Networked, rather than in the gestures themselves. It’s why these interactions work, and again, they’re entirely designed this way. All would be familiar, one way or another, to someone magically transported from 1965, say, even if the outcomes of those gestures would be largely beyond their comprehension.
4. Immaterial weightlifting
But the interactions in the text file should seem odd to the time traveller, just as they seem slightly odd to us when highlighted in this way.
These physical acts all make it evident that there is no separate “virtual world”; our very bodies are shaping our digital interactions. We are part of The Network, and not just intellectually, in terms of our projected persona and identity, but physically. The body is making The Network visible, legible. Tracing the articulation of the hips, hands and arms is sometimes tracing the seams of The Network.
We have a long understanding of how the body creatively interacts with invisible forces. If you watch footage of Jimi Hendrix, you can see how he used his body to shape his guitar’s feedback; the sound is produced by the interplay of his guitar, especially its pickup, the speakers, the room, and his body within electric fields, in space, over time. Sensors and actuators, in contemporary language, at play within invisible fields, but shaped by the body, as well as objects and spaces. We need to think in terms of these fluid immersive interactions, material and immaterial—or “transmaterial as Mitchell Whitelaw would say—which we are part of physically as well as intellectually. This implies the conceptual separations of Hardware and Software, Them and Us no longer stand.
Unlike the objects that are being interacted with—here deliberately represented as characterless rectangles—the bodies reveal the patterns of information exchange. Rather than passive users in meaningless space with screens bringing all the action from elsewhere, these interactions foreground the idea of “Screen as object in the world, rather than window to somewhere else“, as Mitchell Whitelaw puts it. “Window to somewhere else” has been around for a long time, yet as Whitelaw’s essay “After the screen” suggests, we are now working with “glowing rectangles” in a new way, and sometimes interacting with The Network via objects in the world that are without screens, rectangles without the glow.
The kinetic energy expended in the extravagant contortions of the Security Pass Hip Bump, or even the subtle twitches of the Map Smear, suggest the transactions at play rather more the objects themselves do. You can almost viscerally feel the tiles being created during the Map Smear.
As Bruce Sterling put it in “Shaping Things”, the objects are merely “material instantiations of an immaterial system”, and relatively faceless ones at that, whereas the immaterial system is the aspect which is “so overwhelmingly extensive and rich.” Perhaps it’s in the body that we sense that weight of information elsewhere?
In 1985 Italo Calvino wrote “It is true that software cannot exercise its powers of lightness except through the weight of hardware.” Time has passed, and the heaviness of hardware has dissolved into lightness, as a counterpoint to the increased weight of associated information on The Network. But the body has also become a weighty vessel for software, a site for the Network to express its dynamics, well before we explore the next wave of wearable computing.
5. YouTube: “early flying machines”
Speaking of lightness, when we look back at old movies of early flying machines, we see frail bodies trapped in awkward wooden frames, some of which hop unpredictably in and out of the air, as if a plastic bag blown along by the wind, while others plummet headlong from jetty to water with all the sense of purpose that gravity can muster. Those bruised and soggy aviators caught flapping articulated arms we can now discount as pursuing a developmental dead-end, though it was probably worth a try at the time. In others, we can see the blueprints of successful subsequent flights emerging before our eyes.
These contemporary actions are similarly unpredictable. We don’t know if the Security Pass Hip Bump is actually a precursor of what happens when our clothes are made of smart fabric derived from nanocellulose fibres, and we use the combination of body and fabric to receive, transmit and display data. Is it a form of prototype, or is it simply an absurd contortion inflicted upon us by a building contractor placing an sensor slightly too high?
They leave us to wonder whether we have designed “things which are in harmony with the human being and organically suited to the little man in the street”, as Alvar Aalto put it. Perhaps these all too human responses indicate we actually have designed in such harmony, totally inadvertently? Or that, just as the street finds its own use for things, our stereotypical physical movements simply adapt, if the promise of The Network is worth it.
6. Learning from wooden spoons
You might argue over these choices; that’s the point. Such behaviours come and go every day. The point is only to observe, perhaps. These are all transitional, and at this point they tell us about here and now. In other words: at this point, they tell us about this point.
But over time, when you watch enough, and log the iterations, they might hint at where we’re going, at our future physical interactions. Bruno Munari once wrote that the repeated use of a wooden spoon for stirring and testing soup not only wears the spoon down, but also “it eventually shows us what shape a spoon for stirring soup should be.”
We can’t easily see that creative wear and tear taking its effect upon The Network and our interactions with it. Unless we discuss it, we can’t recognise the shifts in physical and social interaction occurring as a result of something that is often invisible, or leaves little trace either way, and whose objects are discarded with a frequency that would make Munari wince, and the simple wooden spoon feel quietly triumphant.
What we’re witnessing here are tentative vernacular sketches as to how we might physically interact with The Network. Just as those early films of flying machines are equally absurd and prescient, these contortions and behaviours might contain the clues of our future interactions.
Building an unfinished catalogue of them, as I have here, is also absurd, clearly; I leave it to you to do the prescient bit.
City of Sound 
Leave a comment