[ This article: HTML, ASCII, PDF ]

Synergistic storyscapes and constructionist cinematic sharing

by G. Davenport, S. Agamanolis, B. Barry, B. Bradley, and K. Brooks

This paper describes recent progress toward a framework and toolset for Very Distributed Storytelling. It focuses on three independent but interconnected axes of development: continuous story playout environments, story authoring systems, and scenarios for interaction, particularly those that operate in casual or formal architectural spaces. Aspects of these are explored in several systems currently under development at the Interactive Cinema Group of the MIT Media Laboratory. Agent Stories invites authors to develop story segments that know about their role in the story and agents that can use these role designations to orchestrate the playout of story parts. Happenstance, an ecologically based, context-sensitive “stage” for the performance of interactive narratives, provides an information landscape that is continuous in space and time, supports multiple viewpoints, and acts as a semi-porous membrane to information and messages from the outside world. The CINEMAT and other large-scale interactive media pieces—situated in sensor-rich architectural spaces—explore the relationships among immersion, interaction, narrative guidance, and public space.

Interactive Cinema reflects the longing of cinema to become something new, something more complex, and something more personal, as if in conversation with an audience.
    —Mission statement at http://ic.www.media.mit.edu

Throughout the ages, storytellers have exploited available technologies to craft compelling tales that entertain and educate their audience. Narrative expressions—manifested in forms such as painting, song, theater, the novel, movies, and television—have provided reflections on the philosophical and sociological state of the local culture. As expression intersected with new production and distribution technologies, new formal modalities were born. In the 19th and 20th centuries, radio, motion pictures, and television created forms of media that made the ordinary story extraordinary; however, these forms maintained a single limited, ordered narrative perspective. Although cinema projects a monolithic, immersive experience to a society of audience within the theater, the audience has no effect on the outcome of the story. Television is even more rigid because it sidesteps the sociability of audience entirely, delivering an ever more isolated and isolating watch-alone experience.

By means of a distributed network architecture and its computational capabilities, digital narratives of the future are capable of expanding the social engagement of audiences while offering intensive narrative immersion in a story experience that plays out in multiple public and private venues. The term Very Distributed Storytelling suggests a new medium in which multicast, point-to-point networking and local computation converge to offer a dynamic, morphogenic experience whose form and content emerge on-the-fly as authors, audience, and machinery engage in the collaborative co-construction of meaning and experience.

Built in the age of networked communications and “remote telepresence,” these electromorphic stories of the future can be delivered to multiple local and global channels in asynchronous as well as synchronous time. Ultimately, the available computational machinery must flexibly reconfigure, extend, and time-shift public and private “spaces” to accommodate the needs, desires, and interactions of the globally distributed audience. This paper focuses on three independent axes of development: a metaphor around which to build a continuous story playout environment, a story authoring system that facilitates the creation of multi-point-of-view stories, and some experimental story scenarios that invite interaction. These research directions, developed in parallel within the Interactive Cinema Group of the MIT Media Laboratory, grow out of assumptions made about the future of the story experience, which are described in subsequent sections of this paper.

In order to attract and maintain a loyal following, these narratives require the continuous production of engaging, immersive content. Professional and novice authors alike need tools to allow them to exert maximum control over the creation, sequencing, and playout of their work. Over and above traditional production tools and formats, simulation tools allow authors to optimize their narrative contributions to the particular idiosyncrasies of the complex playout environment.

In addition to an evolving array of story elements, Very Distributed Storytelling uses a diverse array of presentation modalities and synergistic devices. The narrative of the future will take place simultaneously in multiple venues: on networked computer workstations, in large-scale public spaces, and on small mobile devices (such as pagers, cellular phones, and “wearable” computers). Intentional mapping of actions and interactions between these venues will provide additional interest to the story as a whole. For instance, an audience member ignoring an instrumented performance situated in public space could trigger a message being left in the landscape browser of another audience member. Conversely, through a browser, an individual can proactively engage with story elements that might be posted on a large-screen display in a hallway or a small-screen display on a pager.

Down the rabbit hole with Heisenberg

The seeds for an expansive, audiovisual “virtual reality” were sown in the poetic fantasies of Homer and reached a pre-Holodeck epiphany with the 1865 publication of Lewis Carroll's Alice in Wonderland.

If we are familiar with this story, we remember Alice sitting dreamily on the bank, wondering whether or not to construct a daisy chain, when suddenly she sees a white rabbit with pink eyes running past her. Surprised by the fact that the rabbit is talking to himself, Alice is drawn to follow. In a turn of phrase in the first paragraph of a remarkable fantasy adventure, Carroll manages to invert reality, allowing us to race (with Alice) across the field after the white rabbit, to watch as it disappears down a rabbit hole, and (on the heels of Alice) to follow after. Would that our browsers of today would offer such an enticing invitation!

In synthetic narrative reality, the metaphor of the journey takes on added nuance as we are asked to navigate our own path through story space. As we “pop down the rabbit hole” into the storyscape of the future, reality and fiction merge and blur. The incorporation engine and cognitive processes—triggering a sensor, decoding media segments, constructing messages—are mapped against a represented reality of place, character, and action in order to fully engage us in an infinite, transformational landscape that is metaphorically understandable.

The fluid “interactive” storyscape

Continuous space grounds expectation. Continuous travel through a landscape is a familiar sensation to people who have grown up with automobiles and other motion vehicles. Perhaps this familiarity led researchers of the Architecture Machine Group at MIT to develop “The Aspen Movie Map” in the late 1970s as a visual surrogate travel experience. The Movie Map system controlled two identical videodiscs, each containing thousands of still images documenting the streets of Aspen, Colorado, in each of the four seasons. Some historical photographs were included, as were a very few minimovies that presented brief encounters with the people behind the facade. Participants personalized their journey via icons that were easily read as Turn Left, Turn Right, Stop; indications for “change the season” and “go in there” were slightly more obtuse but easily remembered once they had been discovered. Control was provided via a “point-and-click” graphical interface on a touch-screen display or by sitting in the “driver's seat,” a leather lounge chair modified with a joystick on the arm. Using the concept of limited look-ahead, the computer ensured that the videodisc player not currently busy displaying frames would cue to the next appropriate selection of frames; this removed the latency associated with searching and read-head travel time to give an approximation of a “seamless” experience.

In 1994, almost 15 years after the Aspen prototype was realized, the Miller brothers released Myst**, a CD-ROM that married story to a navigational interface. Myst invited the participant to navigate a computer-generated landscape: by turning left or right, or by moving from a wide shot to a close-up shot, the voyager was eventually confronted with mechanisms that needed to be unlocked in order to proceed further into the storyscape. Beautifully rendered in sound and image, this CD-ROM became a cult hit; in laboratories and homes across the United States and the world, people gathered together in small groups to solve the difficult puzzles. In this way, Myst was perhaps the first digital story that generated a true society of audience. However, Myst was a completely authored environment that presented the same situations to the participant or audience time after time; once the puzzle was solved, there was no compelling reason to return.

Happenstance: Birds and bees, clouds and trees

Brian Bradley's Happenstance system—currently in prototype—takes a more proactive, generative, “society of mind” approach to interactive story construction in an information-rich world. Every object in the virtual landscape, and every piece of information that flows through it, is a semi-autonomous behavioral entity wielding its own story engine. Important aspects of story content and context are made visible, tangible, and manipulable by systematically couching them within the metaphors of ecology, geology, landscape, and weather.

These “smart” story parts, like the audience members themselves, are empowered to point, click, type, and travel with consequence in the story world; thus, audience and content collaborate as equals in the co-construction of meaning. If the audience does not go to the information, the “smart” story part autonomously seeks out its audience.

The beauty of such a landscape is the ability of the presentation to both entertain and provide a logical framework for commonplace activities—sending and receiving e-mail, researching background material, or chatting. The audience activity immediately expands beyond point and click, type, search, transact, and reply since the landscape provides a visual metaphor for navigation, meeting points, and temporal or spatial memory. Conceptual landscapes can grow, change, and evolve as information is sought, delivered, and manipulated over time. For example, current information can behave fluidly by “piggybacking” on the Earth's water cycle (Figure 1); as a cloud floats across an individual's browser, it may deliver the latest weather report for his or her area or offer a background story about the making of a new feature movie. (The cloud is one of many semi-autonomous behavioral entities that move through the story world, actively seeking opportunities to perform.) Older information, history, and complex conceptual constructs—built up by the flow and accumulation of data over time—are made manifest in rock and soil cycles. Directed inquiries, explorations of theory, and activities associated with the audience's personal interests are reflected by plant growth.

Figure 1

Thus, information itself is imbued with its own characteristic “life cycles,” states, and behaviors that are (in part) derived from the “natural” forces, flows, and process cycles of the virtual world. Metaphor (and its resultant interface) is an emergent property of these dynamic “vehicle-tenor” couplings.

In a process that mimics realistic physical modeling, Happenstance implements metaphor as metaphysical systems operating on representational meta-data. As a result, many threads of association simultaneously play out within the landscape, providing opportunities to move the story experience forward, including input from other story venues as we will discuss later (Figure 2) or objects designed to live in a shared landscape. Time, space, theme, and point of view become navigational axes for smooth or discontinuous motion through one's own personal theme park of tools and information. Anything within the vast, networked reach of the computer becomes potential grist for a storytelling mill.

Figure 2

The time is early evening. Marvin (not the real name of the participant or audience) stops by his house before going out for the evening. He passes a Happenstance browser screen embedded in the wall; the screen saver indicates stormy weather, both literally and figuratively. Marvin launches himself into his personalized landscape browser in order to pick up the latest messages. Marvin enters where he left off, at the lake; however, the leaves of his former search are now on the ground. As Marvin makes a quick query about his weekend lodging, a few leaves sprout on the tree. Before he has time to examine these messages, a pigeon flaps its way into the landscape and lands, pecking and cooing until Marvin approaches it. On closer examination, he sees that it is a carrier pigeon with a note around its neck with the message: “Meet me at Madame Zelda's.” Who will he be meeting, he wonders? He agrees, and falls through the note into a vast, arid desert.

Madame Zelda is a famous fortune-teller, working out of a tent at the base of an active volcano; Marvin enters the tent; Madame Zelda draws two cards: entertainment tonight and weekend passage over water. As he leaves the tent, the volcano erupts, shooting an “Eastern Mountain Sports” cloud high into the air. The cloud, now over Madame Zelda's tent, begins to rain on her Tarot cards, and the images on the cards are transformed. “Water” becomes a “canoe.” White water rafting directions and the tales of prior adventurers follow.

A cloud of dust appears in the distance, moving closer and closer every second; what could it be? He soon finds out: a mighty hurricane races across the dunes, leaving a trail of devastation in its wake. Within its whirling, swirling activity, words form in the air: it is a pager message from Margaret, telling Marvin that “Because of an emergency, I must cancel our weekend trip.” Only after it has passed does he realize that Madame Zelda's tent is no longer there.

Almost immediately the scene cuts to another one: Marvin is apparently in a taxi moving toward some urban destination; a couple stand at the street corner as another cloud forms on the horizon.

Agent Stories: Writing for Bob, Carol, Ted, Alice, and even Madame Zelda

Ted can be characterized as a democratic teller who presents story conflict through the eyes of as many characters as possible. Marvin watches as a series of scenes play out. A couple, standing with a bike between them, begin to cross a busy street. The couple only see each other, the way lovers often behave. A cab driver slams his foot on the brake and screeches to a halt in front of the couple in the midst of heavy traffic. The passenger in the back seat appears to be in her early 30s and is in labor. The cab driver hits his horn; “Get it out of the street!” he shouts, just as the young man trips and almost falls to the ground. “Junkies,” the cab driver thinks. The boy, not paying attention, trips and nearly falls; the bike goes flying across the intersection; he hangs onto the woman, who appears to twist her ankle. He is trying to get them both across the street. She becomes increasingly disheveled as he paws her uncontrollably. “Why can't he simply walk upright like other Homo sapiens?” the cab driver thinks, screeching past them. An older man recovers the bike and brings it over to the couple. “Better tie your shoes now.” Marvin looks down; yes, he had best tie his shoes before he leaves for dinner.

But first, Marvin wants to see the story again, this time told by Carol. Carol can be characterized as a one-sided storyteller who presents many versions of conflicts but only one character's resolution—the story is broad, but in the end seen through only one character's eyes. Marvin watches as the new story begins with the cab driver slamming on the brakes in front of a frantic pregnant woman who has stepped out into the street to hail his cab. She is his sister, and she is in labor. They speed toward the hospital. The story progresses through many of the same images Marvin saw in the first story, but this time with more detail and more focus on the cab driver. For playout, the ambient sounds are automatically adjusted to reflect the story point of view: the cacophony of the streets and in-cab noises dominate the soundtrack.

The movies that Ted and Carol presented to Marvin in his personalized browser environment were written by storytellers who, like Marvin, enjoy their own personalized environments. The stories were written to reflect different points of view (awkwardness, love, compassion, hate, pain, fear) and to allow a presentation agent to emphasize different aspects of the human dynamic by selecting and sequencing the appropriate video clips. Viewers engage in these stories multiple times by their own choice in order to engage with an altered view of the circumstance of story. In this way, these multiviewpoint stories enlarge traditional story form, creating an engagement for modern audiences who understand that there is no one story, that their own life story depends on the teller, that many complications in our lives occur because we are too myopic to listen to an alternative teller. In order to succeed, the quality of the story is critical and must support the form; the story must be rich enough to warrant multiple renditions.

The human author used Agent Stories, a writing environment developed by Kevin Brooks as part of his Ph.D. research in the Interactive Cinema Group of the MIT Media Lab. This environment allows the author to write sequences of a story in a lexia format and later develop the video sequences. Lexia are the smallest meaningful chunks of story—a simple tile for building the story mosaic. Each lexia is hyperlinked with others, creating a web of connected statements, actions, and events as perceived by a particular character. As the writer develops the lexia, he or she attaches meta-data tags—speaker intro, character intro, conflict, resolution, diversion, etc.—to the segment; these descriptors reflect the structural contribution made by the sequence to a presentation (Figure 3). The author also builds story-relevant conditions (such as “precedes” or “follows”) to the lexia through the use of a graphical construction set. At any point in the process, the writer can simulate how particular agents might present the material and “tweak” their performances by experimentally revising the descriptors or the text before committing to the final production cycle (Figure 4).

Figure 3 Figure 4

Agent Stories is a significant tool for the development of a multilinear story form because, while it allows the writer to develop a complex web of story material (Figure 5), it also provides the author with adequate structural control over the playout of a story through both the structural annotation of content and the design of story agents. The tool invites writers to develop complete scenarios and structures, as well as to shape teller agents who bring different telling strategies to the constructed story. As a storytelling tool, Agent Stories simultaneously expands the invitation for storytellers to participate in a distributed environment and complements the phantasmagoric browser environment described above.

Figure 5

The entire micro-web structure of lexia or story chunks and their interrelationships, or a simple signal that a particular story action has taken place, can be applied to other storytelling environments such as digital broadband systems, mobile phone, or local and wearable installations for recontextualized use.

Six writers and a handful of researchers have already used the stand-alone, Metropolis-based version of Agent Stories to create bodies of content. Upon completion of the Web-based, Java** language version of the system in June 2000, large-scale testing was begun.

Out of the box: Experiments in casual architectural space

To prototype a framework for these stories, this aspect of the Very Distributed Story world draws inspiration from the peripatetic world of the theme park, the aesthetic pleasures of situated art works, and the thrill of live theater as much as from cinematic theater. How could we build a sustainable, large-scale experience that is infinitely revisitable, flexibly recombinant, emotionally rich, and meaningful to a larger story context? Hallways, lobbies, and storefronts began to attract our attention: Could we reinvent casual architectural space in the digital domain, just as storefronts of the 1950s were reinvented by artists such as Andy Warhol?

At this point, our constructionist selves took over. What could we make in our own workspace that would be arresting and surprising? A few weeks later, we equipped an eight-by-twelve-foot section of a rather narrow hallway with a large rear-projection screen, a sound system, and an array of sonar sensors. The sonar (developed by Joe Paradiso) sensed when someone was present in the space. Stefan Agamanolis' ISIS language was used to manipulate the relevant streams of audio and video in real time and to interpret the data from the sensors.

The first “casual encounter” we projected was Sammy Spitzer's minimovie of pigeons scrapping over little bits of bread in a park. When someone walked by, the startled pigeons flew away, only to return half a minute later. Somehow, this simple, brief experience had universal appeal; passersby immediately understood what they had done and laughed in delight as they sent the pigeons flying. But, where did the birds go? Serendipitously, the pigeons could be thought of as carrier pigeons that leave the live site, travel across the network, and enter the World Wide Web story space that others were engaged in. A simple message broadcast to the larger network made the pigeons' status known and usable to other connected story environments.

A slightly more ambitious project followed: Could we develop a concept that could be implemented by local production crews around the world? We settled on the culturally rich topic of dance. The challenge was to design an interaction with passersby that had meaningful consequence.

For a first-pass experiment, we enlisted Jayshree (a classically trained Indian temple dancer) to perform in our bluescreen studio. As she danced to a prerecorded chant, finger-cymbal, and bell track of her own making, we filmed wide, medium, and close-up shots of her performance from three different angles.

In the final implementation, the isolated image of Jayshree was composited against fanciful background images (which could be stored locally or shipped in from across the network). The scenario begins with a tiny Jayshree floating through the hallway, looking back and forth, scanning for an audience. When a passerby enters the space, Jayshree prepares herself by applying makeup and her ceremonial ornaments. Then, she performs her dance of welcome. If the spectator leaves before Jayshree is finished, the system cuts to a close-up of her glaring at the departing audience; the sensor technology deduces the proper direction to aim her scowl. This piece engaged passersby and—according to audience reports—made those who incurred Jayshree's wrath feel a little guilty (Figures 6 to 9).

Figure 6 Figure 7 Figure 8 Figure 9

Next, we exploited another type of sensing device: Joe Paradiso's “Magic Carpet,” which contains an underlying grid of piezoelectric cables to sense the location and power of footsteps upon it, rather like a huge graphics tablet. This novel interface—which was much hardier than the sonars—led us to develop the CINEMAT “Dream Weaver” that premiered at the 27th International Film Festival in Rotterdam.

Out of the box and on the road: Proof of the pudding

At the Rotterdam Film Festival, the Dream Weaver was installed as a stand-alone, unmoderated attraction: The audience was on its own to deduce how the carpet zones mapped to content. In watching many participants move on the carpet, we could never be sure whether and how specifically stories were imagined; the learning curve was daunting to some and easily mastered by others.

Shortly after the Rotterdam Film Festival, an improved and expanded CINEMAT—strengthened by the addition of two new interactive scenarios—traveled to the Espacio '98 convention in Mexico City. Much to our surprise, Televisa (the sponsor of this event) had built a small theater to house the installation and assigned student “Masters of Ceremonies” (MCs) to staff it. Shows would run every half-hour. At first, we were horrified. How could an interactive story work with an audience of 100 people? Then, we realized that this framework would allow us to explore many social aspects of interactivity that were not available through the unmanned Rotterdam setup.

The “Dream Weaver” became a game where one volunteer would work the carpet while another improvised a coherent tale based on the sequence of images being projected. Invariably, the teller scripted the person on the mat into the story. A video clip of a sad-looking man appears: “She [the person on the CINEMAT] has just broken up with her boyfriend.” A picture of a fish appears: “She is sad and goes to the aquarium to talk to the fish … ” Sooner or later, an image would appear that did not fit their improvised story trajectory; then, the flummoxed storyteller would pause indecisively, and everyone would dissolve into laughter. This signaled the end of the game, and the MC would move on to the next scenario (Figures 10 to 12).

Figure 10 Figure 11 Figure 12

The second interactive scenario, “Endless Pursuit,” was a body-kinesthetic updating of the Aspen Movie Map concept. The story began with a student buying a sandwich from a food truck; then, a thief snatched the food from his hand and led the participant or audience on a merry chase throughout the maze-like complex of interconnected buildings on the MIT campus. The faster one jogged on the CINEMAT, the faster one moved through the video “map.” Forward, backward, left, and right were determined by zones of the carpet. This entertainment was surprisingly popular. Hoots, hollers, and shouted suggestions from the MCs and the larger audience added a substantial element of social performance to the active participant's efforts.

The third scenario, “Not Without Risk,” cast the participant or audience in the role of voyeur. The first challenge was to navigate a creaky stairway to a second-floor doorway. If the mat-walker made too much noise going up (by walking near the center of the stairs rather than the edges), he or she was discovered: A beautiful woman would open the door, tell him to “Go away!” and slam the door shut. If the mat-walker was sufficiently stealthy, he was allowed to peer through the keyhole at an interesting scene: two magnificent women lounging and gossiping as a sneaky burglar ransacks their apartment. The successful voyeur's second challenge involved peering through the narrow keyhole, which allowed only half of the video frame width to be seen at any one time. By moving left on the sensor mat, the false-perspective keyhole view moved to the right, and vice versa. Thus, by choosing the visual point of view, one could follow part of the action in the room—at the cost of missing whatever was happening in the hidden parts of the frame.

The Rotterdam and Mexico installations stood alone; they were not networked to other story venues (although, like Sammy Spitzer's pigeons, they easily could have been). The thief steals jewels—on the Web you can play sleuth and help to locate them! Since thousands of people were playing these scenarios within the span of a few days, we focused on the social aspects of these interactive spaces. This experience demonstrated firsthand how the presence of an intelligent moderator, offering commentary and advice about what to do next, can minimize or even completely eliminate the “learning curve” associated with the audience's mastery of an unfamiliar interface. It also highlighted the fact that rich “back-channel” communications with a real-world audience can add pleasing elements of shared public performance to what would otherwise be private acts of exploration; the tasks of story construction were enriched by a layer of exegetic metacommentary, which fed back to the primary activities of making stories.

Conclusion—Yearning to become something more

In our participation as “users,” we can no longer be doomed to point, click, and type our way through the digital universe, regardless of the task at hand. Complex story environments of the future must actively challenge our ability to act and transform our emotional state. Moving and performing tasks within these environments must bear some resemblance to the body motion and sensory experience of real-world encounters; but, the higher-order use of metaphor and synthetic emphasis is also necessary if the experience is to be perceived as storylike. So far, video games are the only form of electronic entertainment that have successfully shaken the general public's notion of television viewing; however, video games rarely make us gasp with fear or tug at our heartstrings. Over the past 25 years, story makers and engineers have grappled with the implementation as well as with the underlying architecture for interaction.

Theatrical and cinematic theory dwells heavily on the “fourth wall” phenomenon—that psychic barrier that separates the audience from the onstage action. Less has been said about the “fifth wall”—that semi-porous membrane that stands between individual audience members during a shared experience. Real-world audiences—gathered together in the same place at the same time—are conjoined by rich sensory channels and a shared physical context; however, networked interactive experiences must rely on “remote telepresence” to emulate selected aspects of an audience's physical co-presence.

Audiovisual “back-channel” communications (such as “chat rooms” and videophone connections) are already changing the face of networked interactive sharing. However, other significant types of narrative interconnection are just beginning to emerge. The ability of highly distributed story environments—separated by distance, time, and culture—to pass behavioral objects, their states, and pieces of the local context to others signals a revolution of immense importance.

How can a carrier pigeon, dispatched by audience motion in a Mexican “live site,” inform the white rabbit in your Web browser that he is late for tea, initiating events that send Alice tumbling down the rabbit hole? This is the challenge that will redefine the highly distributed interactive narratives of the future.

**Trademark or registered trademark of Cyan, Inc., or Sun Microsystems, Inc.

General references

B. Bradley, Happenstance: Haptic, Happening Circumstance for Life and Play via Metaphoric Narrative Landscape Ecologies, master's degree thesis, MIT, Media Laboratory, Cambridge, MA (in process).

K. Brooks, Metalinear Cinematic Narrative: Theory Process, and Tool, Ph.D. thesis, MIT, Media Laboratory, Cambridge, MA (1999).

L. Carroll, Alice in Wonderland, Macmillan, London (July 1865).

G. Davenport and M. Murtaugh, “Automatist Storyteller Systems and the Shifting Sands of Story,” IBM Systems Journal 36, No. 3, 446­456 (1997).

R. Miller and R. C. Miller, MYST, Cyan/Broderbund Inc., Novato, CA (1995); Cyan, Inc., now at 14617 N. Newport Highway, Mead, WA 99021.

R. Mohl, Cognitive Space in the Interactive Movie Map: An Investigation of Spatial Learning in Virtual Environments, Ph.D. thesis, MIT, Department of Architecture, Cambridge, MA (1982).

Accepted for publication May 30, 2000.

©2000 IBM Corporation