Interactive Transformational Environments: Wheel of Life

Glorianna Davenport, Associate Professor of Media Technology, MIT

Larry Friedlander, Professor of Literature, Stanford

Abstract

Introduction

Collaboration often calls on circumstance. In the summer of 1992 Friedlander came to MIT as a Visiting Professor of Literature and Media Arts. Both he and Davenport had long been fascinated by the literary and artistic possibilities of interactive technology, and both were intrigued by the idea of creating an Interactive Transformational Environment. They agreed to design and build such an installation in the Villers Experimental Media Theater at the Media Lab in MIT as a project for Davenport’s upcoming Workshop in Elastic Movie Time.

The collaboration began with a series of questions concerning the future of interactive technologies and interactive environments. As technology becomes more ‘intelligent’ and more precisely responsive to a user’s wishes and actions, its functions can increasingly be distributed throughout the everyday environment. Eventually, might not the computer itself dissolve into the very fabric of our environment? Could we not imagine a world thoroughly permeated with hidden functionalities invisibly available to us? What would it be like to be in a world that ‘knew’ we were there, and that was totally responsive to our every move; a world that literally transformed itself as we traversed it? What kind of rules might pertain in a landscape dominated by change and by transformation?

These ambitious musings helped us formulate some initial goals for our project: Þrst, we would create an interactive world situated in a real space outside of the computer box, a kind of museum installation cum theater-set. This space, however, would retain many of the

functionalities of the computing environment. Visitors would immerse themselves in this world with their whole body, mind, and feelings. We hoped that it would feel as though they were walking through a computer monitor into a magic landscape.

Second, we decided the space should contain or embody a narrative, and that the narrative should be actualized by the transformative actions of the visitor moving through it. After much discussion, we settled on an overarching theme: the wheel of life, the cycles of change and continuity that whirl us along in the journeys of our lives. The space would be broken down into four sub-spaces, each containing a distinct story but each connected to the space as whole. Each sub-space would take as a starting point for its design one of the traditional elements — earth air, water and Þre — and each element would be associated with a set of attributes, — emotional, historical, technical.

Finally, we wanted to explore the connection between this new kind of place and traditional computing environments. What if we had two kinds of participants: explorers, who would in fact enter and explore the space, and guides, who would sit at a workstation outside of the space and, using the computer, help the explorers navigate through the different areas? And what if, to make matters more interesting (and frustrating!), these two would have to communicate not through words but through the matter and media of the world itself, its colors, sounds, images? Perhaps these constraints would force the participants to Þnd new methods of expression and communication and would encourage us, as designers, to push at the limits of interface and spatial design.

Organizing the Project

Typically the group begins by deÞning the aim and methodology of the project and then divides itself into parallel groups centered around speciÞc aspects of the project. Early exercises build student conÞdence and understanding of the complexity of the design task, and a signiÞcant portion of class time is devoted to student presentations and discussion. An important goal of this style of teaching is to hone student understanding of a fabricated world, and to build communication skills which are necessary for collaboration. This process keeps ideas ßuid and insures that all sub-projects will eventually merge into a single expression.

Class sessions alternated between presentations by experts on issues such as lighting, construction materials, sensor devices, Þlming methods etc.; reports by the students of their progress; and exercises designed to provide insight into conceptual issues. The class also studied the techniques used by some modern artists, particularly the Surrealists, to create starling and unexpected visual experiences: for example, the juxtaposition of dissimilar objects; the deconstruction or reconstruction of such objects; the alteration of viewers’ perspectives; and creation of totally imaginary worlds.

Process and Challenges

The challenges facing each group were enormous: the students, who had strong technical backgrounds, had to turn themselves into set designers, sculptors, story-tellers, and theater people. They had to both develop a workable conceptual scheme and build a large-scale installation, all in a short space of time. Basically students were working in the dark; plans might look good on paper, but until we opened the installation we would not know if visitors would be able to read and interact with our designs. In fact, it took many weeks to evolve and iterate the various narrative concepts.

The Þrst job was to create the conceptual framework for the piece and to develop detailed plans for its implementation. The initial challenge was to envision a space that was also a story, an embodied narrative world with self-deÞned rules and procedures that expressed the symbolic content of the area, — water, earth, air. Moreover, while the space would seem open for free exploration, it would in fact have to lure the visitors through it in a Þxed sequence of interactions, or else the transformations could not take place. Here, the role of the guide was paramount.

But, Þrst the groups had to deÞne the speciÞc relationship between explorer and guide. The guide, seated at a workstation outside of the space, was to direct and intervene in the explorer’s progress. Given the complexity of the Þnal spaces, it was technically difÞcult to give the guide a precise sense of the explorer’s movements (except by using a camera); creatively, it was challenging to search out new and surprising ways for the guide to ‘talk’ to the explorer. And, to be fair to the visitor who became a guide, we had to Þnd really interesting things for her or him to do, tasks as complex and as engaging as wandering through the actual installation.

The second challenge was the actual construction of the installation. The Villers Experimental Theater is a big, box-like space — 50 x 50 x 60 feet — located at the center of the Wiesner building. The space was a Tabula Rasa — vast, empty, and symmetrical — that demanded large but simpliÞed and dramatic shapes to energize it. Our students were mainly programmers and Þlm-makers who had almost no experience building large-scale walk-through installations. The time available for the building such ambitious structures was very brief — a matter of a few weeks, during which the same students also had to create a multifaceted network that linked all the areas to our computer, lighting, and computing systems.

As it turned out, each group developed its ideas and constructed its area in decidedly different ways: some areas were strong on spatial effects and atmosphere, others on story, others on technical wizardry. Some groups had ingeniously difÞcult puzzles for the guide to solve, others concentrated more on allowing the guide to participate in the explorer’s experiences.

The Areas

The Water Group

To devise a suitably aqueous atmosphere, the group played with methods of creating shifting reßections. After rejecting as impractical the idea of covering the space with water itself, the group designed a hoop covered with mylar, a semi-reßecting material, which was set at the top of the Þshbowl form; a small fan was then used to vibrate the surface, thus creating watery-like reßections on the walls and ßoors below. This, together with some ingenious lighting and a ßoor littered with ‘drowned’ objects made for quite a powerful and eerie feel of underwater space. A seventeen-foot whale (created out of rebar, mesh wire, and painted muslin) with a shocking-pink ßuorescent mouth dominated the area; by entering the mouth, the visitor activated a radar system that made the whale talk and sing.

The guide’s part in all this was to direct the explorer towards each area where a task had to be accomplished. The guide’s computer was linked to a camera which showed the movements of the explorer. In order to direct the explorer through the space, the guide could place image-messages on speciÞc television monitors which surrounded the water environment; these images were inserted interactively into the regular image stream for that monitor. Initially the group hoped to use gestures from habitants of the aquarium — a penguin waving its ßippers to the left, for example; however, for readability, the group Þnally settled on a combination of a child’s hand gesturing with a whisper "come here" or "go left." If the explorer did not succeed within a certain amount of time, the air reserve would run out, and s/he would be Þguratively expelled from the space.

The Earth Group

In contrast to the vertical water piece, the earth space was a long, slightly curving rectangle of scrim and wood, designed to suggest an inÞnitely long featureless landscape. The group’s vision was of a world of objects buried in sand, of ruins under a hot sun, of the remnants of technology littering a wasteland. In keeping with this area’s theme of growth, decay, and renewal, however, new life would eventually spring from these ruins. The explorer and guide had to collaborate to effect this renewal.

The Þrst plan went like this: each abandoned object would contain a hidden camera that, when set off by the guide, would capture different elements of the explorer’s body. These images would then be sent to the guide station where they would be re-arranged into a whole image of the explorer. This image in turn would trigger changes in the environment, restoring the wasteland to its former fruitful state, and the explorer would be freed.

While this design had appealing aspects, particularly the idea of capturing and redisplaying the explorer’s image, the narrative was weak and it was unclear how the explorer could be made to move from one object to another in a predetermined order.

However the idea of a ruin did suggest Shelley’s poem Ozymandias, which in turn suggested a different scenario: the task of the explorer would be to reconstruct this lost kingdom. Instead of capturing and assembling the body of the explorer, the explorer would summon up and connect pieces of the past into a Þnal image of its full splendors.

Here is how it worked: the explorer, by entering the space, triggers a sensor. A voice — seemingly coming from the ruins of the world — recounts the history of the lost kingdom and asks the explorer’s help in restoring the palace. At Þrst, all the explorer can see is a broken column, dimly outlined in dawn light. Drawn to the column, the explorer has to discover how to make a Þlm, hidden within it, play. Once that is accomplished, the lighting brightens and the space widens out as if under a noon sky, revealing a stone archway with a sundial. There another task awaits. Once completed, the lighting changes again, and the explorer spies the last object, a ruined wall and window, silhouetted under the twilight sky.

Each time the explorer successfully executes a task, a part of a large-scale computer image appears in the sky. Piece by piece the ruined palace seems to be rebuilt: Þrst the ßoor, then the walls, then the roof. The moment all the pieces come together, the entire space transforms into a spring-like garden, Þlled with sounds and light.

The guide actively leads the explorer by choosing different Þlms to play in the arches and stones. One set of Þlms shows the past of the world, the other describes its downfall. As the explorer learns how to evoke the Þlms, the circle is complete and the area metamorphosed.

It was particularly difÞcult, in this section, to design interactions that the explorer could easily decipher. For example, we placed four light sensing diodes (later replaced by buttons) and a video monitor on top of the broken column. Our idea was that the explorer had to discover and unlock the secret of the column by moving his or her hand around the edge of the column in a speciÞc direction; as the hand blocked the light from each diode, the explorer was rewarded with a short Þlm element. What we wanted was a smooth unfolding; however, constraints of the technology required that rather than mapping a smooth unrolling of the Þlm with the sweep of the explorer’s hand, we had to break the Þlm into granular elements and offer the reward for each individual interaction. This fragmentation, made the task more difÞcult for the visitors.

The Air Group

In response to the themes suggested by our schematic chart — human intelligence, technology, and risk-taking — the air group envisioned a complex puzzle-Þlled environment — a bar in an airship, a bridge over misty fog, and a cylindrical space in which one would seem to rise up into sky. This space proved unworkable: it was at once too literal and too cluttered, and the scenario provided no clear tasks for the explorer. After a few sessions the group decided to concentrate on creating the airship itself, with a bridge that would lead to the Þre space, envisioned as a video wall in which Þre images rose to the stars. Once the group as a whole focused on the airship, the vision evolved at a steady rate. The ship itself was created by using an electric fan to keep a mylar balloon inflated. The interior that resulted was magically shimmering and translucent, and the colored videos glowed in the air.

The group developed the most intricate scenario of all the spaces. This was the story: the airship was caught in a red nebula; the captain and four crew members were in shock or trance, the ‘red’ state. In order to save the ship and the crew, the explorer had to bring all members of the crew into the ‘blue’ or active state at the same time. The crew members would be visible in videos placed along the sides of the cabin; each crew member would show up on a different monitor. The explorer could change the state of a speciÞc crew member by stepping on a sensor which would be placed in front of the monitor. The trick was that the segment length of blue state varied in length for each character, and therefore their sensors had to be triggered in a speciÞc sequence in order for all the crew members to be simultaneously in the blue state which, in turn, would release the captives. In order to help, the guide needed to solve a puzzle which paralleled this idea of duration on the computer; the guide could then signal to the explorer by turning lights on in front of each monitor in the correct order.

As in the other spaces, signiÞcant frustration and confusion was introduced by the delay between stepping on the hidden sensor and playing the required video clip. Visitors had difÞculty connecting their action with the change in the state of the crew member. This seemed to be aggravated by the fact that visitors were overwhelmed when they were introduced to the narrative before they were able to explore the space. Concurrently we optimized the network, and adjusted the digital audio track which situated the narrative so that it played out in two sections; the second section explained to participants that in order to save the ship all of the characters had to be in the blue state at the same time.

The Visitor’s Experience

On average groups of six people entered the space at a time — three explorers and three guides. This group size had strengths and weaknesses. Social interaction tended to strengthen resolve. For instance we frequently observed that participants would hesitate before entering the whale’s mouth; however, once one member of the group entered and triggered the light which made the pink throat ßuoresce, every one else in the group would enter. In Earth, we found that participants talked through the interaction at both the broken column and at the compass; this shared approach to problem solving helped them understand the nature of the transformation. However, having more than one explorer in the space at the same time also created inconsistencies in the narrative which for the most part had been designed for one person. For instance, when the guide sent a message "Go left" into the water space, the multiple explorers were unsure who the messages was meant for. In air, sometimes participants rushed to stand in front of different monitors which made it more difÞcult to decipher the relationship between the length of the video pieces. At other times the participants moved as a group which tended to be slow and unwieldy.

Some explanation was offered at each guide station although some guide groups immediately understood the Earth interaction. Air offered the most obscure mapping between explorer and guide. The paradigm of a board game, while it provided a real puzzle, was startling in the context. In addition, this interface was clearly designed for a single user. A short piece of audio which the guide listened to on headphones explained the narrative — all the logs needed to be launched in the river in order to provide a bridge which the monkey could use to cross to the other side. Because only one person could hear the instructions and because the game moved quickly, the two other partners in the guide group had difÞculty participating.

Our achievement and lack thereof can be evaluated in light of visitor feedback. For the visitor, the experience was one of learning and understanding — learning the rules of the experience Þrst, and then perhaps contemplating why the experience mattered. The simpler the introduction of the activity and clearer the direction through the space, the more quickly visitors grasped what to do. Children in particular truly enjoyed the magical way in which each of the spaces talked to them. While most adults grasped the activities at the guide stations fairly quickly, the experience of being an explorer tended to cause anxiety.

From a research stand-point, the fact that visitors insisted on playing both roles — explorer and guide — allowed the visitors to reßect deeply on their experience. Over and over, we discovered that visitors gained situated knowledge which included the schema of each space when they took on their Þrst role of either explorer or guide; this knowledge made them more relaxed when they participated in the second role. The explorer role in general was more bafßing to those participants who had not Þrst served as guides. One participant who started as an guide stated "You don’t know how interactive you have to be. You think you have a mission." When participants who started as explorers with a mission could not discover how to control the environment, they often became completely bafßed and virtually gave up until some outside observer offered a whispered hint. In some cases these reactions occurred in relation to a speciÞc interaction and the visitor was able to work through to a new level of understanding. For instance, one visitor described feeling a little silly walking into the mouth of the whale, but then reßected that "it is only when you understand the whale as a character, that you understand that the whale is talking to you and think about how to talk back."

Evaluation

At the outset we sensed that we were on the brink of a long and fascinating journey but we found we had underestimated the complexity of our task: as we reÞned and reiterated our conceptual design we were forced to continually adjust our software designs and modes of construction. When we opened the installation it became clear that we needed another substantial period of reÞnement before the installation would be truly hospitable for visitors. Some of the problems were technical, for we had little or no chance to test and correct the installation. We also realized that as our designs grew increasingly complex, we lacked strong graphical scripting tools which would enable us to pre visualize the impact of a new idea on the complex environment.

Other problems arose from the visitor’s reactions to an unfamiliar experience. With so many different kinds of people moving through the space, we discovered a whole range of problems in providing a uniÞed way of orienting them to the experience and teaching them how to participate.

Several limitations were clear: we had relied heavily on puzzle-solving as a way to engage the visitors and have them interact with the space. However, many visitors, especially older ones, were resistant to this mode of game playing; they wanted assurances of how to act and what to do. Moreover, puzzles require an intellectual focus on the part of the visitor which interferes with a reverie-like absorption of the experience. Can we Þnd new narrative forms which invite intervention without depending on puzzle-solving?

In general, we found that creating and communicating a role for the participant was both the most exciting and difÞcult task we encountered. In life we change roles continually without even noticing we are doing so, but when we are enter a fabricated situation which requires on-demand role playing, we often react with fear or embarrassment. The interactive environment needs to engage the visitor and elicit a spontaneous desire to play along with the game. When the environment or the kind of game is familiar from some other experience, the task seems easier. For example, the situation in the Airship seemed familiar to most participants, relating to their memories of Star Trek. Because they grasped the basic situation very quickly, explorers could concentrate on the task at hand: how to change the state of the characters in the video. In fact, as one of the authors commented, participants for the most part ignored the text of the video — what the characters were saying — as soon as they realized that it would not help them solve the puzzle. Visitors were very good at discovering the truly decisive elements in a puzzle.

The tasks facing the guides were more subtle and more varied, and visitors had correspondingly more trouble with them. Each guide station offered a different model of intervention. For instance, in Water the guide tried to direct the movements of the explorer using a limited set of icons. Detailed instruction was not allowed. This put quite a strain on both explorer and guide, for they had to interpret a new code or language in order to grasp the direction. In Earth, the information was more direct. Most participants understood that when they heard audio that said, "It is the east, and Juliet is the sun," it meant they must stand on the eastern point of the compass. What was not explicit was the requirement that the participant do so only after the guide made the request. As soon as the explorers Þgured out that the text might refer to the points of the compass, they rushed eagerly ahead without waiting for further instructions and thus upset the relationship of guide as leader we had planned into the interaction.

The interface for the guide station in the air section was playful but obscure. Each log the guide launched had a speciÞc length which was relational to the length of the video clip for the associated character in the airship. Because the guides did not know what was happening in the space, they had to understand their own environment before they could really help the explorer. While most visitors had fun launching the logs in sequence, they had difÞculty translating what they learned to running the lights. The lack of audio separation between the spaces added a serendipity feature to the guide’s experience, sometimes adding clarity and sometimes not. In general, we found we had to adjust these interfaces more carefully to the range of capabilities and expectations of the visitors.

Conclusion

While many aspects of the technology and narrative could be improved, the project as a whole was inspiring. A design process based on collaboration produced an extremely rich product, for the individual groups of collaborators focused on concepts congenial and stimulating for them. Variety and surprise resulted; and the richness of the narrative context encourages us to look anew at collaboration as an element in global storytelling environments.

Wheel of Life also suggested a new kind of interface for interactive applications, one that changes and evolves with the actions of the user. This dynamic environment allows the user to learn a set of responses of some complexity through actions in time, easing the pressure on the user and allowing for more subtle and more reÞned kinds of interactions.

Because our narratives were spatial, new kinds of story-telling became possible. In particular, this work united the fable with theater and game-playing. The merger of different genres of narrative echoes in some way the rich merging of media that characterizes the interactive medium as a whole, and creates the challenge of inventing appropriate types of narration.

Putting the technology to new use also opens up new possibilities and challenges for the technology. How can we develop more persuasive wrap-around media that truly create the environment they describe, and how can we create sensitive and speedy devices that which respond to complex interaction? Solutions to these and other problems will open up the development of interactive transformational environments to uses and sites we have yet to imagine, and could possibly revolutionize our relationship with the technology.

Acknowledgments

Principle Authors: The piece was co-directed by Professors Davenport and Friedlander; Herrick Goldman designed the lighting; Stuart Cody acted as technical supervisor; Greg Tucker was facilities coordinator; David Tamés was production manager. The administrative assistant was Betsy Brown.

Student authors include: Alan Blount, Kevin Brooks, Cris Dolan, Scott Higgins, Gilberte Houbart, David Kung, Mitchell C.H.Liu, Wendy MacNeil, Lee Morgenroth, Mike Murtaugh, Gene Rhough, Lucy Tancredi, John Taylor, Wendy Saunders, James Jung-Hoon Seo, Cyrus Shaoul, Carleton Sparrell and Koichi Yamagata.

Thanks to Zane Vella who designed the entryway animation, Chris Scovell who provided carpentry assistance, and Alan Symonds who lent us the Melange lighting controller. Finally, thanks to Tom Wong, Maureen Costello, and Andy Lippman who provided us special support when we needed it.

The Wheel of Life was sponsored by: at MIT, the Media Laboratory, the Media Arts and Sciences Section, the OfÞce of the Arts, the Department of Literature, the Department of Theatre Arts, the Media Lab Movies of the Future Consortium, the de Flores Fund, and outside MIT, Bontronics Corporation, Sharp Corporation, Apple Computer, Radiant Technologies, Interactive Media Technologies as well as a number of individual donors.

Photos and line drawings by David Tamés

Appendix 1: About the Materials and Technology

How were the materials and technology selected and brought together for the Wheel of Life? As is the case in most artistic endeavors, we acted as expeditiously as possible while keeping an eye on our at Þrst non-existent budget. The vastness of the Villers Experimental Theatre invites scale; the nature of the experience, moving through worlds, required dramatic theatrical lighting; the nature of the project required an interactive network be put in place for each space.

As technical coordinator, Stuart Cody shared his knowledge of off the shelf house building materials. In Water, PVC piping was used to shape the Þshbowl; bent rebar was used to create a solid skeleton for the whale’s muslin skin. In Earth, lolly columns were used for the broken column and arch; sifted sand was spread out on the ßoor. The most difÞcult structure to build was the wood frame which shaped the scrim around Earth, in part because it had to be hung from non existent points in the ceiling. Alan Blout built the air space in one night using mylar which Tom Wong generously donated and a fan.

Early on in the semester we discovered Herrick Goldman, a talented and experienced theatrical lighting designer, who was working part time in the Theater Arts Department. After his initial introduction, Herrick worked closely with each group to generate a lighting concept which dramatized the space, forwarded the action and which was well integrated in the interactive script. Hanging the lighting grid was the Þrst act of the installation process. The lighting was programmed using a Melange Memory Lighting Controller.

Meanwhile we assembled the necessary hardware to complete the installation. An Apple Macintosh IIfx served as the central software node for each space. The monitor displayed the guide station interface. David Tamés designed a LAN consisting of Ethernet, a Shiva FastPath and Radiant Technology’s CNX Common Sense interface boxes connected via PhoneNet to distribute messaging to sequenced sensors for each space and between spaces.

In order to trigger lighting cues, the three guide stations communicated via AppleEvents to a Macintosh IIfx server that in turn controlled the Melange Lighting Controller via MIDI. This Macintosh also contolled via MIDI an Akai PG1000 audio/video switcher that switched the video from several Sony Vdecks in the Water space. An Interactive Media Technology audio/video controller provided distribution and switching between the two videodisc players in the Earth space. The laserdiscs in Air were controlled by the guide station using a four-port serial card from Greenspring Computers.

Each space was equipped with NTSC video monitors, video projectors and sound systems as appropriate. The floor plan and block diagram (following the appendices and notes) will provide tech-know-bots with more detail about this aspect of the environment.