By Scott Berkun, May 2001

Imagine what you’d learn if you could be a fly on the wall as teams of great designers tried to solve a tough problem. Well imagine no more. We brought together four professional teams, live on stage, and in turns, let a live audience witness them try to solve a real problem. 

Most of the time we learn about how design is really done from essays or talks, where people describe what they think happened, but the real experience can be very different. Of course putting people live on stage creates new challenges for designers, it’s not how design is usually done either, but this was an experiment to see what we might learn. The hope was that by watching different teams approach the same problem, the audience would observe many different approaches and have new questions, questions they’d never think of without participating in a format like this. 

We did the first event live in in front of an audience of 800 people at CHI 2000.  Our organizing team was myself, Chris Konrad, Debbie Cargile, and Sarah Zuberec. We cleverly made this event work in 90 minutes, under the guise of a panel format at CHI. Kudos to Jean Scholtz and David Gilmore, and all the CHI folks that trusted us to put this together.

This page describes the event, and gives some information about how we put it together. We did a reprise of the Interactionary format at CHI 2001 (see the summary, guidebook or ACM paper here).

Interactionary 2000

Interactionary is a game show type format that allowed 4 teams to work on the same design problem, live on stage. Each team worked one at a time, and were given ten minutes to work through the problem. The goal is to expose the dynamic intangibles of design in progress, and allow an audience to listen in on four teams and observing how they work. The challenge was two fold: To make the session fun without being silly, and to make the design process, under stress, visible to an audience. Waiting teams were placed in a separator sound proof room to eliminate any order advantage.

Bruce Tognazzini, Steve Rodgers, and Susan Dray were our expert panelists, and they provided commentary and judging for the event. The panelists were on stage, across from the whiteboards that the teams used when competing The four teams were from Sapient, IBM, Razorfish, and the University of Malmo. Every team member wore a live microphone, an the audience and the panelists were able to hear their conversations, and view their work as they did it.

The education value of the experience was in allowing the audience to watch four different teams of designers approaching the same problem. Viewers could compare and contrast the different techniques and approaches used as they watched different teams. The hope was that we could capture some of the intangibles of brainstorming and designing that are difficult to capture in conference papers and proceedings. Even though this was sport, and not engineering, we felt this simulated enough of the dynamics to be worthwhile.

Scoring and Judging

We used a system of three categories for scoring: teamwork, process, and final design. Each category was worth ten points, and each panelist scored every team. Teamwork considered how well the team worked together, and how they organized their time and approach. Process considered the “methodology” they used for evaluating the problem, using the whiteboards, and evaluating design alternatives. The final design considered the usability, feasibility and quality for their final solution.

After each team was announced, and was ready on stage, we read the problem out loud to them and the audience. Then we started the clock, and gave the team two warnings, at five minutes and one minute left. We recommended that teams use their last minute to describe to the judges their final design idea.

The Problem

We created a pool of potential design problems to use, pulling from a wide set of different kinds of interaction design (web, software, architecture, industrial design, etc.). Many problems came from interview questions that I use when interviewing designers or UI project managers (it’s something of a hobby for me to invent design problems). The challenge was to find problems that were broad enough that they did not require much explanation, while simultaneously being challenging and tractable in only ten minutes. Here is the problem used at CHI 2000:

You are to design an airport food vending machine for the year 2005. The machine will dispense a variety of prepackaged food items including candy bars, potato chips, as well as three freshly brewed hot beverages (tea, hot chocolate, and hot coffee) that are created on demand. The specific problem you need to solve is how users purchase and select the item(s) they want to buy.

The action

The problem with Interactionary is that it’s impossible to document the essence of what took place – this is precisely why we wanted to do it in the first place. Each team took wildly different approaches to both organizing their time and approaching the problem. Some teams such as IBM and Sapient, used the first 5 minutes to divide up certain tasks, such as doing a needs analysis or developing potential approaches to the problem. Other teams such as Razorfish used a group approach, appointing one person to run the whiteboard, while the others discussed and communicated ideas. The University of Malmö divided the whiteboard space into predefined sections, which helped them to organize their time and communicate their final ideas.

Some teams took advantage of the CHI audience, and solicited commentary and informal survey information about how vending machines were used.

The results

The final results were tabulated live on stage, while our M.C. Chris Konrad entertained the audience. We initially reported the wrong scores, due to human error in the tabulation process (First rule of working with Scott: when in doubt, blame Scott). The final results are listed below:

1. Sapient
2. IBM
3. University of Malmo
4. 4. Razorfish

The teams, players and judges

This event required great courage from all of these folks. They all approached this with a professionalism and sense of humor that made it really fun to put together. My thanks go to them.

Sapient	IBM (aka “Big blue brothers”)	University of Malmo	Razorfish
*Isabel Ancona David Zienowitz Alder Yarrow John Payne	Pat Cox John Karat *Alex Little Jack Scanlon	*Zayera Khan Isa Hardeuro Petter Karlsson Edison Gomez Lattes	Elizabeth Hare Vincent Santo Karin Dauch Amy Bassin *Neil Wehrle

* denotes team captains Judges:

Bruce Tognazzini , Nielsen-Norman Consulting
Susan Dray , Dray Associates
Steve Rodgers, Razorfish (Phillips design at time of interactionary)

After hours voting

We handed out questionnaires to the 500+ CHI attendees in the auditorium to collect feedback on the event. We also took the opportunity to give the audience a chance to do their own scoring of the teams to see how closely it would match our panelists. As it turns out, audience scores were comparable to the judges. Here are the averages from the audience scores.

Audience Scoring:
Sapient : 27.6
IBM : 23.8
Razorfish: 21.54
University of Malmo: 21.51

Questionnaire:
Almost 300 audience questionnaires were filled out and returned to us after the event at CHI 2000. All questions were based on a 7 point Likert scales, 7 being the most positive response.

Question	Average Response (1-7)
Was the time limit per team the right amount of time to witness design work without getting bored?	6.5
Was watching other designers work in this format educational or informative?	6.18
Was this entertaining or enjoyable to watch?	6.7
Was this a good use of your time?	6.36

More information

We have more pictures from the CHI 2000 event. Some are large, and the descriptions are sparse. View at your own risk. Thanks to Douglas Pyle for all of these photos.

We performed a second interactionary event at CHI 2001. Different teams, different judges. Modified format. Trumpets included.

I’m planning on writing a paper about the techniques used by the different teams in Interactionary, including some recommendations for how to do well at this sort of thing. The techniques are useful for brainstorming or design review meetings.

As long as we find venues for doing these sorts of events, we’ll continue to do them, and I’ll continue to post reports on them here. If you have an idea for me, or want us to put this on somewhere, Contact me.