I was unfortunately unable to attend BayCHI this week when Peter, Tim and Larry spoke on designing systems with emergent behavior. Lucky for me, Peter posted some of his thoughts and takeaways from the evening.
A couple of things Peter wrote struck a nerve:
Peter first says:
“There are no good user research methods of emergent behavior
The standard tools that a user researcher has tend to focus on an individual’s interaction with the system. Or, if we do engage with groups, we’re limited to the scope of groups we can interact with. So, how can user research inform the design of systems with emergent behavior? We need new methods and approaches that allow us to work with a crowd.”
Peter also goes on to say:
“Embrace the chaos
…I’m surprised that most companies don’t take the Amazon.com approach of testing designs with small percentages of the users and gauging behavior before unleashing it on the world at large.”
Companies do embrace the chaos.
I don’t know if most companies do it, but I know that all the big companies with massive user bases do it. At Yahoo and elsewhere, it’s called bucket testing. Based on any number of factors, including but not limited to revenue risk, research goals and time/budget constraints, a new rev of the product will be launched to a tiny percentage of the population. So when you have 400,000,000 users and you let 5% of them see the new product, you get a rich and reliable data set, as well as a wealth of information on how the product will be used, and how it will perform when you do open the flood gates 100%.
Bucket testing is constantly happening at Yahoo, Google, MSN, Amazon and others. Have you ever seen a new feature for a week or two, and then it vanishes into thin air? You were in a bucket test.
A Strong Statement
I know I’m going to get heat for stating that bucket testing is a user research tool, but I’m going to say it anyway: Bucket testing is a great tool for designing emergent systems. It is not the typical lab-based, or field-based “user research” that we all think of, especially because it happens after the system is designed. However, because of the limitations of human beings’ abilities to predict (unless there are clairvoyant UE people out there) what other people are going to do, bucket testing is the best tool to date that I’ve seen for informing my design decisions in SWEBs (Systems with Emergent Behavior).
As designing emergent systems become increasingly complex, bucket testing is possibly the best user research tool available; something that no amount of up front research can provide, and something that no design firm or user experience strategy firm can account for unless they are embedded in your company for a long-term relationship. I hated to say that last sentence given my consulting history, but it seems to be more and more often the case. It’s time for consultancies to innovate on how to provide this service in a long-term valuable way.
A confession
When I was first exposed to bucket testing, I hated the idea; I was also from a user experience consulting background. I loathed putting my hard work and Fantastic Design Decisions in front of a small piece of the population, just to observe and then tweak and change my minor failures in near-real time. But over time, I have actually learned to embrace The Bucket Test Methodology as a key tool in researching how users respond to the emergent systems I design, and making those systems or products better.
The other tool I should mention, which only few products and companies are lucky to have, is the loud-mouthed users themselves. If you design a product/system that people are passionate about, they will constantly give you, ahem, advice on how the system should emerge to meet their needs. For every five you hear from there are 50,000 that feel the same way. In this case it is your job to be the bullshit filter and improve your product accordingly.
Not the be all end all choice
Bucket testing isn’t The Best Way, or the only way to gain insight into a SWEB. In fact, in many cases, such as games, I don’t see how it has a ton of applicability. However, for large content-driven and transaction-driven and user-driven and recommendation-driven and algorithm-driven online ecosystems, bucket testing might well be one of the best tools at our disposal.
The key is being sure to make bucket testing not just a Product Management-driven exercise; involve User Experience folks from the get go. Otherwise, your bucket tests will be too heavily focused on numbers, and not on evolving based on behavior.
Oh, one last comment on Peter’s post. Many contemporary systems do seem to live in perpetual beta, and I think it’s fantastic. It is a clear philosophical shift from the engineering/shrink-wrapped wait-to-pay-for-the-next-release software model of Yore, and evidence that companies are trying harder to evolve their products at a rate which keeps up with our behavior. And I welcome that. It forces designers and engineers to work more closely; that historical separation are why products have sucked for so long.
[…] Is Bucket Testing a good research tool for emergent behavior? If so, what’s with the weird name? […]
October 13th, 2006 at 3:21 pmWhich weird name? Bucket testing or emergent behavior?
October 13th, 2006 at 3:31 pmIt does sound as if Peter was contradicting himself, but I think overall he was addressing how emergent behavior as a concept transcends what we can learn from existing approaches. Bucket testing only allows you to test/gauge real-time behavior against what is in place. It won’t return as much value in terms of how behavior will evolve. So in a sense, you are limited by the current state - success or failure - of the current design. That said, bucket testing does have great value; I believe the Amazon example was meant to highlight that bucket-testing as a lower cost and data-rich method should be taken advantage of by more companies.
In general, the panel at various points in time tended to refer to product design and traditional social design fields such as planning and architecture. As a trained Landscape Architect, I believe very strongly that today’s internet/technology based designers do not think enough about design as it applies to a person’s environment – physical and social. The value of thinking broader is increased awareness of ALL that shapes behavior at an individual, group and system level.
Let’s be honest, emergent behavior is not new nor are many of the concepts that pop up in our industry (e.g. social networking, micro-formats). I want to believe that these new buzz words will prompt a new desire for designers to look aggressively beyond their own narrow domain and think about design as a big picture.
As we increase our design awareness we will need to evolve our approach. Specifically, ethnography will take on a whole new meaning – static home visits won’t be enough. I think the next few years in internet design will be challenging for designers – it will require us to be humbled and to recognize that our current myopic view of design is not the entire reality.
October 18th, 2006 at 9:22 amI don’t think Peter was contradicting himself at all; I think he’s wrestling with the same thing we all are: how do we develop emperical approaches (and best practices) to the new challenges we face in designing systems that we don’t have control over? It is even harder for design firms, because they generally have a set of deliverables that need to have lasting value, and can’t be easily tweaked like they can with an inside team. That is a very tough nut to crack.
I’m really excited to see what happens at the upcoming IDEA conference. It is truly the first of its kind; I can only imagine that as physical space, people and technology collide more and more, these types of problems will only get more interesting, and more difficult. All of these disciplines can stand to learn from each other.
You stated, “I believe very strongly that today’s internet/technology based designers do not think enough about design as it applies to a person’s environment.”
I agree. And the reason is that until now, we haven’t had to. Technology/bandwidth/mobility didn’t allow for it.
What a cool place to be right now!
October 18th, 2006 at 9:11 pmG -
In the quotes you give from Peter, it appears he contradicts himself, but in his post, where you see his train of thought organized on the page, his contradiction isn’t so apparent.
There are two types of research at play here. The initial up front research to categorize or place the problem and define the solution (i.e. the problem definition), then there’s, what we at IDEO might call, the prototyping stage, where we learn even more from the solution in existance and in the hands of the ‘user’. This leads us to being able to determine the problem solution.
I think you said as much in: “The Bucket Test Methodology as a key tool in researching how users respond to the emergent systems”. And so my thought was that wouldn’t prototyping + iteration be the best method of research activity at this stage? Which I imagine is simply Bucket Testing as you describe.
…something that no amount of up front research can provide, and something that no design firm or user experience strategy firm can account for unless they are embedded in your company for a long-term relationship.
Though, I do (as you predicted) have a problem with your suggestion that up-front research is ever mistaken as the same as prototyping the solution in the hands of the users. I don’t see up-front research trying to compete with Bucket Testing, in fact is really something quite different.
In your comment you explain it better that as a design firm (I might as well say it:) ‘our’ mode of thinking will be in a different context from that of our clients. We think in terms of solving problems, or making something new and improved and our clients tend to think in terms of marketing: with contingency and necessity. So as a design thinker inside a client organization, formerly within a consultancy organization, I wonder how you’ve changed or evolved.
I think consulting can help clients by providing the analysis and insights to tackling complex problems with emergent behaviours, and then planning for and helping to manage the prototyping or testing of the developed solutions. I believe we do this on a near-regular basis.
Assuming that not all consultancies are equal, I think some successful ones realize the value in teaching their clients about the design process as well as developing one specific to their client’s business and environment. New types of research and prototyping methods included. But to put it back on to the client, in this context, much of the time the client also has to be looking for this type of service. Not many are.
My second thought, while reading this, was of Tom Moran’s plenary from DIS2002 on Everyday Adaptive Design (which Dan Hill documented very well here). This doesn’t speak to the research method, but more to the need to consider what does it mean to actually design for adaptive design because of emergent behaviour. And as you mention, stuff stays in Beta for a long time these days, shrink wrapped software is becoming a thing of the past and hackability is not just limited to software, but to cars and all sorts of products.
I’m disappointed to being absent from IDEA - but Gino will definitely document it well for us.
October 21st, 2006 at 4:37 pmSteve Portigal posted a lengthy report on the BayCHI meeting on Core77 here. Thanks for doing so Steve.
October 23rd, 2006 at 8:37 pmGino, interesting defense of bucket testing but to put the method to the test — so to speak — can you identify any innovate or clearly superior products that have been designed using bucket testing?
In my experience it was a fool’s errand that was perhaps useful in optimizing some impulsive, synaptic user twitch but was utterly useless in providing any meaningful understanding of user motivation or desire.
But maybe I just had a bad run of it.
Still, my original question remains: can you name any innovative products or services that were developed with this method?
December 4th, 2006 at 5:47 pm