I was unfortunately unable to attend BayCHI this week when Peter, Tim and Larry spoke on designing systems with emergent behavior. Lucky for me, Peter posted some of his thoughts and takeaways from the evening.
A couple of things Peter wrote struck a nerve:
Peter first says:
“There are no good user research methods of emergent behavior
The standard tools that a user researcher has tend to focus on an individual’s interaction with the system. Or, if we do engage with groups, we’re limited to the scope of groups we can interact with. So, how can user research inform the design of systems with emergent behavior? We need new methods and approaches that allow us to work with a crowd.”
Peter also goes on to say:
“Embrace the chaos
…I’m surprised that most companies don’t take the Amazon.com approach of testing designs with small percentages of the users and gauging behavior before unleashing it on the world at large.”
Companies do embrace the chaos.
I don’t know if most companies do it, but I know that all the big companies with massive user bases do it. At Yahoo and elsewhere, it’s called bucket testing. Based on any number of factors, including but not limited to revenue risk, research goals and time/budget constraints, a new rev of the product will be launched to a tiny percentage of the population. So when you have 400,000,000 users and you let 5% of them see the new product, you get a rich and reliable data set, as well as a wealth of information on how the product will be used, and how it will perform when you do open the flood gates 100%.
Bucket testing is constantly happening at Yahoo, Google, MSN, Amazon and others. Have you ever seen a new feature for a week or two, and then it vanishes into thin air? You were in a bucket test.
A Strong Statement
I know I’m going to get heat for stating that bucket testing is a user research tool, but I’m going to say it anyway: Bucket testing is a great tool for designing emergent systems. It is not the typical lab-based, or field-based “user research” that we all think of, especially because it happens after the system is designed. However, because of the limitations of human beings’ abilities to predict (unless there are clairvoyant UE people out there) what other people are going to do, bucket testing is the best tool to date that I’ve seen for informing my design decisions in SWEBs (Systems with Emergent Behavior).
As designing emergent systems become increasingly complex, bucket testing is possibly the best user research tool available; something that no amount of up front research can provide, and something that no design firm or user experience strategy firm can account for unless they are embedded in your company for a long-term relationship. I hated to say that last sentence given my consulting history, but it seems to be more and more often the case. It’s time for consultancies to innovate on how to provide this service in a long-term valuable way.
A confession
When I was first exposed to bucket testing, I hated the idea; I was also from a user experience consulting background. I loathed putting my hard work and Fantastic Design Decisions in front of a small piece of the population, just to observe and then tweak and change my minor failures in near-real time. But over time, I have actually learned to embrace The Bucket Test Methodology as a key tool in researching how users respond to the emergent systems I design, and making those systems or products better.
The other tool I should mention, which only few products and companies are lucky to have, is the loud-mouthed users themselves. If you design a product/system that people are passionate about, they will constantly give you, ahem, advice on how the system should emerge to meet their needs. For every five you hear from there are 50,000 that feel the same way. In this case it is your job to be the bullshit filter and improve your product accordingly.
Not the be all end all choice
Bucket testing isn’t The Best Way, or the only way to gain insight into a SWEB. In fact, in many cases, such as games, I don’t see how it has a ton of applicability. However, for large content-driven and transaction-driven and user-driven and recommendation-driven and algorithm-driven online ecosystems, bucket testing might well be one of the best tools at our disposal.
The key is being sure to make bucket testing not just a Product Management-driven exercise; involve User Experience folks from the get go. Otherwise, your bucket tests will be too heavily focused on numbers, and not on evolving based on behavior.
Oh, one last comment on Peter’s post. Many contemporary systems do seem to live in perpetual beta, and I think it’s fantastic. It is a clear philosophical shift from the engineering/shrink-wrapped wait-to-pay-for-the-next-release software model of Yore, and evidence that companies are trying harder to evolve their products at a rate which keeps up with our behavior. And I welcome that. It forces designers and engineers to work more closely; that historical separation are why products have sucked for so long.