I was unfortunately unable to attend BayCHI this week when Peter, Tim and Larry spoke on designing systems with emergent behavior. Lucky for me, Peter posted some of his thoughts and takeaways from the evening.
A couple of things Peter wrote struck a nerve:
Peter first says:
“There are no good user research methods of emergent behavior
The standard tools that a user researcher has tend to focus on an individual’s interaction with the system. Or, if we do engage with groups, we’re limited to the scope of groups we can interact with. So, how can user research inform the design of systems with emergent behavior? We need new methods and approaches that allow us to work with a crowd.”
Peter also goes on to say:
“Embrace the chaos
…I’m surprised that most companies don’t take the Amazon.com approach of testing designs with small percentages of the users and gauging behavior before unleashing it on the world at large.”
Companies do embrace the chaos.
I don’t know if most companies do it, but I know that all the big companies with massive user bases do it. At Yahoo and elsewhere, it’s called bucket testing. Based on any number of factors, including but not limited to revenue risk, research goals and time/budget constraints, a new rev of the product will be launched to a tiny percentage of the population. So when you have 400,000,000 users and you let 5% of them see the new product, you get a rich and reliable data set, as well as a wealth of information on how the product will be used, and how it will perform when you do open the flood gates 100%.
Bucket testing is constantly happening at Yahoo, Google, MSN, Amazon and others. Have you ever seen a new feature for a week or two, and then it vanishes into thin air? You were in a bucket test.
A Strong Statement
I know I’m going to get heat for stating that bucket testing is a user research tool, but I’m going to say it anyway: Bucket testing is a great tool for designing emergent systems. It is not the typical lab-based, or field-based “user research” that we all think of, especially because it happens after the system is designed. However, because of the limitations of human beings’ abilities to predict (unless there are clairvoyant UE people out there) what other people are going to do, bucket testing is the best tool to date that I’ve seen for informing my design decisions in SWEBs (Systems with Emergent Behavior).
As designing emergent systems become increasingly complex, bucket testing is possibly the best user research tool available; something that no amount of up front research can provide, and something that no design firm or user experience strategy firm can account for unless they are embedded in your company for a long-term relationship. I hated to say that last sentence given my consulting history, but it seems to be more and more often the case. It’s time for consultancies to innovate on how to provide this service in a long-term valuable way.
A confession
When I was first exposed to bucket testing, I hated the idea; I was also from a user experience consulting background. I loathed putting my hard work and Fantastic Design Decisions in front of a small piece of the population, just to observe and then tweak and change my minor failures in near-real time. But over time, I have actually learned to embrace The Bucket Test Methodology as a key tool in researching how users respond to the emergent systems I design, and making those systems or products better.
The other tool I should mention, which only few products and companies are lucky to have, is the loud-mouthed users themselves. If you design a product/system that people are passionate about, they will constantly give you, ahem, advice on how the system should emerge to meet their needs. For every five you hear from there are 50,000 that feel the same way. In this case it is your job to be the bullshit filter and improve your product accordingly.
Not the be all end all choice
Bucket testing isn’t The Best Way, or the only way to gain insight into a SWEB. In fact, in many cases, such as games, I don’t see how it has a ton of applicability. However, for large content-driven and transaction-driven and user-driven and recommendation-driven and algorithm-driven online ecosystems, bucket testing might well be one of the best tools at our disposal.
The key is being sure to make bucket testing not just a Product Management-driven exercise; involve User Experience folks from the get go. Otherwise, your bucket tests will be too heavily focused on numbers, and not on evolving based on behavior.
Oh, one last comment on Peter’s post. Many contemporary systems do seem to live in perpetual beta, and I think it’s fantastic. It is a clear philosophical shift from the engineering/shrink-wrapped wait-to-pay-for-the-next-release software model of Yore, and evidence that companies are trying harder to evolve their products at a rate which keeps up with our behavior. And I welcome that. It forces designers and engineers to work more closely; that historical separation are why products have sucked for so long.
I came across this post on reveries magazine today that made me stop to think a little. Perhaps not a very coherent or thorough thought, but I’m sharing it nonetheless.
Today’s Cool News of the Day post titled: Wow v. Oh, describes how google engineers set about trying to come up with things that are new and different, that makes people say “wow”. This, I have to admit, has happened to me using some of google’s apps. Whereas, Yahoo tries to create new features within the context of its existing offerings, trying not to create ‘orphans’ of new applications.
The post describes how both companies are attempting to become more like each other, one trying to get engineers to develop integrated add-ons, and the other trying to emulate a fast-moving, innovative company.
CCTV Headquarters by OMA, Koolhaas
This made me think of OMA’s new headquarters for Central Chinese Television, which to me looks “wow” from the outside – but concerns me that it might not carry through that throughout the inside of the building. Inside it might be more of an ‘oh, okay’ kind of response. Google’s wow-ness in it’s applications definitely is short lived, when I find that much of the initial experience is about all there is to see. Not to be horrible, but it’s just that there’s not a lot more to the stand-alone app, like Calendar, than meets the eye. For me, Google and all the apps, feels a little like Adobe illustrator and Photoshop before ’93 – they might as well have been completely separate companies.
Whereas Yahoo’s chess game seems to reveal that it has all of it’s pieces moving in sync with each other. Now I don’t work for Yahoo – and this really isn’t a Yahoo-appreciation blog – but the Mail beta app continued to wow me for some time. I constantly discovered features that delighted me and worked the way I expected. Unfortunately the ‘wow this is slow’ took over and I moved on.
Now all this wraps up with a lingering thought that I had while reading this, and which won’t leave me since reading a preview copy of Dan Saffer’s new book: Are engineers designers?
I considered an observed difference between Yahoo + Google, the so-called Pepsi & Coke of our new media space, which is that Google seems to celebrate it’s engineers who invent and develop the new applications, whereas Yahoo celebrates (or buys) it’s designers and teams who invent and build their products.
So I wonder who has the better chance of sustaining a ‘wow’ both on the inside and out, and who has more work to do in becoming that? I know Google has designers, I’ve met a couple, but I know of a lot more designers at Yahoo and wonder how much the DNA of a team at either of the companies has to do with their product innovation and design?
In other news – we’re going to be posting our review of Dan Saffer’s new book: Designing for Interactions: Creating Smart Applications and Clever Devices, that was recently published by New Riders. Hopefully within the next few days, and I’ll be sure to expand on what it was he said that inspired this post.
It is no secret that Yahoos are involved with regularly occurring Hack Days. When I first started at the company, I participated in a Y! Travel Hack Day - yes, we designers (and all team members) are just as involved in designing the hacks as the engineers that code them. The hack that came from my first Hack Day endeavor was a Yahoo! Widget (a.k.a. Konfabulator widget) for FareChase.
My team of three spent about three hours designing it, four or five hours coding it, and that was that… we thought. Not bad for a day’s work.
The hack was passed around to various people in the company, and it quickly snowballed into a request for a real, living breathing widget. So after six months of politicking, red tape, tweaking, and backend engineering, Yahoo! released the FareChase widget Version 1.0.1a without much fanfare on July 8. We have had a couple thousand downloads at this point, and would love your feedback!
What does it do? The Widget finds the lowest air prices from dozens of travel sites for you and brings them right to your desktop. Hey, we are a lazy bunch, and prefer things to come to us - and now you can benefit from our laziness as well!
Today, Yahoo! Travel brought Trip Planner out of beta. There has been a ton of press coverage today, which is quite exciting. To see folks write about my work in Forbes, and in many national and international newspapers, online news outlets, and in the blogosphere (the most important outlet!) simply stokes my fire.
Some people would say that Y! has bet the farm on social media, and I say that it is an exciting place to be right now. Greg Sterling ponders today, “in many ways it’s the most impressive expression to date of Yahoo!’s social media strategy. Yahoo! Answers has received a great deal of attention recently (there’s an Answers integration with Travel) but the new Trip Planner is more fully realized as a product.” I think Greg hits the nail on the head, and there is only more smart, focused integration of Yahoo properties (in Travel and elsewhere) on the way. If the press and PR folks are missing some points, I would have to say that Trip Planner is focused on SIMPLICITY, and very concise, progressive disclosure of functionality. It doesn’t overwhelm, but hopefully it is helpful in all the right places. We also took great pains toward personalizing the experience based on other actions you have taken within Yahoo Travel and elsewhere - but that’s a hard topic and concept for the PR engine to spin, so you’ll have to read it here. 
While much of the trip creation process of Trip Planner existed when I started at Yahoo back in December, I led the design for many of the new features in Trip Planner, including the ability to explore geo-coded trips across the world. Rather than go into detail describing all of the features and new-skool Yahoo! coolness, I’ll just point you to the Trip Planner home page; go plan a trip!
To the Trip Planner team, I’m proud of the work we’ve done, and can’t wait to get started on Trip Planner 2.0. The Trip Planner engineers are amazing, and thanks to my fellow designers for all of the great feedback, idea generation, and help in bringing focus to the product (and the sweet visual design). There is so much to do still, and I look forward to seeing where we can take it.
Update: Be sure to check out Ivanka Trump’s Trip Plan to Dubai!
It’s no secret: I am a bicycling nut. And of late, my tastes have turned very much toward the classic, bird-like lugged steel frames of a time gone by. I guess my techno-centric work life pushes me toward a personal life of dirt, twine, shellac, and other things that can not be had from the confines of a digital device.
There is a fellow named Chris Kulczycki, who via his Velo Orange blog, has begun to build a classic cyclo-touring business by interacting directly with his customers, and potential customers. The recent post that really struck me as interesting - and ballsy - was this one. Chris asks an open-ended question which has sparked a fervent rash of comments, requests, and of course, doubt. He asks with confidence, “Now don’t get excited, but what if Velo Orange had frames made?” He then lists a few core piece of his frame idea, and finishes his post with, “What else would you want to see on this imaginary frame?”
This is the most concrete example I’ve seen of watching a person build a business, and more importantly, getting explicit and public feedback from his *potential* customers to influence his product design decisions - all via his blog (and probably email).
Are there other interesting examples of this type of thing out there?
Oh, and I have to point you to the Velo Orange store to see the beautiful collection of hand-made bicycling goods that Chris is designing.