At the end of a long week in the Flickr post-slideshow launch forum, I wrote the following bits in response to people repeatedly asking if we usability tested the slideshow before we launched it.
Of course we didn’t!
I just went back and reread what I wrote, and it seems worth it to publish here as well.
Flickr has never “usability tested” anything, at least not in the sense of taking something to a usability lab.
I’ve not been (working) here from the beginning, but I have been a user since almost the beginning. The nature of Flickr has always been to make bold decisions based on experience and opinions, put them out there, see how people respond, and tweak things along the way.
Sometimes they’ve been wrong decisions, and many times they’ve been groundbreaking-amazing-totally-cool decisions. I like to think that the reason Flickr rocks so hard is that Stew, George and the rest of the original crew didn’t take things into a usability lab, or usability test anything in the “proper” sense. How boring!
At its very core, the name Flickr Gamma (used to be Flickr beta) represents this playful, malleable Way of Doing Things.
So while you thought my response was perfunctory (and not funny), and I’m sorry if it came across that way, it really is the truth.
I highly doubt anyone on the team will ever make the decision to bring back the old slideshow. This type of decision is no different than the major redesign of Flickr’s UI a year or so ago, or any other semi-major tweaks that have happened. People always freak out at change.
I hope that answers your question.
In the end, think about this: had we taken it to a usability lab, and some group of 7 people spent 5 minutes in front of it while a researcher took notes, they’d have had their say in that time period. And then, a bunch more people would think it sucked anyway!
Instead, you get to use it, and have public discussions with the people who designed and built it, give feedback, and hear our answers to your questions in semi-real time.
I think that’s better, no?
(I might get smacked here, but whatever)
I think usability labs breed sterile, boring and faceless products. I *wish* all of the things I use on a daily basis came with the ability to ask questions and help debug WITH the people who design and build them.In my humble little opinion, it’s a much better way to do things.
So, thanks for your feedback. As Flickr always has, we’re listening to you, and making changes where they make sense and where we see fit.
I certainly know that there are many exceptions and business cases that fly in the face of this way of thinking, but from my current viewpoint, which is where I want to be, I don’t think there’s a better way to design products.
One question that remains with me, which I’ve discussed with various folks from Adaptive Path and IDEO, is how do design firms, which are limited in their ability to deliver long-term value in this realm, um, adapting?
The name Grant Petersen can bring on cheers or jeers depending on what person you mention it to. To some, he’s the eccentric retro-grouch design mind behind Rivendell Bicycle Works. To others, he’s one of the Premiere Thought Leaders of an entire world of Cycling That Simply Makes More Sense; a lugged-steel, racks-and-lights, wool-and-seersucker-based transportation-centric world that 99.4 per cent of bike shops and cycling magazines don’t cater to. No matter how people slice and dice him, one thing is for certain. He’s as down to earth as any guy could be, and his opinions on cycling are bound to evoke some discussion in any group of people with any kind of cycling priorities. Recently, Mr. Petersen was kind enough to take some time out of his six-days-a-week schedule to answer some burning questions that Chico Gino readers have submitted. For those of you not familiar with Grant or Rivendell, here is an excerpt from the company’s history to get you lubed.
From late ‘84 to late ‘94 I designed and spec’d bicycles and worked on catalogues for the U.S. division of Bridgestone Cycle, Japan’s largest bike maker. Bridgestone closed the U.S. office after ten years of no profit, when the dollar-to-yen exchange rate plummeted to the point where it became impossible to even break even. I was 40, and started Rivendell with $89,000, a mix of retirement money, savings, loans, and money raised by selling stock to friends.True to the cliche, Rivendell was in my garage for two years. Now we have 5,000 square feet at about $0.90 per square foot, one of the cheaper rents in town. We like it here a lot. It’s easy to get to, close to good food and riding, and it feels like home, except that summertime temperatures average 90°F and winter days rarely get above 57°F. Seersucker and wool? You bet!
We’ve been profitable two of the past eleven years, but cash flow is neutral. Sales are about $2.2 million dollars per year. We’re just breaking even, there are no top-heavy salaries, and we fret a lot during slow weeks (and months). I do, at least.
Our mission is to make things that wouldn’t be made if we weren’t here, to offer an alternative to racing-centric bikes and parts, and to espouse a different approach to riding. And to resurrect and keep healthy many of the better ideas, designs, and styles of bicycles, clothing, and accessories that we personally like to use or wear. To sell lots of wool, and wipe synthetic fabrics off the face of the earth by the year 2010. That’s a joke. To still be around in 20 years. That would be great.
If you’d like to know more, just ask. It’s not a secret business we have here. —Grant
And now for the interview!
In your catalogs, web site, and in The Rivendell Reader, you write a lot about simplicity. Why is simplicity important?
Simple things make people feel smart, or at least competent, and complication has the opposite effect. If people feel smart and competent, they’re happy, and happy people are nice to other people, and it all starts or stops with how hard it is to use something.
Do you think simplicity should be a higher priority for designers of other products? In particular, designers of digital stuff?
Especially that, and I’m sure they’re trying, but it’s not reaching me. I don’t like digital cameras with layers of submenus below the surface that you don’t even know exist, and icons that I have to look up to understand. It all goes back to making me feel dumb, and in the case of cameras, it seems like it’s just a consequence of trying to pack too many features into one widget. Just because a camera is capable of fantastic things doesn’t mean that I am, with that camera. I know they have instruction booklets for this stuff, but I think they rely on the instruction booklets too much. I want a digital camera that does ten things well and doesn’t need an owner’s manual. I think, if the designers knew they couldn’t supply instruction books, the designs would be simpler and the world would be a better place. It sounds glib to say that, and it would be too limiting in some cases, but for sub-$300 digital cameras, I think it should be the rule.
The newest bike from Rivendell is the A. Homer Hilsen. Is it true that Mr. Hilsen and Mr. Nigel Smythe knew each other? What’s the story there? 
They not only knew each other, but they were both friends of Paul Bunyan and the Easter Bunny, too. They came from a different time, when “digital” meant “of, or pertaining to, the fingers.” They’re plain and honest, and their names are supposed to reflect that, to set them apart from the cha-cha-cha personalities and products. What “honesty” there isn’t in their actual existence is more than made up for in what they are, in them as products.
Rivendell is in the fortunate position of having a rabid following; some would say it’s a cult. Are there downsides to being a cult leader, even if it isn’t on purpose? And if so, how can future cult leaders learn from your mistakes?
Hmm… well, if I just rush in to an answer, I’m agreeing with the cult thing, and it isn’t quite that. What we have here is a way of presenting information that lacks the “buy me and be cool” voice that smart and mature people find offensive. Most companies can’t get that voice out of their mouth. The ads and image are written by professional copywriters whose goal is to retain the account by flattering the company. It’s all over the place these days, because the “17 to 34” market buys so much stuff, and most of them are still looking to express themselves through their purchases. We have a smaller mission and aren’t out to conquer that world, so it’s easier for us to seemingly take a higher road. As a consequence of just being normal, we do stand out, and every now and then somebody will think we’re infallible, or maybe it comes across as we thinking ourselves are that way. Of course inevitably we’ll do or say something that proves that to be false—maybe we’ll send a package ground instead of 2-day like they requested, or a new and packaged garment might have a flubbed seam or something, or we’ll get too casual with them on the phone and they’ll take offense. Then they turn on us like we betrayed them. They turn on us harder than they’d turn on a company that they didn’t feel as much kinship with, I’m sure, and I guess that’s the risk. It’s a good risk, and it’s not a risk for a business that nobody cares about, but it IS a risk.
You run Rivendell as openly as any company I’ve ever seen. Is there a conscious philosophy behind that? 
Well, I wouldn’t call it a philosophy, but I don’t distinguish between “me” and “my company” when it comes to things like keeping secrets and telling the truth. It’s hard to keep secrets, so it’s best not to have any, but beyond that, I’m not exactly sure of what you mean by “open.” Is that it? If it isn’t, just clarify it and I’ll try to answer it.
Gino’s note: His answer here couldn’t get more to the core of my question.
Of up-and-coming framebuilders in this resurgence of handmade bikes, have any piqued your interest? Do you even care?
I think it’s good that lots of them are out there. It’s hard to be an American framebuilder these days. There are hobbyist builders, many of whom seem to make nice-looking frames, and then there are full-timers. I’m neither, but I’ve been around long enough to have a perspective, and my perspective tells me not to envy them. I’d like to give them advice, but they aren’t asking, and are probably too independent to take it even if it seemed to make sense to them. The thing about framebuilding, though, and this is something few people and especially bike riders understand, is that there are custom framebuilders who are scattered all over the learning curve, with probably fewer than seven or eight at the top. It’s not like being a tailor or a neurosurgeon, where the barriers to entry are long and difficult and people bail out on the way. It’s more like being a desktop publisher or a photographer. Anybody can buy the tools and say, “I’ll build you a custom,” and that’s pretty much all it takes to be a “custom framebuilder.” So, unless you have a way to evaluate the frame, it’s hard to tell how good the frame is.
I am not a builder myself, but I’ve heard more than one builder say that there are three big humps to get over on your way to becoming good. The first comes at about 50 frames, the next comes at 150, and the next comes at 500.
Building a frame is more than being able to braze or weld a frame that holds together. It’s solving problems, and anticipating problems before you cut the tubes and stick them together. It’s also knowing how to braze quickly and thoroughly and minimizing the pulling and twisting that occurs when you braze, so that the frame stays square as you build it. It’s controlling your spillover, so the lug edges are crisp, while the joint is full. Plus, it’s maintaining consistency over hundreds of frames, and when you do that, the challenge is avoiding cockiness and complacency. If you master the mechanical part, the challenge is maintain your consistency from frame to frame, over hundreds of frames, and to not become the “devil may care” brain surgeon, so to speak. A custom framebuilder with 70 frames under his belt hasn’t solved enough problems yet. It’s possible that he’s creating problems he doesn’t even recognize. Another way to look at it—rather than seeing it as being overly hard on new framebuilders who are earnestly doing their best and want to make a life of it but haven’t built a hundred frames yet—is that it’s a real profession that requires lots of experience, and that makes it respectable in the same way that any hard-earned professional status is respectable. I haven’t answered the part of your question about—“have any piqued my interest?” and “do I care?” I think I care, in the sense that I want them to do good and make a living and be happy, and I’m interested in what seems to be the rebirth of interest that’s giving birth to the builders and reviving others who’ve almost given up. As lousy a time it is to be building lugged steel, in some ways it’s the best time for it. Good lugs are more readily available than they’ve ever been, and the same goes for tubing. But then, not having to blacksmith lugs into fine form lowers the hurdle to entry and results in lots of frames that appear to be much finer than they actually are. Good lugs and tubing make good frames in the same way that good wood makes good furniture. Meaning, it has almost nothing to do with it, except as a starting point.
You and I both like to flyfish, and we both ride bikes. If you could take me on a bicycle-flyfishing trip, or a flyfishing-bicycle trip, where would go, and what flies would we bring?
If it’s any trout stream in America with a good food supply and therefore a healthy population of trout between 12″ and up to maybe 24″, and the water is running clear and not too high, and it’s fairly wadeable (so, not too deep), we’d need four flies only. Three would be nymphy. A size 16 with a thin hare’s ear body ribbed with gold, a peacock herl thorax, and a brown partridge hackle, on the long side. Then the same kind of fly with a muskrat body, black herl thorax, and a starling hackle. Then a pheasant tail nymph, tied with copper wire twisted together with 2-3 fibers from a pheasant’s tail, but with a herl thorax, and no hackle. Then a size 18 elk hair caddis. Four flies makes you fish them right and not blame the fly, and maximizes the time you spend fishing, not tying on flies.
Gino’s note: Hey Grant, I can show you where to catch Goldens like this! 
There are an awful lot of young guys, and maybe some gals, graduating from places like UBI right now, with the aim to “make it” in the bicycle industry. Any advice for these new throngs of grasshoppers and greenhorns?
Find a style and stick to it, no matter how many sales you have to give up along the way, to people who want you to build their way, or with this or that material. If you can make your bike distinctive and recognizable in some way, that’s good. Don’t branch out, though, and don’t copy people who are better than you are if you’re going after the same audience. I actually think it’s not a fantastic growth industry, and I wouldn’t be optimistic about it, but the narrower the focus, the better. Only unicycles. Only 650B unicycles. Only this, only that. Find something that nobody is doing, and do it, and THEN figure out how to sell it. It has to make sense, of course, but you don’t need a huge market. It’s way better to build nothing but lugged 650B unicycles in one color only and become the go-to person for anybody in the world who wants one, than to build whatever seems popular at the time, and find yourself competing with everybody else from the other “I do anything and everything” custom builder down the street, to the biggest factories in China. But, the temptation to capitalize on trends is too strong, and the rewards of going your own way are so remote and iffy at best, so you don’t see a lot of that happening. It’s just a hard way to make a living, I think.
In 1992 you said the following on the future on cycling:
“The best use of a bicycle is commuting, it’s not racing or competing or recreation or anything like that. Ultimately its best use is getting cars off the roads and the government is not sympathetic to that idea at all. In terms of the industry, within five years there will be half the number of manufacturers as there are today. Whoever doesn’t get their bills paid will go by the wayside. Also, when things go high tech in terms of their materials and design, it brings in a lot more industries into the marketplace. Consider how carbon fiber and titanium have brought aerospace and other non-bicycle corporations into the cycling industry. Those guys have the money to compete and smash and just snuff out the smaller companies. That happens and that’s one of the consequences of advanced technology. So when we go to electronic shifting, which will happen within about two years, somebody like Westinghouse will get into making them, too. That will make it hard for Mavic and Suntour to compete with a company that large. So we’ll have electronics companies making bike parts.”
Nearly fifteen years later, how much of what you said still holds water? What has changed?
Things aren’t fully electronic yet, but they’re headed that way still. The Westinghouse prediction hasn’t come to pass, but as bicycles become more electric and electronic, it’s only natural that outside specialists will have a bigger hand in them. Westinghouse has its spies watching us constantly, and is ready to pounce when the time is right. I feel more strongly about racing now than I did then. I don’t mind races, or racers, but I sure do mind the influence that racing has on equipment and riding attitudes. It seems like a big trick, to me. You flash it up and make it popular so you can fool new riders into going down this path of pain and not-fast-enoughness, and once you get them there, you assure them that they can go faster if only they ride your $800 wheels or the two-pound frame or chug the squeeze-gel you provide them. It’s like you’ve captured them like slaves, and now they’re serving you and can’t escape, and meanwhile, they’re missing out on the best kind of riding, which is just riding without comparing yourself to racers or wishing you were something you’re not, or not as good as you want to be. Anyway, it’s hard to talk about without sounding like a raving maniac, but the thing is, I feel like a raving maniac about it, too. I keep it under control to maintain a certain dignity, but let me tell you, I am not happy with the way racing has influenced bicycles and riding. I think it keeps too many people away.
One last note: I snagged photos in this interview without permission from Rivendell’s web site, a Flickr user, and Sheldon Brown’s web site. The trout photo is mine, from custom bamboo flyrod maker, Jim Lowe.
Also, this interview originally appeared on my other blog, Chico Gino. After finishing the interview, I saw so many interesting parallels between Grant’s bicycle design focus and the Interaction Design field that I had to post the interview here as well. I hope you enjoyed it, and would love to hear feedback on bringing in perspectives from designers from “vastly different” industries to Push Button For.
Ajax and the Lightbox effect gone bad. Rent.com forces me to “create a password” to “keep my activities private” - when all I’m doing is trying to browse. Plus, they remind me that I haven’t claimed my $100 reward. I have no way to continue until I fill in that password. The funny thing is, on the initial landing page for this property (I got to the page before this one from a Google search results page), I explicitly told them that I didn’t want any $100 reward. Come on eBay, you can do better than this.
And do you think I filled in that password? Or did you lose a transaction and customer due to your Used Car Salesman approach?
Today I am attending the WebGuild Silicon Valley Annual Conference in Santa Clara, CA. One of the keynote speakers was Marissa Mayer. Marissa is the VP of User Experience at Google. The format of the talk was a directed Q&A session, and then they opened it up to the audience at the end of her talk.
Marissa had some really interesting things to say, and I think that it is worth noting a few of my key takeaways. They are nothing but common user experience sense; Product managers everywhere should take notice.
On Google’s homepage/search simplicity
Google’s simple UI went like this: Sergei was an engineer, and didn’t want to build a “better” page. He just wanted to test his search engine. Initially Sergei didn’t even have a Search button. He figured the search field was enough, since the Return keyboard key worked well enough.
On Designing, Developing, and Launching products
Don’t overinvest upfront. Don’t overthink a product. Think lean from the getgo - not about everything it could do. Get it out there early, change it often. Getting it out there will determine if it’s going to take off; getting out there early will also beat the competition.
Google often walks away from revenue for the sake of the user experience. They believe that if they nail the user experience first, the revenue will come. Never compromise the user experience for the sake of immediate revenue. User experience is the best investment they can make.
Speed is at least important in user experience as any other factor. Don’t forget about speed.
On Social/Knowledge Search
Google is focused on social/knowledge search, and they’re looking at companies in Asian countries who have done a great job with the concept.
On advertising
No ads on the homepage. Why? There’s no content, so how could ads be well-targeted? It simply makes no sense. And if there is content, it still isn’t well-targeted, so ads might still not make sense. Wait until the user gives you explicit input about their interests; you can always target ads better that way.
– END NOTES –
You might hear people say that Google is a cold, machine-driven company that is solely focused on technology; that they don’t have any user experience representation or focus. Don’t believe the hype. They’re a big company that still operates its product teams as lean, mean, aggressive-as-hell user experience and business and engineering monsters.
As they spread themselves thinner, and become involved across more and more disparate verticals it will be interesting to see if Google can maintain their momentum.
I was unfortunately unable to attend BayCHI this week when Peter, Tim and Larry spoke on designing systems with emergent behavior. Lucky for me, Peter posted some of his thoughts and takeaways from the evening.
A couple of things Peter wrote struck a nerve:
Peter first says:
“There are no good user research methods of emergent behavior
The standard tools that a user researcher has tend to focus on an individual’s interaction with the system. Or, if we do engage with groups, we’re limited to the scope of groups we can interact with. So, how can user research inform the design of systems with emergent behavior? We need new methods and approaches that allow us to work with a crowd.”
Peter also goes on to say:
“Embrace the chaos
…I’m surprised that most companies don’t take the Amazon.com approach of testing designs with small percentages of the users and gauging behavior before unleashing it on the world at large.”
Companies do embrace the chaos.
I don’t know if most companies do it, but I know that all the big companies with massive user bases do it. At Yahoo and elsewhere, it’s called bucket testing. Based on any number of factors, including but not limited to revenue risk, research goals and time/budget constraints, a new rev of the product will be launched to a tiny percentage of the population. So when you have 400,000,000 users and you let 5% of them see the new product, you get a rich and reliable data set, as well as a wealth of information on how the product will be used, and how it will perform when you do open the flood gates 100%.
Bucket testing is constantly happening at Yahoo, Google, MSN, Amazon and others. Have you ever seen a new feature for a week or two, and then it vanishes into thin air? You were in a bucket test.
A Strong Statement
I know I’m going to get heat for stating that bucket testing is a user research tool, but I’m going to say it anyway: Bucket testing is a great tool for designing emergent systems. It is not the typical lab-based, or field-based “user research” that we all think of, especially because it happens after the system is designed. However, because of the limitations of human beings’ abilities to predict (unless there are clairvoyant UE people out there) what other people are going to do, bucket testing is the best tool to date that I’ve seen for informing my design decisions in SWEBs (Systems with Emergent Behavior).
As designing emergent systems become increasingly complex, bucket testing is possibly the best user research tool available; something that no amount of up front research can provide, and something that no design firm or user experience strategy firm can account for unless they are embedded in your company for a long-term relationship. I hated to say that last sentence given my consulting history, but it seems to be more and more often the case. It’s time for consultancies to innovate on how to provide this service in a long-term valuable way.
A confession
When I was first exposed to bucket testing, I hated the idea; I was also from a user experience consulting background. I loathed putting my hard work and Fantastic Design Decisions in front of a small piece of the population, just to observe and then tweak and change my minor failures in near-real time. But over time, I have actually learned to embrace The Bucket Test Methodology as a key tool in researching how users respond to the emergent systems I design, and making those systems or products better.
The other tool I should mention, which only few products and companies are lucky to have, is the loud-mouthed users themselves. If you design a product/system that people are passionate about, they will constantly give you, ahem, advice on how the system should emerge to meet their needs. For every five you hear from there are 50,000 that feel the same way. In this case it is your job to be the bullshit filter and improve your product accordingly.
Not the be all end all choice
Bucket testing isn’t The Best Way, or the only way to gain insight into a SWEB. In fact, in many cases, such as games, I don’t see how it has a ton of applicability. However, for large content-driven and transaction-driven and user-driven and recommendation-driven and algorithm-driven online ecosystems, bucket testing might well be one of the best tools at our disposal.
The key is being sure to make bucket testing not just a Product Management-driven exercise; involve User Experience folks from the get go. Otherwise, your bucket tests will be too heavily focused on numbers, and not on evolving based on behavior.
Oh, one last comment on Peter’s post. Many contemporary systems do seem to live in perpetual beta, and I think it’s fantastic. It is a clear philosophical shift from the engineering/shrink-wrapped wait-to-pay-for-the-next-release software model of Yore, and evidence that companies are trying harder to evolve their products at a rate which keeps up with our behavior. And I welcome that. It forces designers and engineers to work more closely; that historical separation are why products have sucked for so long.