User Research & Testing Gotchas

One of my favorite user researchers and psychologists Dr. Susan Weinshank lists 10 things people shouldn’t do in user testing and research:

#10 Skip the pilot
#9 Draw conclusions from early and insufficient data
#8 Test too many people
#7 Too many cells
#6 Do a data dump
#5 Too much stock in after-test surveys
#4 Test too late
#3 Skip over surprises
#2 Draw conclusions not supported by the data
#1 Skip the highlights video

I couldn’t agree more with this list. You can read them more in-depth in her blog post. I’ve added some commentary of my own, which you might find useful:

Testing too many people

So you have this swanky new user interface you want to check out. And you schedule 10 people to test it. Problem is the first 3 people all have the same or similar problem. Guess what? Those other 7 are likely going to as well. You’ve scheduled too many people. Just schedule about 5. If the first 3 people have the exact same problems, stop now and iterate on your design.

Testing too late

Ah, don’t we all like to have beautiful, full-fidelity comps to show testers? Fact of the matter is I usually wait myself until the designs are in pretty good shape before I show them to customers. But I work super fast fast so its not a huge issue. However, for really complex, linear, flow, or other highly interactive projects, I get sketches in front of people really quickly. Are their mental models lining up with the intent of the design? Great, keep going. If not, stop now and iterate. Remember the cost pyramid? Design phase is a prevention. Correction is development. Failure is launching the wrong product / design.




Other than the one-question Net Promoter Score survey, there is a lot of data to show that user surveys are rife with problems. Use with extreme caution. I personally never use them.

Highlights Video

Making videos is a snap in this day and age. I use Quicktime on my Macbook and can record screencasts and walkthroughs in a snap, which provide an amazing amount of fidelity to developers, product, UX designers – pretty much any stakeholder. Because you can scrub the video, you can watch any portion of it as much as you need until everything becomes clear. Here’s an example of one I did for a client recently.

Keep your user testing and research focused and just like design – it should solve specific problems you’re looking to solve.

Speeding up Apple OS X Mavericks Finder

The Finder in MacOS X Mavericks is a tad slower than I’d like. This particular fix

sudo defaults write NSAppSleepDisabled -bool YES

seems to speed things up considerably. Open Terminal, copy and paste this command, then restart your computer. Finder should be a lot faster. What’s happening here? We think the Finder is entering some kind of low-power or sleep mode. Alas, I’m not a computer scientist.

Good design is everyone’s responsibility

I’m not shy about sharing my views when it comes to great design. When I see it I applaud it. But when I see bad design, I tend to want to help make things better rather than bitch about it.

Case in point this email from ConEd. As you can see its text on a busy background. They did an OK job lightening the background, but I am sure there is a large amount of people that found this hard to read. Your eye focuses on text, then background, then text, then background.

Screen Shot 2014-05-08 at 11.49.19 AM



Rather than do nothing, I tweeted at ConEd with my concern, and they graciously replied and said they’d look into it. Whether they do or don’t is on them, but I feel good that I, as a customer of ConEd, tried to make ConEd a better communicator, and thus, organization. Millions of New Yorkers rely on ConEd and I think behooves all of us to pitch in and help steer them whenever our expertise allows.