Is it possible to do usability testing on the cheap?

So, you’ve never done a usability test or focus group before. “How long is it going to take?” you might be wondering. As we all know, time is money.

It turns out that there is a huge range of time that can be spent on testing, and it doesn’t seem to matter what the subject matter is. What matters is the level of detail you desire.

In my opinion, for a well designed test, 80% of the usable information is revealed during the test and is observable by anyone watching.

There are two important points here. First of all, the test has to be well designed. If the script or discussion guide is not well thought out, the information that is gathered is muddy at best. For example, if the moderator asks, “What do you think about batteries?” you are likely to illicit responses from “I don’t like them” to “they have to be rechargeable” to “they cost too much.” All of these responses are useful, but how are you to know whether you should put batteries in the product or not? The subjects are not responding to a specific enough question. It might be better to ask, “If we used batteries in the product, when would you expect to have to replace them?” Or perhaps, “If we had a corded version for $20 and a battery powered version for $30, which one would you buy?”

The second point is that someone has to be watching the test in order for him or her to take the information away. The very subjects that you are asking the questions of, best deliver the information. Compiling videotape highlights or creating an executive summary is usually a back up to actually having the product development and management teams there during the testing. I did a set of focus groups for Intel where the boss’s boss watched almost every test. That’s commitment to understanding the data.

So, if you follow these two simple points:

  1. Design the test well, and
  2. Be there during the test,

you can save literally thousands of dollars and months of time.

Item Contractor Hours Risk of Eliminating
Develop goals 2 As long as goals are developed, a contractor doesn’t have to be involved.  But, a contractor can help focus on things that are testable versus those that aren’t.
Identify target subjects 2 to 4 Again, a contractor doesn’t have to be involved here.  Subjects need simply be categorized into separable groups. For instance, age, location, income level may all be considerations. Bear in mind that every time you add another category, you double the number of test subjects you need.
Prepare discussion guide / scripts / questionnaires 6 to 12 If you have never done a usability test discussion guide and questionnaires, you probably need an expert to make sure that you develop unbiased questions.  To save time, though, a preliminary list of questions (based on the goals) can be developed by the project team and used as a starting point.
Recruit participants Other, see notes Obviously this step needs to be done.  In my opinion, recruiting test subjects should always be done either through an agency that specializes in recruiting test subjects or by the project team. It’s simply too expensive to ask a usability contractor to do this work.
Conduct the test 2 per session

– plus –

4 to 10 for setup

Obviously this step needs to be done as well. Running the test requires being unbiased and good at leading people without leading them down the garden path.  These techniques are easy to learn, but can be hard to stick to if you are close to the project content. Remember, as many people from the project team should watch this unfold as possible.
Verbal debriefing ½ to 1 per session I find that simply taking the time after every session to discuss among the team what happened is the most valuable way to cement the test in the project team’s minds. It’s quick and worth it!
Postmortem 2 for prep
2 for meeting
A postmortem can be an alternative to a verbal debriefing, but usually data is turning stale after all the tests have been run (since tests usually are spread out over a few days).
Executive summary 1 to 8 An executive summary is moderately valuable if project team members attend the tests attend the tests and absolutely essential if they don’t. With project team members in the tests, it simply forms a way to bring together the results of each individual test into one cohesive written document.
Questionnaire analysis often 4 to 6
depends on the format of the questionnaire
Analyzing the questionnaires is fairly essential. The written data can sometimes be more enlightening than the spoken word.  Numerical answers are easy to analyze and can yield clear surprising results.
Transcripts 2 to 4 times the length of the sessions at transcriptionist rates Transcription is valuable for projects with long lives where the memory of the tests can fade. It is also helpful in projects that are so large that not everyone can attend.  Having transcripts can provide more efficient way to find a key moment in a test than videotapes.  Transcriptionists cost about 1/3 of the cost of usability contractors. For short-lived projects with quick turnaround, they may be over kill.
Categorized transcripts ½ hour per session Transcripts are sorted by category or question.  The answers to a single question for all sessions are grouped together.  If you have transcripts made, I’d suggest doing this as well.  It can help you quickly jump to the results of a particular issue.
Video highlights roughly the length of each sessions Video highlights are usually valuable only for upper management if the project team attends the sessions.  If they don’t, the video highlights and executive summary may be their only source of data.

I’ve presented some guidelines here for how much time things take if an experienced usability contractor is hired. Use these guidelines as a menu. Pick and choose which things are important to you, but bear in mind that there are risks to eliminating things from the list.

Bear in mind that after you have done a few tests, although you may not be an expert, you may know enough to take the place of the contractor for some of these tasks the next time.

If the project team attends the session, many of the results gathering steps can be avoided. So, bear that in mind when deciding if it is worthwhile to ask the entire team to be there. Their time is valuable, but so is the money it costs to hire someone to analyze the results. It isn’t always cheaper to have someone go off and “summarize” for you.

By spending enough time up front, usability testing and focus group analyses can be really inexpensive. Do it cheap and do it often!