Find out what they like,
and how they like it,
And let ‘em have it just that way.
Fats Waller
Usability Testing 101
Gina Pearson, Web Manager
January 29, 2003
Today’s Goals
Primary purpose is to learn how ERS
is integrating usability testing into its Web
design & development process
Review why user-centered design and
testing (usability engineering) are
important
Explain the usability testing process at
ERS
Today’s Goals
You will learn how to:
Assemble your test team
Recruit & select test participants
Formulate a test plan
Create testing tasks & scenarios
Facilitate a test session
Test without a lab
Record, compile & analyze results
Communicate results effectively
Why Is Usability Important?
Ballot Usability
What should have been addressed?
– Font size
– Lighting
– Text vs. background color
– Instructions
– Layout
– Act of voting (punching ballot card)
Ballot Usability
Costs
$20,000 to run usability tests
addressing
Layout
Multiple votes
Dimpled ballots
About $1 billion spent by both
candidates to try to get elected
Failure of Web Sites
Users could find information only 42%
of time (15 large commercial sites Spool study)
Sixty-two percent of web shoppers
gave up looking for an item (Zona study)
Only 51% of sites complied with simple
Web usability principles (Forrester study
of 20 major sites)
User-Centered Design
Philosophical & Practical Approach
User-centric - not developer, designer
nor organization-centric
Who are the users?
What information do they want?
Based on research and data not
opinions
Testable & verifiable
The Traditional Development Process
User-Centered Design
A methodical approach to producing a
web site or any user interface. It is a
practical and systematic way to deliver a
product that works for users.
Test early & often
Build a little, test a little, fix a little
At ERS, we are incorporating usability
testing & other user assessment
techniques as a regular part of our web
design & development process
User-Centered Design
Types of user assessment techniques
Interviews
Ethnographic observation / field research
Focus groups
User personas
Expert (heuristic) reviews
Usability testing
Customer satisfaction surveys
Customer email trend analysis
Search engine query analysis
What else?
Myths About Usability Testing
Usability testing:
Is time consuming, expensive &
otherwise resource intensive
Typically involves hiring a specialized
consulting company
Requires a formal testing lab
Must be videotaped
Always involves writing a detailed
analysis and report of results
Diagnostic User Testing
Find out what is and is not working well on the site. In
a diagnostic usability test, you usually want to answer
questions like these:
Do users complete a task successfully?
If so, how fast do they do each task? Is that fast
enough to satisfy them?
What paths do they take in trying? Do those paths
seem efficient enough to them?
Where do they stumble?— What problems do they
have?— Where do they get confused?
What words or paths are they looking for that are not
now on the site?
Methods
Assemble test team
Administrator
Facilitator
Observers / recorders
Video recordings operator (optional)
Methods
Define goals & scope of what you are
testing
Recruit & compensate participants
Develop testing tasks & script
Set up testing environment
Draft participation agreement
Rehearse & refine test protocol
Conduct tests
Analyze & report out results
Methods
Test Environment
Methods
How many participants are needed for a
usability test?
It depends!
Typical range is from 5 -12 users in each test
Our approach: 3 users per test but conduct
several rounds of testing throughout an
application or feature’s development
If you have different potential user groups
(for example, small business owners,
unemployed, blue-collar workers), try to
include representatives of all these groups
Methods
Participant recruitment & compensation
Typically time and labor intensive - consider
outsourcing (average $200 per participant)
Go through professional networks &
contacts for volunteers
Conduct testing at conferences & other
events with volunteer representatives from
your target audiences
Always give volunteers a small token of your
appreciation (t-shirt, mug, etc)
Methods
Task construction: where do ideas for tasks
come from?
Concerns about information architecture,
design, terminology, functionality, etc that
have surfaced among the product
development team
Trend analysis of emails from site
customers
Search engine query analysis
Trend analysis of call center inquiries
What else?
Methods
Constructing usability testing tasks
Find or scavenger hunt tasks
Verb-based tasks
Scenario-based tasks
Methods
 Conducting the Test (about 2 hours)
Pre-test briefing for observers
Greet participant & administer participation
agreement
Introduce recorder & observers
Explain testing process to participant
Administer pre-test interview & activities (if any)
Facilitate testing tasks
Administer post-test interview questions
Involve observers in general Q&A session at the
end
After participant leaves, debrief recorder &
observers
Methods
Developing the testing script &
facilitating the test
3 parts of a usability test:
Pre-test interview
Testing tasks (“think out loud”
methodology)
Post-test interview
See another sample script at:
http://www.sensible.com/Downloads/scri
pt.doc
Methods
Characteristics of a Good Facilitator
Quick learner
Ability to establish instant rapport
Empathic “people person”
Good listening skills
Patient
Flexible & adaptable
Long & focused attention span
Impartial: not wedded to a
particular design
Methods
Analyzing & reporting the test results
using the KJ method
Assemble your testing team immediately after
the last test for a brainstorming session.
Ask each participant to write down each major
observed problem on an index card or sticky
note.
Put all index cards or sticky notes on a large
wall or board, no discussions!
Read each others cards silently, add additional
problems.
Methods
Analyzing & reporting the test results
using the KJ method
Sort problems by area. Eliminate duplicates if
total agreement.
Name each group of problem areas.
Vote for most important problems, each
participant has ten votes.
Assign one team member the task of writing up
the results of brainstorming session.
Disseminate results of brainstorming session to
development team via email
For Further Information
Contact:
Gina Pearson, Web Manager
694-5119
Brenda Powell, Web Analytics Manager
694-5134
For Further Information
Don't Make Me Think! A Common Sense Approach to
Web Usability
Steve Krug
The book every manager should read!
Handbook of Usability Testing: How to Plan, Design
& Conduct Effective Tests
By Jeffrey Rubin
Practical, step-by-step guidelines, requires no engineering
or human factors training.
A Practical Guide to Usability Testing
By Joseph Dumas & Janice Redish
Premier text in the field.
For Further Information
Jakob Nielsen (www.useit.com)
Why You Only Need to Test With 5 Users
http://www.useit.com/alertbox/20000319.html
Success Rate: The Simplest Usability Metric
http://www.useit.com/alertbox/20010218.html
Usability Metrics
http://www.useit.com/alertbox/20010121.html
For Further Information
Jakob Nielsen (www.useit.com)
Cost of User Testing a Web Site
http://www.useit.com/alertbox/980503.html
When to Outsource the Recruiting of Test Users
http://www.useit.com/alertbox/outsource_recruitin
g.html
For Further Information
Jared Spool (www.uie.com)
Using Paper Prototyping to Manage Risks
http://world.std.com/~uieweb/paper.htm
Usability Labs: Our Take
http://world.std.com/~uieweb/labs.htm
Eight Is More Than Enough
http://world.std.com/%7Euieweb/eight.htm
For Further Information
Interaction Architect
13 Common Objections Against User
Requirements Analysis & Why You Should
Not Believe Them
http://www.interactionarchitect.com/articles/arti
cle20000609b.htm
Observation Methods & Tips for Usability Testing
http://www.interactionarchitect.com/knowledge/art
icle19991212shd.htm
For Further Information
DialogDesign
(www.dialogdesign.dk)
Ethics in Human Computer Interaction
http://www.dialogdesign.dk/chi2001.html
Usability Test of www.TowerRecords.com
http://www.dialogdesign.dk/utestreports.html#sam
ple
For Further Information
Usability.gov
 Why Should I Do Iterative Usability
Testing?
 How Can I Encourage People in My
Organization to Conduct Usability
Engineering and Testing?
 Do I Need a Lab to Do Usability Testing?
 How Many Participants Are Needed for a
Usability Test?
 How Much Does It Cost to Do Usability
Testing?
For Further Information
Training Opportunities
 National Cancer Institute
Usability Seminar Series & Education Training
Program,
http://usability.gov/events/seminars.html
 Society for Technical Communication WDC
Chapter, www.stcwdc.org
 User Interface Engineering
 Human Factors International
 User Experience Conferences (Nielsen Norman
Group)