Usability Testing - A Primer
Usability Testing Getting Started Guide
When you visit a website, use an app on your phone, or software on your computer, do you have a good experience or perhaps not? If your experience is good, it’s likely because usability testing was a part of the product development process.
If your company is doing user testing for the products you work on, then you know how important it is to build user experience into an iterative design process to test, incorporate user findings, make changes to the interface, and test again.
This article is for those who may know something about usability testing but want to learn more or for those who don’t know much about it but want to get started.
What is usability testing?
Usability testing is the activity that focuses on learning how your users engage with your product or interface by watching them perform tasks that are real and meaningful to them.
Usability testing uncovers the pain points, confusing terminology, unclear navigation, and wrong assumptions about how users expect to engage with the interface (based on their “mental model” of how it should work) versus how the interface forces users to engage.
The goal of usability testing is “discovery” of your user’s experience so that you can build this learning into your product in development by strengthening what works well and reducing the pain points.
How do you conduct usability testing?
Usability testing can be conducted in person in the same space with the user and moderator or remotely from your location to your user’s location using meeting software such as Teams or Zoom. Usability testing can also be conducted through one of the software testing platforms like usertesting.com, in which users perform tasks and provide feedback without observers being present. The sessions are recorded for viewing later by stakeholders.
In this article, I will focus on in-person, do- it-yourself usability testing, which can be formative or summative.
What is formative usability testing?
Formative usability testing – also called generative testing – is testing that is done early in product development. Formative usability testing provides insights about your users’ experience at early stages of design and development and results in recommendations for changes, which allow developers and designers to course correct before code is frozen.
Whatever budget you have for usability testing, you should allocate a part of it for formative/generative testing. Why? Because if you wait to do usability testing when the product is nearly done, you miss the opportunity to make meaningful changes without investing substantially in time and cost at this point.
But if you do formative usability testing, you utilize a low-cost way to build user experience into the product’s development. Even one formative test is better than none. But repeating the formative test process along the development timeline increases your ability for continuous improvement of your users’ experience. The mantra of formative usability testing is “test early, test often.”
What are the elements of a formative usability test?
Happily, you don’t need much in terms of resources or requirements to conduct effective formative usability testing. Gone are the days when you need a formal lab like the one shown here on the left with the one-way mirror for observers to see the action taking place in the evaluation room, although it can be useful in some kinds of testing to have observers on site.
Usability testing can be done with just a simple set up like the one shown on the right, using a conference room or office for the sessions.
Also gone are the days when you needed to schedule lots of testing sessions. Five to eight users is typically sufficient.
We know that small numbers are effective from research conducted in the early 1990s by Jakob Nielsen and others who noted that when you recruit the same subset of your users doing the same tasks, you will reach the optimal cost-benefit ratio at five users, weighing the cost of testing against the benefits gained.
At five users, you will attain discovery of 85% of the findings for the study. Rather than spend more time and money to learn the remaining 15% of findings from the study, it’s better to iterate on the design and test again.
Small studies, incremental changes, big improvement in UX.
The elements of a formative usability test are:
- A moderator/facilitator – working from an informal agenda or a more structured moderator’s guide to maintain consistency between testing sessions
- A note taker – taking notes into an Excel spreadsheet or Word or Google doc to capture what users say and do
- A participant – the target user selected because they have the need or goal that your interface can satisfy
- A camera – not required, but good to use for session recordings if you are in a space where you can set up a camera or you’re conducting remote testing where the participant has a webcam
How do you recruit study participants?
It’s really important to recruit the right participants – your actual users – for usability study results to be meaningful. Getting participants is the hardest part of planning, so you want to start this task as soon as you have identified who your users are.
Small studies work well when you have real users. Grabbing “Joe down the hall” or your neighbor down the street is – dare I say – a waste of your time and budget because you can’t know if these participants’ experiences reflect your real users’ experiences. The same goes for using internal users: Even if you think internal users are OK because they are far removed from your product, they are still too knowledgeable about what your company does, the internal jargon you use, and other learned experience that should disqualify them. Unless, of course, the product is for internal users.
Recruiting participants yourself
You can recruit participants yourself via social media, your company’s website, your marketing department, your current customers, and other no-cost methods. You may not need to provide an incentive for their participation, as they may be interested enough to participate because they like to learn about new products and services that relate to their needs. However, even when these participants don’t expect an incentive, it’s nice to provide one. Consider your budget to determine what you can offer. At the lowest level, it could be a coffee mug or branded t-shirt. A better incentive is a gift card from Amazon or elsewhere in whatever amount fits the budget. And, of course, actual cash as an incentive is always an option if your budget allows it.
Recruiting participants through a recruiting company
You can also hire a recruiting company to recruit your users and pay their incentive for participating. The recruiting company will tell you what the incentive needs to be and also their costs for recruiting. Some recruiting companies recruit on a national level; others are local to your area.
If you use a testing platform, you can use the platform’s participant panel to recruit users for your study. You fill out a brief screener to target your user group, and the platform does the rest.
What happens in a formative usability study?
We’re already identified the basic setup of a usability test, which involves three people – your user, a moderator, and a notetaker. So, what do they do together?
The moderator typically does the following:
- Greets the participant and makes them feel welcome
- Reviews forms like an NDA (non-disclosure agreement) and video consent form
- Introduces the study goals and emphasizes that “we are not testing you; you are testing the interface (or product) so that we can learn from you.”
- Describes the “think aloud” protocol, which is the methodology of having the participant share their thoughts as they engage with the interface.
- Using the information you obtained about your participant from the screener, asks pre-test questions to learn about the specific experience of the participant
- Asks pre-test questions to learn about the specific experience of the participant's current or prior experience with your product or competitor products and their needs and goals as they relate to your product
- Describes the first task (and it can be helpful to also provide the task description in a handout)
- Observes and listen and take notes as the user engages with the tasks and shares their thoughts about it. Typical think aloud responses include:
- I’m not sure I’m in the right place here…
- This is exactly what I thought would happen when I clicked on this link
- I love how it lets me do...
- Asks follow-up questions based on what was observed; probes for insights using open-ended phases, such as:
- Tell me more...
-
I observed that you did …. What was your thought process at that time?
- Avoids asking leading or biased questions, such as:
- How difficult was that for you?
- Did that go well?
- Were you frustrated when you were trying to do that task?
- If the participant asks you questions, answers with a question back to the participant, such as this exchange:
- Participant – Did I do that right? Moderator – What did you expect to have happen when you did that?
- Participant – I don’t know how to find what I’m looking for. Moderator – What would you want to do at this point?
- Describes the next task and the next, using the same method, until you have presented all the tasks (or run out of time)
- ·Conducts a post-test interview to gain overall user insights on the experience, using questions, such as:
- Was there anything particularly easy or difficult about doing these tasks?
- If you could make one change, what would it be?
- What was the best part of the experience?
- Ends the session by thanking the participant and showing appreciation for the insights you have gained.
What is summative usability testing?
Summative usability testing is the opposite of formative usability testing in these ways:
- Testing takes place at or near the end of product development
- The product can be evaluated against benchmarks and metrics for time on task, task completion rates, and pass/fail performance
- More sessions are required – typically 15 -20 participants
What happens during summative usability testing?
The general approach to summative testing is similar to formative testing, engaging a target user in tasks with the help of a moderator and notes by a notetaker. Recruiting is similar, but a stipend is generally expected in summative testing, reflective of the commitment of time expected of the participant and to help assure that you reach your target for the number of sessions.
Summative testing specifics include the following:
- Users engage in tasks without interruption and without thinking out loud or otherwise talking to the moderator
- The moderator’s guide is highly structured so that the same information is consistently presented to each participant; moderators often read a script to the participant
- Data is generally quantitative, with counts and completion rates being noted
- Formal post-study questionnaires, such as the SUS (System Usability Scale) are often used as a standard quantitative measure that can be compared against other studies of your product or the industry
- The SUS is just 10 questions, so it doesn’t take much time
- It is not technology dependent, so it can apply to any interface; you might need to change the word “system” in the questions to “interface” or “application,” but that’s OK
- A metanalysis by Jeff Sauro of SUS scores across numerous industries and a variety of applications found that the average SUS score from a usability study is 68. You can use this number as a benchmark for your studies.
Which is better? Formative or summative usability testing?
The answer depends on your goals. Each type of usability test – formative or summative – can provide useful and important insights about UX (user experience).
Formative usability testing is the technique to use during product development. It’s easy to incorporate into the design cycle and, because it’s qualitative in nature, it both shows you how users engage and tells you in their own words what pleases, confuses, or frustrates them as they work through tasks. These qualitative insights form the basis of recommendations for design improvements while the product is in development.
Summative usability testing is the technique to use at the end of development, as it provides the metrics to determine if you got the usability right. In summative testing, you validate the design against measures that determine user success, user satisfaction, and user outcomes.
Whatever budget you have for usability studies, make it a priority to conduct both formative and summative usability testing. A small formative study or two early in development can greatly increase the likelihood that your product will measure up to users’ expectations in a summative study.
Want more practical insights? Check out these blogs.
Carol Barnum
Carol brings her academic background and years of teaching and research to her work with clients to deliver the best research approaches that have proven to produce practical solutions. Carol’s many publications (6 books and more than 50 articles) have made a substantial contribution to the body of knowledge in the UX field. The 2nd edition of her award-winning handbook Usability Testing Essentials is now available.
Size Matters.
We are a small, boutique UX consultancy.
That means you work directly with the top UX experts.
We are with you every step of the way.
Your Users.
Our Experience.
