Here are the questions we get asked frequently about usability testing
How many users does usability testing need?
This is one of the most frequently asked questions! The simple answer is… it depends. But, more specifically, for qualitative research, we recommend small studies with goal-directed users who represent a particular part of your user base, so that results from a small sample can be realistically compared. Our typical studies have five users in a day in one-hour sessions, with a findings meeting at the end of the day.
If you have budget for 15 to 20 users, we recommend testing with several smaller groups of users and creating scenarios that work for each user subgroup. For example, with 15 users, you could get results from three different user groups. Or, you could conduct three separate studies at different stages in the development cycle for your product.
However, if you we are conducting human factors validation testing for medical devices, FDA requires 15 participants per user group.
How does UX Firm recruit users for studies?
Unlike some companies that use market research firms to recruit users from their active database of prior participants. Our user recruiting process is highly effective in contacting and contracting your specific, targeted users who share your users’ qualities, motivation, and goals.
When it’s appropriate to recruit your users from your own customer or employee databases, we work with you to screen, schedule, and support your users in arranging for their involvement in testing.
We also typically arrange for and pay the incentive or stipend that each user receives for participation.
How do we plan a test?
Because we use a team approach to user testing, planning is the essential starting place. In a 2-hour planning meeting, with a focused agenda. Our team meets with your team to establish the essential elements for the usability test. These elements include setting goals, determining tasks (which become scenarios), determining questions to ask during testing and feedback mechanisms to use after testing, and creating the all-important user profile for recruiting the users from the specific subset(s) of the user population.
We can conduct this planning meeting face to face or via web-based conferencing. Minutes of the meeting become the test protocol for the study.
Describe testing methodologies. What happens in a usability test?
A typical test session is an hour. The timeline for the hour is often like this.
- 10 minutes—greet participant, provide orientation to the study, including coaching on “think aloud process” to learn user’s thoughts during activities with the interface
- 40 minutes—participant works through scenarios of use, based on the plan for the study
- 10 minutes—post-test feedback mechanisms (questionnaires, qualitative feedback instruments, interview)
Does the product need to be nearly complete to conduct usability testing?
Testing can take place in any stage of product development, but the earlier the better. Paper prototypes, clickable prototypes, and wireframes are just some of the products that work well for early reaction and interaction with your users.
If you wait until the product is at Alpha or Beta, you will likely find that it is too expensive and too time consuming to address the issues users uncover. When you test earlier, you can build user experience into product design. And when you test often, you can continue to refine and confirm the improvements in design, based on the results of prior testing
Does usability testing need to take place in a lab?
Lab testing provides a formal setting for the moderator and participant to interact, with a separate room for observers to see and hear everything with the ability to talk to each other without affecting the testing session.
But, with today’s state-of-the-art portable lab equipment, it is easy to set up testing equipment at any location that suits the situation. You may not need the overhead and formality of a usability lab to get great results.
Is remote usability testing an option?
Remote testing is a great option which allows you to increase the reach to users around the United States and around the world. We frequently combine remote testing with in-person testing. We also regularly conduct studies solely using remote testing. In a remote testing session, you get to record and observe the participant’s interactions with the interface and hear everything the participant says.
What are the usability test deliverables?
Test deliverables are typically, but not exclusively, the following:
- All materials for each user (completed screener, pre-test questionnaire, post-task questionnaires, post-test questionnaire)
- DVD recording for each user
- Spreadsheet of log for each participant
- Summary notes from findings meeting
Other deliverables we can provide on request include:
- Executive summary report
- Formal report with analysis
- Illustrated formal report with screen captures/callouts
- In-person presentation of findings with PowerPoint and video clips
- Video highlights of top findings
What's the difference between a usability test and a focus group?
If you’ve never done usability testing before, you may assume it’s like a focus group. But it isn’t where a focus group brings together a number of people to share their thoughts about a product or what they might like in a product, usability testing is one participant at a time, showing you what is happening as the user interacts with the product. You learn what users like and don’t like because they are “thinking out loud” as they perform typical tasks with the product.
Is usability testing the only research option?
Usability testing is such a game-changing experience that it is often everyone’s preferred source of insights into your user’s experience.
But it is one of many tools in our UX toolkit, including:
- Site visits (contextual inquiry) to learn about users, tasks, and environments
- Expert review (heuristic evaluation) to inspect a product or interface against a set of rules or guidelines
- Card sorting to gain insight into information architecture and naming strategies
- Surveys to broaden the reach to your users and gain feedback
- Diary studies—for qualitative, longitudinal insights into your users’ experience over days or weeks
- Training to educate your staff on the benefits of a user-centered design process and to give them the tools to conduct user testing