Remote Moderated Usability Testing

Remote moderated usability testing primer

What is remote moderated usability testing?

Remote moderated usability testing is a testing technique in which you meet your participant in real-time but not in the same place. You engage your participant in typical tasks using your product (software, website, mobile device, app) and ask them to think out loud so that you can understand what they are thinking as they do the tasks. You then interview them in a semi-structured interview format to gain insights into their experience, both the things that worked well and the things that did not.

When should you conduct remote moderated usability testing?

Anytime is a good time to conduct remote moderated usability testing, and the more often you do it during product development, the better. Designing products with user experience built in is called an “iterative design” process, which means that you respond to the results of usability testing throughout the design and development process, making the product a reflection of good user experience.  Testing early at the prototype stage or focusing on a single feature is a great use of remote moderated usability testing. But it’s also effective to conduct remote moderated usability testing for current products and even competitor’s products to gain insights that support customer requirements for the next version or release of your product. 

So many tools (most of them free for first-time or limited use) support your ability to conduct remote moderated usability testing.

• Using collaborative meeting software such as Zoom or Teams, you can conduct individual testing sessions with your participants from the convenience of your workspace and their home or workspace.
• Using a calendar program such as Calendly, you can make your testing schedule available to your prospective participants, and they can choose the session time that works for them.
• Using social media or your website for recruiting, you can invite prospective participants to participate by completing a screener in SurveyMonkey or Google forms.
• Using Amazon, you can set up gift card payments for their participation.
  
  

Remote moderated usability testing doesn’t take the time or cost of in-person, in-the-same-place moderated usability testing, but it still requires planning to go off well.

Here’s what you need to know to plan for success:

• Start recruiting right away: Recruiting takes the most time in planning, so you want to start recruiting as your first activity. If you are recruiting yourself, you can schedule participants as they become available. More often, though, you need to schedule your complete study within a block of time. Small studies of 5 – 8 participants can be accomplished in a few days to a week or longer. So, start your planning with a plan for recruiting participants. This plan includes how you will find them, screen them for eligibility, and pay an incentive for their participation. If you don’t have the staff resources to recruit yourself but you have the budget resources, you can work with a recruiting company to do the recruiting for you. These companies typically require two – three weeks to set up, screen, and confirm the participants.
• Create a test plan: Every usability test needs a test plan, but remote moderated usability testing needs some additional components in the test plan to support success. The test plan needs to include information on how you will provide the information (tasks, instructions, interface access, consent forms) to the participant. To put the participant at ease in the remote context, your introduction will likely be a bit more detailed than is needed in face to face in person testing.
• Prep your participants for their session: This task may require several communications (email or text, depending on the preference of the participants). You need to let them know they have been selected, confirm their session date and time, and specify the general plan for the session. If they need to use a specific browser on their computer or laptop or their smartphone, if they need to download some software for the study, if they need to review and sign a video consent and NDA (non-disclosure agreement) before testing, if they will be expected to have their camera on during the session recording, if they will be asked to take mouse control during the session, you need to provide them with all of this information. If they are uncomfortable or unfamiliar with any of these technology requests, you may want to do a dry run with them before the session to confirm that they can access the session and do what’s expected. This may be more relevant for some user populations than others.
• Invite observers: Providing the opportunity for your team or relevant stakeholders is a great way to share the benefits of usability testing with everyone who has an interest or a need to understand your users’ experience. When you invite your team to your sessions, remind them of a few ground rules, such as being on mute, not having their camera on, and using Chat or another messaging app like Slack or What’s App to communicate with you and other observers during the session. Of course, when observers are in the meeting, you will want to let your participant know. I typically say something like, “We also have some observers from the company, as we are all very interested in learning about your experience today.”
• Conduct a pilot session: It’s always a good idea to “test the test.” In a pilot session, which you can conduct with a team member who stands in for the participant, you want to practice working with your moderator’s guide to go over the logistics of how you will provide the tasks to your participant and manage their access to and engagement with the product interface.
• Will you read each task to the participant, or put the task into the Chat for the participant to read or access via links to each task and scenario created in Google docs or something similar?
• Will they need to be able to access instructions as part of the study? If so, how will you make the instructions available?
• Conduct the testing session: Get into the meeting software at least 5 minutes ahead of the scheduled start and invite your observers to do the same. You want to be ready when your participant arrives. Make small talk for the first minute or two, while you also express appreciation for their taking the time to engage with your product in the session. Confirm that they can access the product and are able to get mouse control, if that’s required. Review the video consent form and confirm that they are comfortable allowing the session to be recorded.  Then start the recording. Then, using your moderator’s guide, go through the study components for the tasks and post-task questions. When you have finished, these, engage in the post-test interview, then thank the participant again, review how the incentive will be handled, and remember to stop the recording. 
  

Remote unmoderated usability testing is conducted without a moderator, making it asynchrononous, meaning that the test session can happen whenever the participant is available because there is no moderator needed. The deliverable is the recording, which you and your team can review afterward.

In contrast, remote moderated usability testing is synchronous, meaning that the participant and the moderator are in the testing session together at the same time. Because remote moderated testing allows for engagement of the moderator with the participant while they are performing the tasks in the study, it provides the opportunity for the moderator to probe for insights based on observations and to ask questions “off script” from the moderator’s guide, as the situation presents itself.

Remote unmoderated testing does not provide for this interaction, but the advantage to remote unmoderated testing is that it can happen at any time without the requirement of coordinating schedules with participants and moderators. Several software platforms provide the tools for conducting remote unmoderated usability testing, which include fast recruiting from the platform’s panel of potential participants and fast setup of the study specifics using the platform’s tools. 

Platforms such as UserTesting and Lookback generally require a monthly (or annual) fee (although some allow for a free trial). If this approach fits your user research goals, you can use one of these platforms to get quick insights into user experience about a feature or task on one or more platforms, such as a laptop and a mobile device. If you don’t get a definitive answer to a research question, you can set up another study just as fast to dig deeper or learn from a wider pool of users. You can also get a much larger number of sessions conducted in a short period of time if you want to produce quantitative findings. 

Keep in mind that testing sessions are typically short, usually around 20 minutes. And there is a risk that some test participants in the panel may not be a good match for the limited number of screener questions you can ask.

How to choose the right testing method?

It comes down to what best fits your situation.

Moderated remote usability testing has the advantage of engagement with your participants and the flexibility to ask questions in the moment. But it takes time to recruit and schedule participants, not to mention scheduling your time for these sessions.

Unmoderated remote usability testing has the advantage of speed and scale. But you get what you get: a recording of the session without the chance to ask followup questions.

One method is not better than the other. Know what you want to get out of remote usability testing and pick the method that best matches your goals. The main objective is to do usability testing so that you can understand your users and build that knowledge into better user experience.