Testing in a COVID-protected usability lab
Before COVID-19, I would often conduct face-to-face usability testing in a conference room setting, typically in a Regus Suites location, which rents rooms of various sizes by the hour or by the day. This isn’t a viable option for me in the current pandemic, however, because I would not be able control so many variables with participants coming and going throughout the day.
To get ready for face-to-face testing, I decided to check out what market research/usability lab locations were doing to address COVID concerns for both participants and researchers. My first study was scheduled for mid-June, after shelter-in-place restrictions had been lifted in some states, including Georgia, but most other restrictions were still in place.
Where could I find a usability testing facility that would make me and the study participants feel safe? The first to open under CDC (Centers for Disease Control and Prevention) guidelines in Atlanta was Jackson Associates (offices in Atlanta and Los Angeles). Soon afterward, Schlesinger Group announced rolling openings around the U.S. that would be adhering to CDC guidelines to protect the health of participants and UX researchers. As the months of the pandemic roll by and the need for in-person testing continues, there are likely to be more testing facilities adhering to these guidelines.
I signed with Jackson as their first client using their usability lab under the new COVID-19 practices and procedures.
Adhering to a protocol to keep everyone safe
The protocol was impressive. I expected to wear a face mask, use hand sanitizer often, and keep my social distance. All of that was, of course, in place.
But so much more was also done to keep everyone safe. Among the practices were these:
- Acrylic shields separating the participant, the logger, and the moderator from each other
- Acrylic shield and masked greeter at reception desk
- Electronic, touch-free check in of participants
- Temperature check for all on arrival
- Touch-free hand sanitizer stations throughout the facility
- Individually boxed meals and pre-wrapped, packaged snacks, bottled water
- Personal silverware packet for the UX team to be used throughout testing
- Individualized notepads with personalized hand sanitizer included
- Gloved and masked staff wipe downs of all surfaces and products used after each session, using the most advanced products, including Micro ban 24
- UV sanitation wand available for use by the staff and UX team, as needed
Build it, but would they come?
We felt safe using the facilities under these strict protocols. But would participants? Because this was a medical device usability study as part of research for an FDA pre-market 510(k) submission of a new medical device, the study had to have 15 participants in each group.
As an extra precaution in recruiting, Jackson scheduled more backups than usual: we had 5 extra backups for each of two participant groups, making the recruit total 40 participants.
Because we were the first client doing face-to- face research in their lab, Jackson didn’t know what the no-show/cancellation rate would be for the participants. But they had done their research so they had a pretty good idea. In two surveys to their panel to take the pulse of their panel members—the first in April/May and the second in early June—they learned that their panel’s confidence in their safety precautions and willingness to participate in studies was strong.
Their study covered markets in Atlanta and Los Angeles with panel participants representing consumers (n=5570) and health care professionals (HCP) (N=881). Here’s what they found:
- Consumer confidence in in-person testing went up 10% from the May to June surveys
- Men were slightly more confident than women in participating (83% for men; 79% for women)
- 84.2% of consumers and 93.2% of HCPs said they would likely participate in an in-depth interview/usability test.
In general, those under age 60 were significantly more likely to participate in-person than those over 60. This comfort level has increased significantly between May and June in the 30-49 year-old age groups.
That all sounded promising in terms of recruitment for our study, so how did it go? Very well! There were only 3 participant cancellations over the entire study, and we were able to cancel the last several sessions (paying the participants, of course) when we knew we had reached our required participant counts for each group.
In a second study we conducted in August for a different medical device, one in which the preference was for participants over 60 years of age (presumably a difficult group to recruit in the pandemic) we had NO cancellations. It appears that people are ready, willing, and eager to participate in research studies if the conditions are safe. And they were.
We will be going back for more studies in October, and we expect that the participant recruiting will go as well as it has to date.
In-person testing is safe if the conditions are right
Not many usability companies are willing to do face-to-face testing. I know this because of my conversations with several colleagues who do not intend to go back to in-person testing until we have a vaccine.
This article is not about trying to convince anyone who is uncomfortable to change their position on in-person testing. Rather, it’s to let you know that if you are interested in conducting in-person UX research, you can do it.
My UX colleague Cory Lebson has written a 2-part blog about testing face to face in a conference room.
I’ve described how to do it in a testing facility that adheres to the types of guidelines I experienced in testing in Atlanta.
My intent in sharing what I have experienced is to keep the conversation going. As with everything in UX, “it depends” on how you feel and what you are willing and able to do in support of UX research. If you think it can’t be done in person during the pandemic, I’m here to tell you that it can. If you want to give it a try, you should find out which facilities in your area are ready to support your interest by providing protocols to keep everyone safe.