CUE-10 study debates the pros and cons of moderation style in usability testing

A recent CUE study (Comparative Usability Evaluation 10), led by Danish UX researcher Rolf Molich, brought 16 UX researchers together in Boston to debate the pros and cons of moderation style.  The study participants each had to record three moderated usability tests of the budget European airline, Ryanair.com, and then view the others’ recorded sessions and analyze what makes for an effective moderation style.

If the moderator spectrum has “talkative” on one end and “quiet” on the other end, where does your moderation style fall?

 

Talkative moderators engage more frequently in a conversational style with participants, asking more questions and probing for more insights during the session.

Quiet moderators are likely to be less chatty and take less time with meeting and greeting.  They still cover all the required preliminaries, but they do so in a more reserved way.  Their engagement with the participant can be viewed as an active listener.

Turns out, the CUE study analysis shows that we were fairly well divided on the moderator spectrum between quiet and talkative.

Are you talkative or quiet?

Which style are you? The style you use to greet the participant and set the stage for the usability testing very likely reflects your moderation style.

When you conduct the briefing at the start of a usability testing session, how do you get the participant comfortable with the process of thinking out loud?  What do you tell the participant about your role as the moderator?  If you sit with the participant in a face-to-face test, where do you sit in relation to the participant--next to the person, slightly behind the person, or across from the person?

Talkative moderators are likely to be chatty from the start, as they meet and greet the participant and get them comfortable.  

Quiet moderators are likely to be less chatty and take less time with meeting and greeting.  They still cover all the required preliminaries, but they do so in a more reserved way.  

Coaches are actively engaged.

Talkative moderators are likely to be using a coaching style in which they actively engage with their participants.  For instance, they laugh with the participant when something funny happens, empathize when the participant experiences problems, remind the participant how valuable their reactions and thoughts are, encourage them during the process to continue sharing their feedback, and generally strive to make the participants feel like they are collaborators in the process of discovery of their experience. 

Active listeners are mostly quiet.

Quiet moderators are likely to be engaged in active listening.  They don't say a lot during the session, but they indicate interest in the participant by using good body language (leaning in, for instance) and taking good notes to show interest. If the participant makes a comment or asks a question, they might reply with a nonverbal response (um hum) or a short verbal response (tell me more….)

"The goal of usability testing is to get viable, actionable results based on unbiased, effective moderation."

Is one moderation style better than another?

Maybe.  According to a recent research study that compared three moderator styles for the same tasks, the coaching condition participants had higher completion rates for tasks than participants in the other two conditions (active listening and silence).  The same was true for the number of users who gave up before completing a task.  Those in the coaching condition gave up the least often.

However, participants in the coaching condition took more time to complete tasks than those in the other conditions, which isn’t surprising, given the greater engagement between the moderator and participant.

As far as satisfaction ratings, participants in the coaching condition expressed higher satisfaction with the website being tested compared to participants in the active listening condition.

And here’s another plus for the coaching style, based on a different research study: participants were more satisfied with the experience of being a participant when the moderator used the coaching style over the quiet style.  Better results, happier participants.

Sounds like the coaching style is better all around.

But wait.  There can be too much of a good thing, and this is true in the coaching style of moderation.  The moderator can get carried away, becoming too chatty, too conversational, too invasive in the testing session, thereby skewing the results of testing by interspersing too many questions and interruptions.

That’s where judicious silence can be effective.

Intentional silence has its place.

Most of us feel awkward when silence occurs in the flow of conversation. A Dutch study showed that  it takes only 4 seconds of silence for someone to become uncomfortable in a conversation.

As a moderator, are you inclined to rush in when your participant goes quiet or is slow to respond to your question or pauses in mid-sentence?

If so, you should count to four or maybe even five or six before saying something.  Resist the temptation to finish the participant’s sentences.  Give your participants time to think.   While you are waiting, keep your body language positive to indicate that you are listening and engaged.

Moderation is an art, not a science.

Whichever style is yours, you need to work at your craft to perfect it. Moderators are not born; they are made.

That means a novice needs to learn the rules before he or she can break them.  So, what are the rules?

Best Practices when Moderating Usability Tests

  • Make the participant feel comfortable and valued.
  • Set the stage for what will be happening, why, and how, but provide flexibility.
  • Engage with the participant enough to show your interest in their actions and reactions.
  • Work with a discussion guide to follow a structured set of activities/scenarios.  That generally means that you should start the scenario and then get out of the participant’s way.
  •  Focus on your goals for the study: Is it more important to complete all of the tasks/scenarios or allow for a deep dive in any particular scenario? How much control do you need to impose regarding time on task and completion of the session?
  • Novices in learning mode need to follow these rules with whatever structure (a script, for instance) is required to be competent and consistent in each test session.

 

If/when mastery is achieved, moderators can engage in poetic license to do what works best for them to engage with the participant at the appropriate level and uncover unbiased findings from the study.

Finally, it should be noted that some people do not make good moderators, no matter how much they practice.  If you find yourself in the wrong role and you have other team members to call on, switch roles to another one that fits your personality and temperament better.

After all, we probably all can agree that the goal of usability testing is to get viable, actionable results based on unbiased effective moderation.

CUE-10 is the tenth in a series of Comparative Usability Evaluation (CUE) studies. Previous studies were conducted from 1998 to 2011. For an overview of the nine previous CUE-studies and their results see www.dialogdesign.dk/CUE.html

 

Avatar

Carol Barnum

Carol brings her academic background and years of teaching and research to her work with clients to deliver the best research approaches that have proven to produce practical solutions. Carol’s many publications (6 books and more than 50 articles) have made a substantial contribution to the body of knowledge in the UX field. The 2nd edition of her award-winning handbook Usability Testing Essentials is now available.