This post is part of a 3-part series on using Microsoft’s Product Reaction Cards in usability testing. In Part 1 we presented the background on how Microsoft developed the product reaction cards as part of a “Desirability Toolkit.” In Part 2, we shared the ways in which we use the cards in our usability studies. In this part, we provide examples of how we present the results in our reports.
Show and tell the results
The product reaction cards are a powerful tool for gathering qualitative feedback from participants in a single usability study and as a measure of improvement in iterative studies.
In a single study, we typically present the results in a word cloud, which quickly shows the words chosen most often. The image at the top of this blog post is an example.
We often include a table of the positive and negative words chosen with a count of how many of each word was selected by each user. For example, the table below shows a partial list of the words presented in the word cloud, indicating the number of times it was selected and whether it was deemed a positive or negative word.
Cards are great in small studies
You don’t need big study numbers to get great results from using the product reaction cards. Here are the cards from a single study with six participants in a day.
The same powerful consistency results.
Cards are great in iterative studies
We can also use the words to establish a baseline for user satisfaction in the first study, which can then be used in iterative studies to track improvements in user satisfaction as the product goes through development or redesign. The example below is from 3 studies of a web-based application for hotel properties worldwide to implement and monitor green initiatives.
User testing of the client’s first version demonstrated that the general idea of the application was motivating; however, the product had significant problems that slowed or stopped users from achieving success. Participants’ repeated positive card choices were low with only comprehensive, professional and usable selected twice each; but the themes of quality, appearance, ease-of-use, and motivation emerged from participants’ card choices.
The second study was of the prototype of the redesigned application. The transformation of the users’ experience was night to day. The positive card choices from 12 users now represented 82% (compared to 42% in the first study) with the most often selected card being useful.
With such a positive and significant measurement of change, the development team focused on the remaining issues, and a small test of the pilot version was conducted with 4 users just before launch. The pilot version results showed that all participants chose only positive words. The theme of speed now predominated and confirmed that the earlier negative issue of slow speed was now a positive feeling of fast speed—the application was fast, time-saving, and efficient.
Become a card dealer
You can do this. Using the product reaction cards is easy and the results are powerful. What’s more, Microsoft has given us permission to use them with only the following disclaimer:
“Developed by and © 2002 Microsoft Corporation. All rights reserved.”
You don’t have to do fancy diagrams to show the results. You can just count the number of times each word is chosen and present these results.
Or use Wordle or another free tool to turn the word counts into a cloud.
Then you can do some quick clustering of similar words to present themes emerging from the words users chose.
And, of course, use these results along with your other feedback mechanisms, such as The System Usability Scale, to compare your results from different perspectives. By triangulating your data from different sources, you strengthen your case for understanding what users showed you about their experience.
This post is part of a 3-part series on using Microsoft’s Product Reaction Cards in usability testing.