UX Firm: Your Atlanta-based solution for usability research and testing.

Microsoft’s Product Reaction Cards Unlock User Satisfaction. Part 3

By on Sep 13, 2016

This post is part of a 3-part series on using Microsoft’s Product Reaction Cards in usability testing.


In Part 1 we presented the background on how Microsoft developed the product reaction cards as part of a “Desirability Toolkit.” In part 2, we shared the ways in which we use the cards in our studies.  In this part, we provide examples of how we present the results in our reports.

Show and tell the results

The product reaction cards are a powerful tool for gathering qualitative feedback from participants in a single usability study and as a measure of improvement in iterative studies.

In a single study, we typically present the results in a word cloud, which quickly shows the words chosen most often. Here’s an example from a study of 23 participants:

complex word cloud

 

 

 

 

 

 

We often include a table of the positive and negative words chosen with a count of how many of each word was selected by each user.  For example, the table below shows a partial list of the words presented in the word cloud, indicating the number of times it was selected and whether it was deemed a positive or negative word.

Word # Categorization
Useful 9 Positive
Familiar 8 Positive
Clean 4 Positive
Expected 4 Positive
Rigid 4 Negative
Unrefined 4 Negative
Approachable 3 Positive
Connected 3 Positive
Innovative 3 Positive
Integrated 3 Positive
Intuitive 3 Positive
Slow 3 Negative
Sophisticated 3 Positive
Straight-Forward 3 Positive
Usable 3 Positive
Valuable 3 Positive
Accessible 2 Positive
Annoying 2 Negative
Attractive 2 Positive
Busy 2 Negative
Clear 2 Positive
Comfortable 2 Positive
Compatible 2 Positive
Consistent 2 Positive
Essential 2 Positive
Exciting 2 Positive
Hard-to-Use 2 Negative
Impersonal 2 Negative
Ineffective 2 Negative
Not-Integrated 2 Negative
Organized 2 Positive
Practical 2 Positive
Predictable 2 Positive
Relevant 2 Positive
Understandable 2 Positive

 

Cards are great in small studies

You don’t need big study numbers to get great results from using the product reaction cards.  Here are the cards from a single study with 6 participants in a day.

The same powerful consistency results.

 word cloud

 

 

Cards are great in iterative studies

We can also use the words to establish a baseline for user satisfaction in the first study, which can then be used in iterative studies to track improvements in user satisfaction as the product goes through development or redesign.   The example below is from 3 studies of a web-based application for hotel properties worldwide to implement and monitor green initiatives.

User testing of the client’s first version demonstrated that the general idea of the application was motivating; however, the product had significant problems that slowed or stopped users from achieving success. Participants’ repeated positive card choices were low with only comprehensive, professional and usable selected twice each; but the themes of quality, appearance, ease-of-use, and motivation emerged from participants’ card choices.

The second study was of the prototype of the redesigned application. The transformation of the users’ experience was night to day. The positive card choices from 12 users now represented 82% (compared to 42% in the first study) with the most often selected card being useful.

 

word graphic

With such a positive and significant measurement of change, the development team focused on the remaining issues, and a small test of the pilot version was conducted with 4 users just before launch. The pilot version results showed that all participants chose only positive words.  The theme of speed now predominated and confirmed that the earlier negative issue of slow speed was now a positive feeling of fast speed—the application was fast, time-saving, and efficient.

Become a card dealer

You can do this.  Using the product reaction cards is easy and the results are powerful.  What’s more, Microsoft has given us permission to use them with only the following disclaimer:

“Developed by and © 2002 Microsoft Corporation. All rights reserved.”

You don’t have to do fancy diagrams to show the results.  You can just count the number of times each word is chosen and present these results.

Or use Wordle or another free tool to turn the word counts into a cloud.

Then you can do some quick clustering of similar words to present themes emerging from the words users chose.

And, of course, use these results along with your other feedback mechanisms, such as The System Usability Scale, to compare your results from different perspectives.  By triangulating your data from different sources, you strengthen your case for understanding what users showed you about their experience.


This post is part of a 3-part series on using Microsoft’s Product Reaction Cards in usability testing.

Read Part 1
Read Part 2
Read Part 3