Using Measurable Criteria for User Testing

In the workplace, people usually use measurable criteria in their user tests when they have one of the following goals.

• To determine whether a draft communication is "good enough" to be used as a final draft. For example, an employer may use measurable criteria to find out whether draft instructions for a new product are good enough to be distributed with the product.
• To determine whether a new version of a communication is superior to the version now in use. For example, an employer may want to learn whether a new version of its online help is more usable and persuasive than the current version.
• To determine whether the employer's draft is superior to a comparable communication by a major competitor. For instance, an employer that sells products online may want to find out how well its current website stacks up against a major competitor's website.

Achieving any one of these goals requires comparisons. To determine whether a draft is "good enough," you must compare its performance against a predetermined standard. To determine whether your draft is superior to another version or a competitor's communication, you must compare the two. These comparisons can be made informally. In the workplace, however, employers often want formal comparisons that rely on numerical data.

Numerical Data for User Tests

Communication experts Jakob Neilsen (2001) and John S. Rhodes (2000) identify numerical data that can be used to evaluate a communication's usability and persuasiveness. They divide the data for measuring usability into two groups.

Effectiveness Measures. These measures describe the accuracy with which users are able to complete the tasks that the draft is supposed to enable them to perform. Examples:

• Percent of the task or tasks the test users can complete
• Number of errors test users make
• Percent of the communication's content or features test users use
• Ratio of successful actions to failed actions
• Percent of the material test users are supposed to learn that they actually learn
• Percent of the material test users are supposed to apply that they can correctly apply

Efficiency Measures. These measures describe the speed with which users can complete their tasks. Examples:

• Time (minutes or seconds) to complete a task
• Time to learn what the draft is supposed to enable the test users to learn
• Time to apply the knowledge is supposed to enable the test users to apply
• Time spent on errors
• Time needed to correct errors
• Number of times test users make the same error
• Number of times users need to ask for help

Neilsen and Rhodes identify a third group of data that can be used to measure a draft's persuasiveness.

Persuasiveness Measures. They measure the draft's influence on the test users' attitudes.

• Test users' rating on a numerical scale of the draft's usefulness. Examples:
• Test users' ratings of their satisfaction with the draft
• Test users' ratings of the draft's credibility
• Test users' rating of their attitude toward the topic (e.g., product, procedure) of the draft

Choosing which Data to Gather

In any particular user test, it would be a very large task to gather and analyze all possible measures of effectiveness, efficiency, and persuasiveness. Instead, chose the measures that will give you the clearest idea of how well the draft achieves your communication's objectives (see Chapter 3).

Presenting Measurable User Test Results

When presenting the results of user tests involving numerical data, Nielsen recommends using percentages. Here are some examples.

• Test users made 45% fewer errors with the new draft than with the existing communication.
• Test users completed the procedure 60% faster than the speed we established in advance as our criterion.
• Test users reported 20% greater satisfaction with our draft website than with our competitor's website

A Final Note

While they can answer certain practical questions about how well a draft communication works, user tests that rely on numerical data cannot help you determine how to make your draft more effective, efficient, or persuasive. See Chapter 15 for ways to design user tests that produce ideas for improving a draft.

Works Cited

Nielson, J. (2001). Usability Metrics. Retrieved January 3,2007 from http://www.useit.com/alertbox/20010121.html.

Rhodes, J. S. (2000). Usability Metrics. Retrieved January 3, 2007 from http://www.webword.com/moving/metrics.html.