Omni Marketing Interactive - Designing for People who Search Home    
  Site map  
  Contact us  
Services Clients About Omni FAQs Design tips Links and resources Request a quote
     Home > Design & UX tips > Website usability tips > User Experience Smackdown: Usability Testing Vs. User Testing   
Tips overview
Search engine optimization
Website usability
Information architecture
UX website design
Omni photo - top
Photo - tips
Omni photo - bottom

User Experience Smackdown: Usability Testing Vs. User Testing

What's the difference between user testing and usability testing? UX/Usability pro Shari Thurow explains the distinction.

Usability testing vs. user testing image

Throughout our user experience (UX) careers, we have encountered many people who honestly, truly believe that their company conducts usability tests on a regular basis. But when we probe for details, we quickly learn the truth.

In This Article

Usability testing vs. user testing

For example, we were hired to produce wireframes and advanced prototypes for the health section of a major brand’s website. After conducting the appropriate card sort tests to determine the optimal global, local, and contextual navigation, we presented our search-engine friendly, usability-tested wireframes to the new health manager.

The health manager essentially trashed all of our wireframes and test results because she claimed to have regularly conducted A/B and multivariate tests on another health website with great success.

We have seen the end result over the years. Here’s a summary:

  • Incorrect information architecture: The site’s information architecture and corresponding navigation system is based on date and pagination. In reality, patients and patient caregivers do not organize and health information by date and page number.
  • Improper supplemental navigation. Supplemental navigation is organized “By Topic” — but topics are not organized in a way that makes sense to users. The topic labels are paginated, and topic labels are not presented in any particular order. Heck, even alphabetizing the topics would make them easier to locate and discover.
  • Search rankings. Not surprisingly, the health site does not rank well at all (on Google and Bing) for most of their targeted keyword phrases.

In another scenario, we were at an event where we provided site reviews for usability and the user experience (UX).

One attendee claimed that her company worked with multiple firms that offered user testing.

However, when we looked at the website, it only had one page. The site had no site navigation…only a form to fill out. The page did not provide any benefit to users for filling out the form.

We realized the problem: the term user testing. Somehow, throughout the years, the term user testing has become synonymous with usability testing.

Usability testing and user testing are not the same thing. Just because a person or company conducts A/B tests, multivariate tests, and focus groups does not mean that the company is a usability firm.

That previous sentence is a rather bold statement, isn’t it?


List of usability tests

Usability is about contextual task completion. Website usability isn’t about one’s personal opinion. Website usability is not about a “coolness” or “wow” factor. It’s not about forcing your personal beliefs or design preferences onto users. It’s about task completion.

The ISO defines usability as:

"The effectiveness, efficiency and satisfaction with which specified users achieve specified goals in particular environments."

User satisfaction is directly related to task completion If users can complete their assigned tasks quickly and easily, they often report higher satisfaction, especially if there is an element of delight in the interaction.

If users have a difficult time completing their tasks or cannot complete them at all, they often report low satisfaction.

To determine user mental models and the effectiveness of a website or application, usability professionals conduct ongoing usability tests. We don’t use the term testers when we refer to usability test participants.

Reason? What is being evaluated is the website or application, not the person participating in the usability test. If a test participant has a difficult time completing a task, the problem lies in the interface. Perhaps the labels are confusing. Perhaps the functionality is inconsistent from page to page, screen to screen.

Below is a short list of usability tests that professional usability firms typically conduct:

Brand perception test

  • Measures: Brand message
  • Formative or Summative: Formative
  • Notes
    • Identifies the feelings and attributes associated with a website or application.
    • Done with mock-ups of the visual brand.
    • Generally a document that asks test participants to circle the adjectives that best describe each mock-up.
    • Another variation is to show test participants two screenshots. Of the two, which is more (the following adjectives are examples):
      • Professional, reliable, dependable
      • Cool, hip, cutting edge, state-of-the-art
      • Sophisticated, polished, professional
      • Warm, fuzzy, friendly
    • Should be done separately from performance tests.


Closed card sort test

  • Measures: Organization of content; whether or not navigation labels make sense to users; user mental models
  • Formative or Summative: Formative
  • Notes
    • Show test participants a website’s pre-defined navigation or a group of categories.
    • Ask test participants where they would go to find ________________.
    • Focus on:
      • Number correct
      • Amount of time to answer
      • Confidence in answer
    • Often used after an Open Card Sort Test to confirm whether or not the navigational structure is self-evident.
    • Another version of this test is called a Tree Test or Treejack Test.
    • Usability/UX professionals also monitor where test participlants clicked (or tapped) first, which is similar to a First Click Test.


Expectancy test

  • Measures: Initial mental model
  • Formative or Summative: Formative
  • Notes
    • Early prototype test that evaluates users’ initial mental models of an interface.
    • The facilitator instructs test participants, “Without using the mouse, keyboard, or hand, please tell us what you think [this] does.”
    • The facilitator can also ask, “What would you expect to find under each category?”
    • Accept all test participants answers without indicating whether or not the answer is “correct”.



  • Measures: Actual usage
  • Formative or Summative: Formative
  • Notes
    • Usually used on a detailed design.
    • Answers the questions:
      • What attracts and holds users’ attention?
      • How long did they focus on a specific area of the screen?
      • What is the order in which users looked at items on a screen?
    • Evaluate what test participants scan vs. what they read.
    • Research shows that where test participants look depends on what you have asked them. Please see Eyetracking Studies – 7 Traps to Avoid.


5-second usability test

  • Measures: Initial mental model, aboutness, keyword focus
  • Formative or Summative: Formative
  • Notes
    • Used to evaluate individual content pages, not home pages or category pages.
    • Present a scenario and task to test participants.
    • Show test participants the web page for only 5 seconds.
    • Remove the page it by either covering it up or switching to another window.
    • Ask participants to write down everything they remember about the page.
    • When they finish jotting down their recollections, ask 2 or 3 useful questions to assess whether or not users accomplished the task.


Free exploration test

  • Measures: Actual usage
  • Formative or Summative: Summative
  • Notes
    • Use when navigation is not an issue.
    • Observe users at work on a detailed website design or application.
    • For example: “I will give you ~five minutes to freely explore this website. You may go anywhere you like on the site. Please remember to speak aloud as you do so. I will tell you when the five minutes are up.”


Functional salience test

  • Measures: Importance of functions
  • Formative or Summative: Summative
  • Notes
    • Give test participants a list of possible functions.
    • Instruct participants to only select 3 functions that are the most important to them.


Open card sort test

  • Measures: Organization of content
  • Formative or Summative: Formative
  • Notes
    • Test participants are asked organize a list of items into categories that make sense to them.
    • Test participants actually name and describe the categories.
    • See the Open Card Sort Example below.


Performance test (navigation)

  • Measures: Navigation design
  • Formative or Summative: Formative
  • Notes
    • Observe and listen to test participants as they interact with your website or application.
    • Assign test participants a task with a scenario. Indicate:
      • 2=Completed
      • 1=Difficult or close
      • 0=Fail
    • Identify potential roadblocks encountered and any perceived error corrections.


Performance test (content organization)

  • Measures: Organization of content, layout, task flow, controls and content
  • Formative or Summative: Summative
  • Notes
    • Evaluates the navigation design, labeling, and functionality on an advanced prototype: a working website or application under realistic conditions.
    • Present individual scenarios and tasks to test participants.
    • Measure success rate, time on task, and user satisfaction.
    • See how test participants completed each task.
    • Identify potential roadblocks encountered and any perceived error corrections, though there should be less (or zero) of these in summative tests.


Sample-of-one test

  • Measures: Roadblocks or errors in interface layout, labels, content organization, affordance, and functionality
  • Formative or Summative: Summative
  • Notes
    • A type of performance test primarily used to identify roadblocks and errors. Larry Constantine coined this type of test.
    • Uses only one test participant who fits the primary persona/profile.
    • Test past comfort zone.
    • As with all usability tests, the facilitator must be careful to not ask leading questions or lead the test participant in any way.
    • See how test participants completed each task.
    • Identify potential roadblocks encountered and any perceived error corrections, though there should be less (or zero) of these in summative tests.


Visual signifier test

  • Measures: Clickability/tappability
  • Formative or Summative: Formative
  • Notes
    • Show test participants a web page.
    • Give them a printed version of the web page. Ask them to circle every item on the web page that they believe is clickable/tappable.
    • Option: After circling the clickable items, give test participants another printed version of the web page. Ask them to circle every item that they believe is NOT clickable/tappable.


Open card sort test example

Types of website navigation from the book Information Architecture for the World Wide Web

An open card sort test is often used by both information architects and usability professionals to determine the organization, labeling, and prioritization of content. A skilled usability professional knows how to measure the results. Here’s a good article: 10 Things to Know About Card Sorting.

Why we on a rant about usability testing vs. user testing? We listen to numerous sales pitches. We hear people claim that their site does ongoing testing without specifying the types of tests that are conducted.

We observe people using the wrong methods to collect the wrong data, such as using keyword research tools to determine a website’s information architecture.

Use this list to help you determine whether or not you are working with a true usability professional or organization. Usability is a critical facet of the user and searcher experience.

You can delight or entertain your website users all you want, but if they can’t complete their desired tasks? If they can’t locate or discover desired content? That is not much of an experience.


  • 5-Second Tests: Measuring Your Site’s Content Pages by Christine Perfetti
  • Albert, W., & Tullis, T. (2013). Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Newnes.
  • Bojko, A. (2013). Eye Tracking the User Experience: A Practical Guide to Research. Rosenfeld Media.
  • Card Sorting 101 by Optimal Workshop
  • Constantine, L. L., & Lockwood, L. A. (1999). Software for Use: a Practical Guide to the Models and Methods of Usage-Centered Design. Pearson Education.
  • Dumas, J. S., Dumas, J. S., & Redish, J. (1999). A Practical Guide to Usability Testing. Intellect books.
  • Krug, S. (2014). Don't Make Me Think, Revisited. A Common Sense Approach to Web Usability (ed.). New Riders.
  • Nielsen, J., & Pernice, K. (2010). Eyetracking Web Usability. New Riders.
  • Pirolli, P. (2007). Information Foraging Theory: Adaptive Interaction with Information. Oxford University Press.
  • Rubin, J., & Chisnell, D. (2008).Handbook of Usability Testing: How to Plan, Design and Conduct Effective Tests. 2nd edition. John Wiley & Sons.
  • Spencer, D. (2009). Card Sorting: Designing Usable Categories. Rosenfeld Media.
  • Thurow, S., & Musica, N. (2009). When Search Meets Web Usability. New Riders.
  • Usability Testing 101 by Optimal Workshop
  • Web User Behaviour Directed by Information Scent

This article originally appeared in Marketing Land. It has been updated since its original publication.

Related articles

If you liked this website usability and user experience (UX) article, here are more links to related optimization tips and articles from Omni Marketing Interactive:

You can also read our articles from other online publications.


Information Architecture Services


Website Usability Services

UX design


If you have any questions about Omni Marketing Interactive's search engine optimization (SEO), website usability, information architecture (IA), or web design services, please call us at 847-426-4256.