17 Oct 2024
|11 min
Summative usability testing
Discover the importance of summative usability testing in UX design and research. Explore methods, benefits, and best practices for obtaining valuable insights.
Depending on where you’re at in the product development process, your UX research will fall into either of these categories: formative or summative.
As explained in our formative usability testing article, there may be some debate about which type of testing is better. However, there’s no doubt that using both methods is good practice for successful UX design and research.
This guide will dive into the details of summative usability testing, including:
What is summative usability testing?
What’s the difference between summative and formative usability testing?
When you should use summative testing
Benefits of summative testing
Summative usability testing methods
Summative testing best practices
What is summative usability testing?
Like formative usability testing, there’s a small clue in the name – ‘sum’-mative testing helps to test the ‘sum’ of the product, i.e. its overall design or the big picture. The goal of summative usability testing is to figure out how well a current iteration of a product is working compared to previous iterations, or against competitors in the market.
What’s the difference between summative and formative usability testing?
The main difference between summative and formative usability testing is the stage of product development at which you use these testing methods (more on this in a moment). Another big difference between them is that summative usability testing tends to be more on the quantitative side, whereas formative testing typically uses more quantitative methods of data collection and analysis.
As explained in our formative usability testing article, you can think of formative testing as a ‘bottom-up’ approach and summative testing as a ‘top-down’ approach.
When should you use summative usability testing?
You should use summative usability testing when you have an existing product or highly-developed/complex prototype to test, i.e. at the end of the product development process, before the cycle begins again. The main reason for this is because of the testing methods used for summative evaluations – they typically need an existing product or usable prototype for the tests to work (though there can be exceptions).
However, another reason is that summative usability tests often work best when there’s a large sample size – which means a product with an established customer base will likely see the best test response rates.
Benefits of summative usability testing
There are a few obvious benefits to doing summative testing, including:
They’re generally cheaper than formative testing methods. That being said, the cost efficiency of summative testing depends on the scale of the tests you do. If you need thousands of global responses, this could end up costing you more than a series of 1:1 in-person user interviews.
They’re easier to scale. If you have the means for it, summative testing methods are far easier to scale than formative methods – you create the test once and then send it out to however many participants you need.
They’re nearly always quicker to complete. You can reasonably receive results from a summative usability test (especially when you do them remotely) within a day or two, as opposed to needing to source potential interviewees and organize for interviews to take place.
While I mention the comparison to interviews as a method (for formative testing), you can also use interviews as a summative testing method, providing you ask particular kinds of questions – I’ll cover this in more detail shortly.
Summative usability testing methods
As you might have guessed, many summative testing methods are unmoderated and you can do them remotely. Here we’ll look at some common methods and what makes them particularly useful.
Card sorting
An example of a card sort in Lyssna
If you need to test the information hierarchy of your site, such as item categories on an ecommerce store, a card sorting test is a great option. With a card sorting test, you can ask your participants to organize items into product categories that make sense to them. Another use case for card sorting can include figuring out how your audience would categorize support documentation or helpful information in a knowledge base.
Card sorting also has the benefit of being highly visual, and you don’t need to use a lot of mental effort to complete them. When tests are easier to complete, you have better chances of getting higher completion rates.
First click tests
The results of a first click test in Lyssna, shown as a heatmap
Another way to test your information architecture or the effectiveness of your CTAs is using a first click test. These tests measure the usability of a website, app, or design by finding out how easy it is to complete a task. For instance, you could ask your participants, “Where would you click to see a list of your saved content?”. You could then ask follow-up questions about what drew your participants to click where they did.
First-click tests are important for helping you figure out how your audience interacts with your product or website. The First Click Usability Testing study by Bob Bailey and Cari Wolfson found that if the first click was the correct option, users had an 87% chance of completing the action correctly, as opposed to 46% if the first click was incorrect.
Five second tests
An example of a five second test in Lyssna
Remember how I said most summative methods are quantitative? Well, five second tests are one of the few unmoderated, remote testing methods that are more qualitative (that being said, you could also do this test in person).
A five second test measures what your audience recalls about your product or webpage in the first five seconds of exposure. An example question here would be, “Please write down all the main elements you remember from the page you saw.”
From the responses you get, you can analyze the results using word cloud visualizations to see if there are any trends (both expected and unexpected).
Interviews and/or focus groups
The final methods you can use (that I’m detailing in this post) are interviews and focus groups. Again, this method is more qualitative, and it mostly suits formative usability testing, but you can still do summative-based interviews and focus groups.
For instance, I recently took part in a focus group for a utility company that wanted to figure out how people would react to changes in its compensation criteria (for bad service). This session was very much summative as there was already an existing set of criteria, but they were testing potential changes.
While interviews and focus groups can be slow and expensive, if you’re looking to make big changes where the wrong decisions could cost the business even more in the long run, they can be worth handing the cash over for those deep insights.
Summative usability testing best practices
It’s all well and good to know what kinds of testing methods are at your disposal, but it’s another to know how to run summative usability tests in the best possible way to get meaningful results. Here, we’ll look at a few summative testing best practices to help you hit the ground running.
Ask the right questions
We have an in-depth guide on usability testing questions that covers what to ask throughout the product development process, however it’s easy to steer off the path of summative testing by accidentally asking questions that are better suited to early discovery work.
For example, the questions you’d ask in a formative usability test would be along the lines of:
“What type of product would you use to complete X task?”
“Can you give an example of when you would use X type of product?”
For summative testing, you need to get more specific about using your product/website. For example:
“How would you describe the language on this page?”
“What features do you find most valuable about this product?”
“Can you name any competitors that you currently use to do X? [related to existing feature]”
These questions can help you narrow the scope of your research and make sure you’re not doing extra, unnecessary work during your analysis.
Avoid giving hints to participants
While providing some guidance during formative usability tests is okay (e.g. a participant tries to click on a non-functional part of a prototype design), you want to avoid this practice during summative testing.
The reason for this is to make sure participants take the paths that are natural to them. By doing this, for example, during first click tests, you reveal whether your existing information hierarchy has any issues you need to address in later product updates.
Thoroughly plan and prepare
I said earlier that with some summative testing methods you can build the test, send it out, and get responses within a matter of hours (or even minutes!). However, don’t let this potential for quick turnarounds get in the way of good planning and preparation.
On this subject, David Watkins, Director of Product Development at EthOS, explains:
“By carefully planning and preparing for the testing process and approaching it with an open mind and a willingness to learn from feedback, you can gather valuable insights that can help to improve the user experience and drive the success of your product.”
He also gives some practical tips on preparing to conduct summative tests by saying:
“It would help if you first started by defining clear objectives, selecting appropriate participants (based on personas if they are available), creating relevant tasks, and selecting a straightforward platform that will be easy for your staff and for participants to use.”
Start using summative testing for your UX research
Summative usability testing is a vital component of the product development process. Just because you’ve launched your product doesn’t mean the work is finished! There are always improvements you can make. As you gain feedback from customers and practice a continuous product discovery mindset, you’ll give your product the best chances of success in the long run.
Your go-to user research platform
The best teams use Lyssna so they can deeply understand their audience and move in the right direction — faster.
Frequently asked questions about summative usability testing
--
Alexander Boswell is the Founder/Director of SaaSOCIATE, a B2B SaaS, MarTech and eCommerce Content Marketing Service and a Business PhD candidate. When he’s not writing, he’s playing baseball and D&D.
You may also like these articles
Sign up to our newsletter
We'll keep you updated with the latest UX insights and more.
