If you’re relatively new to usability testing, you may not have come across the debate of formative versus summative usability testing. Even if you’re a seasoned UX veteran, it’s easy to miss some of the nuances between these two research methods. 

This guide will help you get a deeper understanding of formative usability testing – what it is, why it’s different from summative testing, how to do formative testing, and overall best practices. 

Understanding formative usability testing

There’s a clue in the name: ‘form’-ative usability testing helps to form and influence design decisions early in the development process. 

The goal of formative testing is to identify potential design problems as early as possible, both for a more efficient product development process as well as saving you some time later down the road trying to fix issues that were preventable. 

What’s the difference between formative and summative usability testing?

You can almost consider summative testing to be the opposite of formative testing – while you conduct formative testing early in the design process, with summative testing, you do it after designing the product. Another difference between these methods is that formative testing most often uses qualitative research methods, while summative testing most often uses quantitative methods. 

Another way to distinguish between formative and summative testing is to think of formative testing as ‘bottom up’ and summative testing as ‘top down’. These terms, commonly used in academic research, refer to different approaches in research methodology and philosophy. 

As you might guess, ‘bottom-up’ research is more explorative and lays foundations before trying to pin down any kind of theory. It helps with generating ideas and putting together potential solutions for moving forward. 

A ‘top-down’ approach looks at using ‘established’ theories and ideas and testing against them. With usability testing, this refers to the scenario of testing an established design to see how users interact with it.

Elevate your research practice

Join over 320,000+ marketers, designers, researchers, and product leaders who use Lyssna to make data-driven decisions.

Sign up for free

Sign up for free

Sign up for free

Formative usability testing methods

Before you decide to start a formative usability testing study, you first need to determine if it’s the right way to go. To do that, simply ask yourself whether you’re working on a new design or redesign, or evaluating the usability of an existing product, feature, or service. If you’re evaluating an existing product, you should move forward with summative testing first. 

For the others, let’s do formative! We’ll look at some of the main usability testing methods you can use; namely interviews, focus groups, design surveys, and prototype testing. Each method has its own pros and cons, so it’s important to evaluate which one works best for the goals of your study.

Interviews

Interviewing existing or potential users about your product or service is a common approach for formative usability testing. It helps you get super detailed feedback and you can ask ad hoc questions easily when you need to. The difficulty with interviewing is that it can often be both time-consuming and expensive – few people will volunteer their time for free. It can also be more difficult to get a representative sample of users.

What can help make interviews more cost-effective is doing them remotely over the phone or via video call, which means neither you nor the participants need to go out of the way to meet somewhere (reducing organization time and the need to reimburse travel costs). People are also more used to doing interviews in this manner since Covid-19 – for example, while it’s a different field of research, 41% of market researchers used online in-depth interviews regularly towards the end of 2020, and a further 33% said they used it occasionally.

If you're looking for a remote user interview solution, check out Interviews from Lyssna.

Focus groups

For the same benefits as interviews, but with more people, you can look at running a focus group. An additional benefit of using focus groups is that participants often bounce ideas off each other, which encourages deeper discussion or forming better solutions. Besides the same drawback of time and cost, focus groups can come with an additional issue: groupthink. 

The term groupthink refers to a “mode of thinking in which individual members of small cohesive groups tend to accept a viewpoint or conclusion that represents a perceived group consensus, whether or not the group members believe it to be valid, correct, or optimal.”

So if you decide to run focus groups, you’ll need to be especially careful of individuals falling into “groupthink” mode and encourage diverse opinions. 

Design surveys

At the early–middle stages of the product development process, you can use design surveys to help narrow down design choices. Design surveys are a great method of testing because they can be either quantitative or qualitative (or both at the same time!). 

However, with formative testing, you’ll want to use design surveys in a more qualitative manner. For example, instead of using Likert scales and multiple-choice options, you can build in text-based responses to get answers in the participants’ own words. 

The main downside to using this method is that the more effort participants need to put in, the less likely they are to want to complete the survey. Open-ended question surveys had a lower completion rate (83%) compared to multiple-choice questions (89%) according to SurveyMonkey (n=25,000).

Watch this video to see how to set up text-based or design surveys in Lyssna.

Prototype testing

The final method we’ll cover in this guide is prototype testing. Similar to design surveys, you can do prototype testing in both a formative and summative manner — the only real difference is the timing and level of detail in the prototype.

For formative testing, you’ll be working with early-stage or low-fidelity prototype designs, likely static pages, or even paper-based prototypes of design concepts (for in-person testing). To help make early-stage prototype testing run more smoothly, it can be better to run them as moderated tests. This way, the moderator can help guide the participants through the designs and explain how not-yet-functional elements of the design will behave.

Moderators will also be able to get more in-depth responses to usability testing questions based on the prototype and how the participants might interact with a more complex version.

Want to try out your low-fidelity Figma designs? Our template for evaluating early prototypes can help you get started.

Watch this video to see how to set up a prototype test in Lyssna.

Formative usability testing best practices

Now you’re aware of the different methods you can use, let’s cover some best practices for conducting formative usability tests. 

Ask the right interview questions 

As demonstrated in the SurveyMonkey research, the more open-ended questions you include in your tests, the lower the completion rate is likely to be. This is slightly different if you’re holding in-person or remote interviews, but still bear in mind that the longer the interviews go on, the energy participants have will drop. As someone who regularly participates in research interviews, I’ve found around 1 hour to be the sweet spot.

Because of this, you need to make sure you’re asking the right questions. Otherwise, you’re wasting your (and your participants’) time. Here are a few checklist items to help you slim down your list of questions:

• Remove leading questions, e.g. “How good was your experience in doing X task?”

• Remove double-barreled questions (two questions in one), as they cause confusion and can make analysis more difficult.

• Remove questions with biases, i.e. questions that include personal opinions or perspectives. 

• Remove leftover/irrelevant questions that don’t directly support the aim of the usability test. 

Avoiding these types of questions will help make your tests more reliable and give them better accuracy/validity. 

Encourage participants to think aloud 

If you decide to conduct in-person or virtual interviews/focus groups, you’ll need to make sure you encourage participants to say what they’re thinking, i.e. “There are no wrong answers.” This can help you gather even more data, but again, be mindful of potential groupthink, or social desirability bias behavior. 

You can choose to either account for these effects in your analyzes, control them, or attempt to reduce them during testing.

Don’t provide too much guidance during the tests

Another best practice for formative testing is to not provide too much guidance — sure, you’ll need to explain that certain parts of your early-stage prototype don’t work yet, but if you tell the participant exactly what they need to do, you won’t get any meaningful results.

To make sure you’re not overly guiding your participants, give them enough time to respond or perform a task. Take note of how much time participants need and include this in your analyzes. 

Conduct a pilot study first

An easy way to make sure your tests will run smoothly is to conduct a pilot study, recruiting colleagues or friends to take part. This can help reveal any issues with either the technology, the workflow, or the questions you ask. Doing a pilot study can also help you figure out the feasibility of the tests and how much time you’ll need to analyze the results from a larger-scale version. 

Pilot studies also have the benefit of being cheap and quick to complete, especially for formative tests, where you may only need one participant to assess the test. 

Get started with formative usability testing

Formative usability testing is the best approach when you’re exploring ideas for a new product or overhauling an existing one. However, that doesn’t mean summative testing isn’t valuable. As you may have guessed by now, UX research teams should be in a cycle of formative and summative testing, never fully relying on only one approach. 

With the information we’ve given you in this guide, you may feel inspired to start a new UX research study using formative testing. If so, check out the features available on Lyssna and sign up for a free plan to get access to unlimited active tests!

Your go-to user research platform

The best teams use Lyssna so they can deeply understand their audience and move in the right direction — faster.

Sign up for free

Sign up for free

Sign up for free

Frequently asked questions about formative usability testing

What is formative usability testing?

What’s the difference between formative and summative usability testing?

What’s the goal of formative usability testing?

--

Alexander Boswell is the Founder/Director of SaaSOCIATE, a B2B SaaS, MarTech and eCommerce Content Marketing Service and a Business PhD candidate. When he’s not writing, he’s playing baseball and D&D.