09 Oct 2024
|14 min
What is a leading question?
Learn how to identify and avoid leading questions in UX research.
If your company or organization has ever tasked you with conducting research, chances are you've heard the phrase, "Make sure you don't have any leading questions!". Or something along those lines. But what if you're not sure exactly what a leading question is and what they look like?
Leading questions are a surprisingly easy trap to fall into if you're not vigilant or simply don't understand the implications of them. So, in this quick guide, we're going to cover:
Understanding leading questions in UX research
Different types of leading questions
Leading questions vs loaded questions
Examples of leading questions
Best practices for avoiding leading questions
By the time you finish reading, you'll have a great understanding of what leading questions are, why you need to avoid them, and common question types to look out for when you're drafting questions for user interviews or surveys.
Understanding leading questions in UX research
The simplest explanation of a leading question is this: it guides respondents to answer in a particular way. Whether you frame the question intentionally or accidentally, you introduce a bias, influencing the way someone answers it.
In UX research, leading questions can have a significant impact on the research outcome.
For example, you might be tempted to ask a question like, "When you use X feature, how useful do you think it is?"
It might sound innocent enough, but this question assumes that the participant has actually used the feature (they might not have) and also influences them to "agree" on some level that it's useful when they might not feel that way – leading you to make conclusions that don't reflect reality.
We'll be looking at more examples of leading questions later, but for now, here are a few reasons you should avoid leading questions in UX research.
They can skew your results and invalidate your study
When your UX research studies use leading questions, you end up skewing the results towards a (generally) more positive outcome, which, as illustrated in the introduction above, often doesn't reflect reality.
As a result, all the effort you went through in preparing for the study, recruiting participants, and analyzing and synthesizing the results is wasted. Invalid studies can then have a greater impact on your research budgets and other resources for redoing the study and conducting future ones.
They can come across as rude or unprofessional
At a minimum, leading questions can confuse your participants and reduce completion rates. In the worst-case scenario, leading questions (particularly coercive ones) can be read as both rude and unprofessional – which can make your participants uncomfortable and unwilling to continue contributing to the research.
It's also not too much of a stretch to say that such leading questions can also lead to negative public relations if any of the participants decide to publicize or complain about them.
They can have a negative impact on brand decisions
Ultimately, the purpose of UX research is to improve your brand's products and services for users. If you include leading questions in your research and don't catch them, you run the risk of producing inaccurate results that your brand uses to make crucial decisions.
Imagine making changes to your product or service based on a survey that used leading questions, only to find that users don't like the changes, and many of them decide to churn and go to a competitor.
These are the main reasons for avoiding leading questions, but hopefully, you now understand how important it is to avoid accidentally using any in your research.
Elevate your research practice
Join over 320,000+ marketers, designers, researchers, and product leaders who use Lyssna to make data-driven decisions.
Leading questions vs loaded questions
Before we discuss the types of leading questions in detail, it's worth noting the difference between a leading question and a loaded question (it's easy to mistake them both or think they're the same).
While a leading question is framed to influence how a participant answers it, a loaded question assumes a fact about the respondent in the question itself, sort of like a trick question – no matter how they respond, they're forced into a response they might not agree with.
An example of a loaded question in UX research would be:
"Have you stopped using X feature?"
Here, if the participant said "no", they're stating they still use the feature; if they say "yes", they're stating they used to use the feature – but if the participant never used the feature in the first place, they can't appropriately answer the question.
Different types of leading questions
Unfortunately, leading questions don't always look the same or have the same effect on the participant's responses. However, they’re a lot easier to spot when you know the different types of leading questions to look out for. There are mainly five, which we'll look at below.
Questions based on assumptions
You got an idea of an assumption-based question in the sections above, and it's worth repeating here – these types of leading questions use an assumption that usually skews the answer in a positive way. They're also often the most similar to loaded questions.
Another example of an assumption-based question that's quite common uses positive emotional language, such as:
"How much do you enjoy using X product?"
The question assumes that the user had at least a positive experience when they might not have. Of course, the participant could respond with, "Actually, I didn't enjoy using it," but social desirability bias may influence the participant to say what they think the researcher wants to hear.
A simple fix to this type of question is to balance out the positive with the negative, e.g. "Was your experience using X product enjoyable or not?" or better yet, "How do you feel about X product?"
Direct implication questions
As the name suggests, a direct implication question implies that answering the question in a certain way will result in a reward or other type of consequence. It's not quite as abrasive as a coercive question, but it still makes the question seem transactional.
For example, let's say you're researching the impact of a loyalty program on a brand, and you included a question like:
"If you liked the introduction of our new loyalty tier, would you like to provide ideas for future rewards?"
This kind of leading question adds unnecessary context, as well as implies that you only want people who liked the new loyalty program structure to contribute new ideas or provide feedback. An easy fix to this type of question is to remove the extra context and stick with the direct question.
Coercive leading questions
Potentially, the worst type of leading questions is coercive – these questions are framed to persuade the participant to answer them in a specific way (usually positive), often with forceful language.
It goes without saying that you should never use this type of question. Thankfully, they're easy to spot because they tend to end with another question of upward inflection (imagine a sleazy salesperson raising their eyebrows with a grin at the end).
For example:
"As a supporter of our brand, you'll recommend us to friends and family, right?"
Or:
"This new home screen design is far better than before, don't you agree?"
Even if you've accidentally written a question framed this way, an easy fix is to use an odd-numbered Likert scale (usually one to five) to give the participant a chance to answer positively or negatively. For example, "On a scale of 1–5, how satisfied are you with the new home screen design, with 1 being Very Dissatisfied and 5 being Very Satisfied?".
Interconnected statements
Similarly to direct implication questions, an interconnected statement uses unnecessary context to influence the participant to answer more favorably when they may not have wanted to.
For example:
"Our most active customers use X feature every time they log on. How many times do you use this feature?
Or:
"Many of our satisfied customers provide valuable feedback after using our product. Have you thought about leaving feedback for our product?"
Both of these examples could cause the participant to feel guilty or uncomfortable about their experiences with your product. However, as with direct implication questions, the easy fix is to remove the context and stick with the straightforward question.
Scale-based leading questions
This type of question is quite tricky and can be easy to miss when you're going over your draft questions because it's based on math.
When you write a question that uses a scale (Likert or otherwise), the options to choose from need to be balanced. Otherwise, you risk skewing the results towards one side of the scale. An example of an unbalanced scale would be:
1 - Extremely satisfied
2 - Very satisfied
3 - Somewhat satisfied
4 - Somewhat dissatisfied
5 - Very dissatisfied
As you can see, there are more positive options than negative ones – leading to an increased probability that you'll gather more positive responses. Thankfully, fixing this question only means having a neutral response in the middle and an even number of positive and negative options on either side.
Examples of leading questions in UX research
We included some examples of leading questions in the section above with explanations of what type of leading questions they were, but in this section, we'll just give you a list of more leading question examples. You get bonus points if you can identify what type of leading question each one is.
"If you liked our recent brand design changes, would you like to participate in future design research for our brand?"
"Many of our customers who enjoyed X product find Y feature valuable. Do you find Y feature valuable for your workflow?"
"How much do you love our community help forum?"
"Our new product update is the best we've ever launched, correct?"
"How much do you enjoy using our product compared to [competitor]?"
"You do agree that X design is better than Y design for our new logo, right?"
"Many of our users report finding our navigation very easy to use. How easy or difficult is it for you to use our site navigation?"
"How satisfying is it to use our help desk service?"
"We appreciate customers who leave reviews of our product on the app store. Would you consider leaving a review?"
"Many of our other customers group our website categories this way. How would you group our website categories?"
These are just a few examples of leading questions that could accidentally make it into UX research studies. If you find it difficult to see leading questions, check out the best practices below.
Best practices for avoiding leading questions
The hardest part about avoiding leading questions is knowing what they are and what they look or sound like – with the knowledge you now have, they should be a lot easier to avoid. However, here are some quick tips to prevent them from ever making it into your draft questions.
Avoid using your own opinion or preferences
The questions you ask should always be objective, even if you feel a certain way about the topic, as this could introduce confirmation bias. If the question requires it, it's the participant's responsibility to add their personal experiences and feelings.
Make sure your question is balanced
If you absolutely need to include emotive language in your question (e.g. satisfaction, enjoyment, usefulness, etc.), make sure you balance it out with equal "or not" phrasing. That way, your participants don't feel compelled to respond positively.
Get rid of any unnecessary modifiers or context
Sometimes, less is more. Make sure your questions are straightforward and to the point regarding context.
Using modifiers is less common in UX research questions compared to other types of research, but you might accidentally find yourself referring to "loyal customers" or "active users." Simply drop the modifier to make the question more balanced and less likely to influence.
Test your questions on other people first
Lastly, read your questions to other people and ask them if they felt like they needed to answer in a certain way. Other UX colleagues are a great option as they'll be more likely to spot a mistake and make suggestions.
However, friends or family can also provide valuable feedback as they're less likely to overthink their response.
Your go-to user research platform
The best teams use Lyssna so they can deeply understand their audience and move in the right direction — faster.
Use Lyssna to help you conduct effective UX research
Now that you know what types of questions to look out for and how to avoid asking leading questions, you can start writing effective user research interview questions or quantitative survey questions.
With Lyssna, you can conduct UX research studies such as surveys, interviews, prototype testing, and more, so you can get valuable insights from your users. For even quicker results, you can use Lyssna's research panel to recruit participants in a matter of minutes or hours rather than days.
--
Alexander Boswell is the Founder/Director of SaaSOCIATE, a B2B SaaS, MarTech and eCommerce Content Marketing Service and a Business PhD candidate. When he’s not writing, he’s playing baseball and D&D.
You may also like these articles
Try for free today
Join over 320,000+ marketers, designers, researchers, and product leaders who use Lyssna to make data-driven decisions.
No credit card required
