ChatGPT in UX research

If you've found your way to this blog, you're no stranger to the vital role that user experience research plays in shaping product development and design choices. So what happens when artificial intelligence tools, such as ChatGPT, enters the realm of UX research?

In a recent conversation with Daniel Moon, a senior UX researcher at the Chamberlain Group, we delved into the evolving landscape of UX research and the growing concerns surrounding the use of AI, particularly ChatGPT, to answer surveys. 

In this article, we highlight the concerns raised by Daniel and explore the potential impacts and opportunities of AI tools being used in UX research.

Concerns about ChatGPT in UX research

UX researchers often rely on surveys to gather valuable insights into user needs and behaviors. Within this area of research, there’s always the risk of receiving careless responses, however Daniel’s concerns focus on survey participants who use tools likeChatGPT to automate their responses. The motivation? To quickly complete surveys for financial incentives. 

"It's a relatively new area and so we don't really know too much yet. But if more people start to implement ChatGPT in surveys, it will start to muddy the waters of our data," he adds.

With the potential increase in AI use in research responses, there comes a new set of challenges, such as skewed data, compromised data accuracy, and distinguishing AI-generated responses from human responses.

Data accuracy

One of the main concerns with participants using ChatGPT to automate survey responses is the increase in inaccurate or biased results. As Daniel elaborates, "The main purpose of surveys is to gather the thoughts, opinions, perceptions, attitudes, and beliefs of people. If we have somebody ask a machine, we're not really getting all that from a person. This is why it makes it difficult for us to be confident in our claims based on what we're shown."

ChatGPT and other AI tools are unable to accurately capture customer sentiment and emotions due to their reliance on predetermined scripts and language models.

Additionally, ChatGPT (at the moment) isn’t able to keep up with rapidly changing trends in UX research, as it can only draw upon data from previous interactions and user behavior. With participants using the tool to create answers, it can only use this previous information, causing bias. 

Daniel believes that if this trend continues, “it will negatively affect our ability to make confident claims from our results because it will be difficult to distinguish machine-generated versus human-generated responses.” 

Elevate your research practice

Join over 320,000+ marketers, designers, researchers, and product leaders who use Lyssna to make data-driven decisions.

Sign up for free

Sign up for free

Sign up for free

Distinguishing AI-generated responses from human responses

As AI tools continue to evolve, it will only become harder to be able to differentiate between AI and human responses, which could make it difficult to trust the integrity of the data.

For example, say you’re looking through the results of a survey and one response is overly detailed, perfectly formatted, and written in impeccable language. This response raises your suspicion that it’s AI-generated, but could it be a well- thought out answer?

As Daniel notes: “What happens when you accuse somebody of writing in an AI-generated format when it was really them giving a thoughtful response – they’re very deliberate with words and are very well spoken?”

This will be an ongoing battle for UX researchers going forward, discerning between authentic and AI-generated responses to ensure the reliability of their data.

Anticipating the impact of AI-generated answers in UX research

The potential impact of ChatGPT on UX research is multifaceted. While Daniel hadn't observed a significant rise in AI-generated answers in his work, he believes it could be a precursor to a detrimental trend. 

Daniel continues, "It's just a matter of time, because if researchers don't address this or they don't do anything about it, and people who did it once start to realize they can do this again, then what's to stop others from eventually coming to the same conclusion? It enables them to do it because nothing was done about it when the warning signs were first there."

How to spot ChatGPT responses in your UX research

Spotting ChatGPT responses in your UX research can be challenging, but there are a few indicators that can help you identify them. 

Firstly, pay attention to the language and tone of the responses. ChatGPT often produces overly formal text, lacks personalization, or exhibits a consistent pattern. Look for generic phrases or responses that seem too perfect or robotic. Another way to spot AI-generated responses is to look for overly similar answers between participants who took the survey at different times. If there are almost identical responses from more than one person, there’s a good chance an automated system was used. 

Additionally, examine the coherence and relevance of the answers. ChatGPT may generate responses that are unrelated to the question or fail to address the specific context of the research. 

Finally, consider the timing and speed of the responses. ChatGPT can generate answers rapidly, so if you notice an unusually quick response time, it could be a sign of AI-generated content. By being vigilant and analyzing these aspects, you can better identify and mitigate the impact of ChatGPT responses in your UX research.

Opportunities for using AI in UX research

There are plenty of AI tools on the market with powerful capabilities designed to save you time, increase efficiency, and provide valuable insights into user behavior. 

“I think that when it comes down to the impact of AI, we've talked about the bad, but there's also good,” shares Daniel. “I would say the two main areas in which I think it could be good are digesting and summarizing result findings, and automating workflow.” 

Data management

The use of AI-powered platforms, like HeyMarvin, can help you better manage and archive research data. 

Platforms like this, "can help us store and easily access research data, making it more digestible. This would prevent us from reinventing the wheel and save time for researchers," states Daniel.

“That's where I can see AI being very beneficial because I'm sure many organizations have tried to accomplish something similar, where they're able to share their research and have people who want to conduct future research look back and see what was already learned before, potentially doing essentially the same thing,” adds Daniel.

Automate workflows

The second key opportunity that Daniel mentions is the ability to automate workflows, specifically internal recruiting. The use of AI-driven platforms that store contact information and enable personalized email communication can streamline the self-recruitment process. This can help improve response rates and enhance the relationship between researchers and participants.

An example of a workflow for internal recruiting that Daniel shares: “You’re sending out an email to ask if anyone is interested in providing feedback on a project. You can set up a workflow that, once someone responds with a yes, an event is triggered to respond with an automated, personalized email along with creating a calendar invite for both you and the other person.”  

“Something like that would allow you to streamline the ability to get that connection. All I’ve got to do really is just show up to the meeting. All the admin work starts to become taken care of by itself,” he continues. 

There’s also the use of chatbots that can assist in screening and scheduling participants for interviews or usability testing. This saves you time and ensures you select the right participants.

Inspiration for writing questions

There’s value in using ChatGPT for user research questions, especially if you’re a newer UX researcher who is just starting your career. As Daniel explains, "I think that it would be very helpful for newer UX researchers who are entering the field and don't really understand what it is supposed to look like."

When starting out, many new researchers rely on examples and templates to structure their research questions. ChatGPT can play a crucial role in providing guidance and framing questions effectively. It can act as a valuable tool to bridge the knowledge gap, assisting in crafting questions that are clear and concise, addressing specific framing challenges, and ensuring research questions meet the desired standards.

Future of UX research with AI

While concerns regarding the use of AI in UX research, particularly in survey responses, are valid, there’s also potential for AI to be a valuable tool. Striking the right balance and implementing safeguards to ensure the authenticity of responses will be essential as AI continues to evolve in the field of UX research.

As Daniel eloquently puts it:

"With the right approach, AI has the power to enhance data management, streamline recruiting, and support newer UX researchers in framing effective survey questions. As technology advances, the UX research community must adapt, learning how to leverage AI without compromising the integrity of their findings."

Your go-to user research platform

The best teams use Lyssna so they can deeply understand their audience and move in the right direction — faster.

Sign up for free

Sign up for free

Sign up for free