LLM use in research: A study into mitigation strategies
AI is changing the nature of research. It’s a challenge faced by everyone in the online research community.
A recent study explored how AI is affecting crowdsourcing platforms. It looks at how we can keep researchers one step ahead of the challenges AI presents - and how they can embrace its opportunities.
In this study, participants on the Prolific platform were asked to write a summary of a large chunk of text. The method required to do this was hard for humans - but easy for Large Language Models (LLMs).
Two distinct strategies were tested to reduce the number of participants using LLMs.
In this approach, participants were explicitly asked not to use LLMs.
For this strategy, hurdles were created to deter participants from using LLMs. The original abstract text was converted to an image, or copy-pasting was disabled entirely.
Findings showed that:
The methods weren’t perfect. And there were some challenges in the analysis.
LLM-based tools and how people use them are evolving fast.
It would be a mistake to define LLM use as foul play for every study run online. In some research, LLMs assisting crowd workers might even be beneficial. Also, as more people use LLMs to complete everyday tasks, the distinction between ‘synthetic’ and ‘human’ data will get increasingly blurred.
So, rather than focus on shunning LLMs from crowd work, it’s worth investigating how researchers can embrace them. There are many ways for crowd workers to use LLMs. And different approaches may have different effects on the output of the research.
The truth matters in research. The best decisions - and biggest discoveries - are built on the highest-quality data.
Every online research platform has to manage rapid change brought on by AI. But we’re working hard to help people navigate that - through this study, and new platform features.
We’ll soon launch a new addition to the Prolific platform that will help us better detect participants who use LLMs. Based on our own machine learning models and informed by participant behaviour, this feature is 96% effective.
And we have more in the pipeline to help researchers deliver world-changing work.
You can read more about the mitigation methods tested as part of the study in the full paper.
Fresh out of YC's Summer 2019 batch, we want to share some of our most interesting learnings. If you're a startup founder or enthusiast and want to learn about product-market fit, growth experimentation and culture setting, you're in the right place!
One viral TikTok video, 30,000 new participants, and one month later - here's what we've learned and what we're doing about it.