How researchers ran a 3-week study on AI sycophancy with 1,364 participants using Prolific

AI often tells us what we want to hear. It can flatter us, validate our beliefs, and encourage our intentions, even when they're harmful or irresponsible. Sycophantic AI is a well-documented phenomenon, but what does it do to our real-world relationships over time? Do people still want messy, complex human interactions when they can turn to a frictionless chatbot whenever they want?   

This is a topic that fascinates Lujain Ibrahim, a PhD student at the University of Oxford. Her research explores how AI systems affect society and shape our beliefs, judgments, and relationships. She recently co-authored a paper on sycophantic AI, which shows how AI sycophancy changes the way people relate to friends and family. 

The research involved thousands of participants and 12,766 human-AI conversations across five studies. Using Prolific, Lujain and her team were able to recruit the participants they needed, manage them with ease across a complex longitudinal study, and gather evidence that sycophantic AI can reshape our closest relationships.  

The challenge: How to study AI sycophancy at scale, for weeks  

Most research into AI sycophancy has looked at why it happens or how it affects people in short, one-off conversations. Lujain and her research team wanted to explore how it changes relationships and well-being over a longer period. 

"We wanted to know how sycophancy affects users over several weeks,” Lujain explains. “Specifically, how it affects the way they socialize with others, how they feel about the people around them, and how they feel about the AI systems they're interacting with." 

This project would involve recruiting thousands of participants to take part in five studies. Participants were asked to chat with different AI models for up to 20 messages, with each model set to behave with varying degrees of sycophancy. Participants were then asked questions about their experience, how they felt about what they’d discussed, and how it shaped their offline lives.  

Successfully running research at this scale would require a reliable source of thousands of high-quality participants, specifically a census-representative sample of English-speaking participants from the US. The fourth study in particular posed some challenging requirements - participants would need to talk to an AI model about personal topics every other day for three weeks. 

This meant finding a provider that could deliver both scale and quality: a large enough sample of engaged, representative participants for the five studies, and the tools to engage, manage, and pay them over three full weeks. For that, Lujain turned to a platform she already knew well. 

The solution: A platform built for scale and quality

Lujain has used Prolific for the past four years, across most of her PhD. Before this, she had used MTurk for two papers involving human-subject studies, but then switched to Prolific to get higher-quality data, and has continued to collect data for most of her published studies since.

With Prolific’s global pool of 300k+ vetted participants and intuitive API, Lujain and her team were able to access representative samples with thousands of participants and manage them across the three-week study.

Built-in representative samples

For the longitudinal study, Lujain recruited 1,364 US-based adults using Prolific's representative samples, which balance participants by biological sex, ethnicity, and age. This built-in tool allowed the team to get a census-matched US sample without any additional costs.

Automated participant management via API

The three-week longitudinal study was the hardest of the five to run. It depended on bringing the same 1,364 participants back every other day for three weeks, across 12 sessions, retaining as many as possible along the way. 

Prolific’s API made managing participants at this scale much easier. The team used the API to automatically approve or reject sessions, send participants reminder messages, and track who had completed each stage of the study.

Messaging and bonuses that scaled with the research

Beyond the API, Lujain used Prolific's messaging function to communicate with participants, debug issues, and respond to feedback. The bonus payments feature also proved to be extremely useful in the longitudinal study, where she needed an easy way to administer extra payments to participants who returned over multiple weeks.

Automatic checks for data quality at scale

With thousands of participants across five studies, Lujain used Prolific's auto-reject feature to flag participants completing sessions "inhumanly quickly," alongside her own pre-registered attention checks. 

Together, these excluded a very small share of participants in each study, screening out low-effort submissions without requiring manual checks of every session. "That was a good, easy data-quality check," she says.

The results: Evidence that sycophantic AI narrows the space for real relationships

Thanks to Prolific, Lujain and her team were able to run all five studies successfully, including the demanding three-week longitudinal design, with strong retention throughout. 84.3% of participants completed all 12 sessions, with just 15.7% dropping out over the full three weeks. 

Across the five studies, sycophantic AI immediately delivered the emotional and esteem support people typically associate with close friends and family. Over three weeks, that translated into something more consequential. Participants were nearly as inclined to seek personal advice from their AI as from those closest to them, driven by a sense of being understood rather than by any real increase in certainty.

But this came at a cost. Participants in the sycophantic condition reported lower satisfaction with their real-world social interactions, with no decrease in time spent with others. The effect tracked how far sycophancy narrowed the emotional gap between AI and humans, rather than how good the AI conversations felt on their own. And feeling understood by the AI didn't translate into feeling more understood by real people.

The final study showed that a preference for sycophantic AI isn't just a default that people passively accept. Given a choice between a sycophantic, neutral, and challenging AI, 54.6% chose to keep talking to the sycophantic AI, a majority that held regardless of topic or the order they tried each model.

"People didn’t choose the sycophantic model because they thought it was higher quality”, Lujain explains. “They felt it understood them best, and it was the easiest to talk to. That gets at the real mechanism. People choose sycophantic AI because of a powerful feeling of being seen, understood, and heard."

Together, these findings suggest that by providing frictionless understanding, AI may gradually raise the bar against which we judge human relationships. 

What's next: How to build models that are better for people

The findings of this study have a number of practical, real-world applications. A better understanding of what makes sycophantic AI so compelling can help pinpoint concrete ways to intervene and mitigate its harmful effects. It also means we can be more cognizant of other features or model behaviors that might have the same impact. 

“For example, models now let you turn on memory, so you get personalized models that know a lot about you,” says Lujain. “They can give a similar feeling of being seen and understood, without the friction that's normally part of human relationships. This could create a similar dynamic to what we saw in our studies.”

Looking ahead, Lujain is interested in exploring how we can build models that are better for people. “It’s the question of how you give people a pleasant experience that’s also good for them and their human relationships, without trading one off against the other,” she says. “Studying over-reliance is something I’ve been thinking about too.”

"That tension - building models that people like, but that also account for people's long-term preferences, well-being, and satisfaction - is core to dealing with sycophantic AI systems." 

Easily recruit and manage participants for complex studies

Looking to run complex studies like this one? Prolific gives you instant access to 300k+ verified participants, as well as the tools you need to collect high-quality data at scale. Get started today.