Case Studies

How politically persuasive can AI be? Kobi Hackenburg, The Alan Turing Institute, and Prolific

October 16, 2024

Introduction

Few issues occupy so many human minds right now as AI. From being a fringe area of study even ten years ago, the field has exploded in size and level of public interest. And for good reason.

The scale and scope of recent advances have been spectacular, with AI being deployed at scale and in the mainstream. We’re only just starting to feel this rapid societal shift, but the implications are staggering.

Kobi Hackenberg is a PhD candidate at the Oxford Internet Institute, as well as a member of the technical staff at the UK AI Safety Institute. His cutting-edge research explores how people and AI interact. Now that AI systems can have more human-like interactions with people, can they be used to change people’s minds on key subjects?

“As the sort of text generated by these models becomes more compelling and more human-like, concerns have been raised that this could cause issues for the political process, where certain actors would co-opt these models and use them to generate large amounts of very compelling political texts,” Kobi explains. “If these texts were able to influence human attitudes, this could cause problems for politics where voter autonomy would be challenged.”

When he was a research assistant at The Alan Turing Institute, Kobi wanted to test how persuasive AI-generated messaging could be. This led him to Prolific.

The goal of the research

With a frontier this unstudied, goals have to be broad. Often, the goal is simply to define the landscape, to help researchers design further studies.

Kobi:

“People were really concerned, I think understandably, about the ways that AI could be used to influence political outcomes and the way campaigns could use them now.”

A particular focus for Kobi and the team was trying to understand the relationship between the size of a large language model (like Chat GPT) and how persuasive it can be on political issues.

The challenges

The team were asking a big question. And they faced big challenges in defining and finding a data set. Costly, too - because to get a robust measure of persuasiveness, they needed variance in both the issues addressed, and how they're addressed.

As Kobi explains: “To plot a scaling curve of how persuasiveness increases with as a function of size, you obviously need many models or many data points along that curve.”

They needed so many models and so many data points, scale was of the utmost importance to the study. Otherwise, the design wouldn’t be robust enough. They needed a large number of participants to hit statistically significant thresholds.

Put simply, without enough participants, there would be no point in carrying out the work at all.

The solution Prolific provided

Kobi and the team at The Alan Turing Institute needed something which would give them access to verified and vetted participants at scale, but without incurring the sort of costs that this sample size has been known for in the past.

This is when they turned to Prolific.

Easy and rapid access to a large and 100% verified pool of participants was key, and that’s exactly what Prolific provided. It’s not just scale though - Prolific was able to provide precise targeting and pre-screening of participants to create the exact mix that the team required, and which could be monitored by researchers all through the project.

Results and key findings

A key question in this study was that of the scale of the large language model involved in designing political messaging and material, versus how persuasive it was to participants.

“I think the top line finding was that there were diminishing returns,” Kobi said. “As we got closer to these frontier models, these very, very large models, the marginal increase from increasing the size of the model was smaller and smaller.”

This reflected initial assessments, especially when relating to short, punchy political messaging. The AI was limited to eight-sentence messages which gave limited scope to play with.

 “It seems like intuitively there should be a ceiling on how persuasive these short messages can be,” Kobi noted. “So in that sense, I think it was something that myself and other co-authors were broadly expecting.”

Recognizing predictive successes is necessary, but in this case acknowledging the limitations of the study was too:

“It's important to note that these sorts of relationships might be different for a more prolonged multi-turn dialogue, where the model does have more room to really adapt to participants' responses and maybe employ more advanced persuasion strategies.”

So what next for Kobi and the team?

“Scaling this research to a multi-term domain is something that we're working on actively right now,” he said. “Doing similar research, but in a more dynamic, prolonged fashion where participants are going to be able to converse with the models instead of just reading previously generated messages. That’s something I'm really looking forward to.”

If you need access to participants at scale for your AI research, Prolific can help. We make it easy to source samples of all sizes from our 200k+ active respondents, so you can get tasks completed honestly, accurately, and almost instantly. Find out more.