Diagnosing virtual patients with a classification algorithm

The challenge

Carrying out studies with doctors can be very challenging—they’re busy professionals with competing priorities. This can make finding physicians to participate in research difficult, especially if you’re trying to recruit internationally. 

That’s why researchers at three German universities turned to Prolific as part of their international recruitment strategy for help when they began looking for doctors to take part in an experimental validation study. The research team needed to find participants to help evaluate a new diagnostic method, and used Prolific to:

• Recruit Domain Experts with verified medical expertise
• Find participants from around the world
• Invite them to take part in a virtual experiment
• Reward them for their time

A new diagnostic tool for pain

Chronic pain places a significant burden on healthcare systems, with researchers estimating it accounts for around 40% of all primary care consultations. But pain is complex, with a wide range of symptoms and underlying causes. In the study, the researchers wanted to evaluate the effectiveness of a new tool for diagnosing chronic pain. 

Around the world, countless medical practitioners use the ICD-11 disease classification standard, which is developed by the World Health Organization (WHO). Physicians access it via a browser, and it helps them narrow down and diagnose all known health conditions by entering patient symptoms. In 2021, researchers in Scotland developed the Classification Algorithm for Chronic Pain (CAL-CP), a decision tree intended to improve on the ICD-11 browser’s current pain diagnosis process. 

The German researchers wanted to verify whether the CAL-CP decision tree would indeed be useful for doctors diagnosing chronic pain, and whether it would be more effective than the standard ICD-11 process. 

To do this, they designed an online study where doctors would interact with four virtual patients suffering from various kinds of chronic pain. The ‘patients’ were chatbots that had been pre-programmed to answer questions related to their conditions. For two of the patients, the doctors would use the ICD-11 browser for diagnosis. For the other two patients, they would use the CAL-CP algorithm. 

The solution

The researchers needed to recruit a variety of medical practitioners to take part in the study, interact with the ‘patients’ online, and give their diagnoses. Prolific supported the research team with: 

Participants with strong English skills

The study required all participants to have strong English skills, since they would need to interact with the ‘patients’ via a chat tool. Prolific’s participants tell us which languages they speak during the sign-up process, which allows researchers to filter out anyone without the required abilities. 

Participants who can go off-site

As part of the study, participants needed to visit the researchers’ online portal to take part in the experiment. Unlike other crowd worker sites, Prolific allows you to invite users to third party platforms.  

Building a representative sample

Thanks to Prolific’s large, global crowd of more than 200,000 individuals, the researchers were able to create a diverse, representative sample. Their pool included practitioners from multiple countries, a range of ages, and years of clinical practice (from final year medical students to those with many years of experience).  

Register with Prolific and to start searching for research participants for your next study. 

The results

The study confirmed that the CAL-CP algorithm does help doctors to more efficiently and accurately diagnose the causes of chronic pain. Using the algorithm led to significantly more correct diagnoses than using the ICD-11 browser: for example, correct diagnoses for chronic primary pain reached over 75% with CAL-CP.. 

Prolific made an important contribution to this study, helping the researchers build a representative, international sample of verified Domain Experts. Using Prolific allowed the team to find participants far more quickly than traditional sample building methods (such as contacting medical associations or calling up GPs), while also being much less costly than using medical research recruitment agencies. 

CAL-CP is intended as an enhancement to the WHO’s ICD-11 browser, which is a resource for doctors around the world. Prolific’s ability to find verified experts internationally was also invaluable, as it helped to gather insights about how users in different countries would interpret CAL-CP. 

Now available: Verified Domain Experts

Does your study require respondents with advanced knowledge or experience? Prolific’s pool of Domain Experts includes highly educated participants with backgrounds in diverse fields, including medicine, finance, law, IT and beyond. We have a robust screening process to ensure all our Domain Experts hold genuine qualifications and relevant experience, and we help researchers seek them out. 

Register with Prolific and begin searching for research participants, or contact our Sales team for more information. Already a user? Find out how to use Domain Experts. 

Citation: Hay, G. et al, (2024), Clinicians diagnosing virtual patients with the classification algorithm for chronic pain in the ICD-11 (CAL-CP) achieve better diagnoses and prefer the algorithm to standard tools: An experimental validation study: https://onlinelibrary.wiley.com/doi/full/10.1002/ejp.2274

Research institutions: Marburg University, University of Duisberg-Essen, Münster School of Business