Human feedback for Agentic AI training and evaluation.

200,000+ identity-verified humans, filterable by 300+ demographic, professional, behavioural, and credential attributes. General population through Domain Experts, all credentialed and verified. One pool, one API, one audit trail.

Talk to the team

Get started for free

Prolific is the human eval layer for agentic AI

Why Prolific

What Prolific gives an agentic AI team.

A participant pool that spans user to specialist.

Filter 200,000+ identity-verified humans by 300+ attributes — demographic, professional, behavioural, credential. General population through Domain Experts across software engineering, clinical, legal, finance, and research — with range across specialisms and seniority. Cohorts are re-recruitable across runs for longitudinal comparison between model versions.

Preference and feedback at trajectory level.

Pairwise preferences, Likert ratings, and step-level rationale collected over complete agent trajectories — tool calls, intermediate state, multi-turn exchange, and outcome. Captures the process failures that outcome-only benchmarks miss. Suitable for RLHF, DPO, and reward-model training pipelines.

Turnaround that fits a training loop.

Self-serve studies launch in minutes and return first data in hours. Managed red-team and preference programmes deliver in days. No long procurement cycle.

Research · HUMAINE

"Outcome benchmarks miss process failure. Single-turn preference data can't see multi-step reward hacking. The signal post-training needs is trajectory-level judgement — from the right humans."

From HUMAINE: a framework for representative human evaluation of large models. Placeholder pending research-team review.

Read the paper

What you can run on Prolific.

Four workflows across one human data network.

Trajectory preference data

Pairwise preferences, Likert ratings, and step-level rationale over complete agent trajectories. Anonymised participant data accessible via UI, CLI, or API; suitable for RLHF, DPO, and reward-model training.

Credentialed specialist review

Domain Experts across software engineering, clinical, legal, and finance review trajectories where domain judgement is required — code review, clinical reasoning, citation accuracy, regulatory compliance.

Human user simulation

Participants role-play end users in multi-turn, tool-using evaluation. Demographically pre-specified, reproducible across runs, filter provenance preserved.

Red-team & adversarial evaluation

Prompt injection, tool hijacking, jailbreaks, policy probing. Participant provenance preserved for downstream safety cases.

How fast-moving AI teams use Prolific

Trusted by AI/ML developers, researchers, and leading organizations across industries.

Talk to an expert

Unpacking human preference for LLMs - The HUMAINE framework

The evaluation of large language models faces significant challenges. Technical benchmarks often lack real-world relevance, while existing human preference evaluations suffer from unrepresentative sampling, superficial assessment depth, and single-metric reductionism.

Read the paper

Building breakthrough AI faster

Ai2 reduced human data collection from weeks to hours with Prolific, building state-of-the-art multimodal AI models faster without sacrificing quality.

Gemini 3 Pro: Frontier safety framework

The frontier safety framework report for Google’s latest model.

Start collecting human evaluation data for your next release.

Talk to the team

Get started for free

FAQ

End-to-end Agentic AI Evaluation FAQ

How quickly can we start collecting evaluation data?

Our platform is designed for immediate deployment. Self-serve video and preference projects launch in minutes, with results arriving within hours. Managed teleoperation or safety projects depend on scope, hardware integration, and evaluator specialisation requirements.

How much work will my team need to do versus what Prolific handles?

With our self-serve platform, you control the process. We provide infrastructure and participants. You design tasks - video review, preference, teleop, or survey - in your evaluation tool or our AI Task Builder, set criteria, and analyse results. With managed services, we handle everything from participant sourcing to quality assurance. You define requirements and get verified results.

How does Prolific ensure evaluation data quality for AI evaluations?

We combine participant verification, specialised qualification tests, credentials checks, performance tracking, and automated quality controls to maintain a high-quality pool. For physical AI evaluations, we recommend AI Taskers or Domain Experts when you need robotics, autonomy, mechanical, or safety expertise for your tasks.

How does Prolific compare to traditional data vendors?

Traditional vendors use large annotation or teleop teams on hire, with little transparency into evaluator profiles and selection criteria. Prolific gives you direct access to verified evaluators through self-serve or managed options - the quality assurance of managed services, the transparency and control of direct access, and faster turnaround times.

Roles that get the most from Prolific on agentic AI.

Post-training / RLHF lead running DPO, reward-model, or preference-data collection on multi-turn agent trajectories.
Applied AI lead on a coding, clinical, or legal agent needing domain-expert review at scale.
Eval lead running multi-turn or human-user-simulation benchmarks, needing a representative user distribution beyond internal testers.
Agent safety lead red-teaming for prompt injection, tool misuse, and policy violation pre-release.

Do you support RLHF, DPO, or reward model training data?

Yes. Trajectory-level preference pairs, Likert ratings, free-text rationale, and structured process labels — all programmatic via API with formats matched to DPO, reward-model, and RLAIF training pipelines. Use self-serve for fast iteration, managed services for calibrated production programmes.

How do you handle long trajectories and multi-turn evaluation?

Our task tooling and AI Task Builder are designed for whole-trajectory review — full tool-call sequences, conversation histories, and intermediate reasoning — with structured step-level and outcome-level judgement. Process reward models and long-horizon DPO are first-class use cases, not afterthoughts.

Do you provide domain experts when the task requires them?

Yes. Domain Experts include licensed clinicians, engineers, lawyers, finance specialists, and researchers. For tasks that mix expertise with population scale — for example, clinical agent deployment requiring both specialist judgement and patient-population acceptance — use both pools in the same programme.