Optimise agents for real human preference
Prolific is the human eval layer for agentic AI
For the moments metrics can't solve
Autoresearch optimises a single value. Prolific tells you whether humans agree.
"Autonomous agents optimise what they can measure. They need humans to verify they're measuring the right thing."
We ran the experiment ourselves: 50 autonomous agent-led training runs, followed by a human eval with 300 Prolific participants. The metric said one model won. 300 humans picked the other.
Build with the right human insight
The human layer your training loop needs.
Read the research
From our own experiment to peer-reviewed evidence - the case for human preference eval in agentic workflows.
Start collecting human evaluation data for your next release.
End-to-end Agentic AI Evaluation FAQ
Our platform is designed for immediate deployment. Self-serve video and preference projects launch in minutes, with results arriving within hours. Managed teleoperation or safety projects depend on scope, hardware integration, and evaluator specialisation requirements.
With our self-serve platform, you control the process. We provide infrastructure and participants. You design tasks - video review, preference, teleop, or survey - in your evaluation tool or our AI Task Builder, set criteria, and analyse results. With managed services, we handle everything from participant sourcing to quality assurance. You define requirements and get verified results.
We combine participant verification, specialised qualification tests, credentials checks, performance tracking, and automated quality controls to maintain a high-quality pool. For physical AI evaluations, we recommend AI Taskers or Domain Experts when you need robotics, autonomy, mechanical, or safety expertise for your tasks.
Traditional vendors use large annotation or teleop teams on hire, with little transparency into evaluator profiles and selection criteria. Prolific gives you direct access to verified evaluators through self-serve or managed options - the quality assurance of managed services, the transparency and control of direct access, and faster turnaround times.
- Post-training / RLHF lead running DPO, reward-model, or preference-data collection on multi-turn agent trajectories.
- Applied AI lead on a coding, clinical, or legal agent needing domain-expert review at scale.
- Eval lead running multi-turn or human-user-simulation benchmarks, needing a representative user distribution beyond internal testers.
- Agent safety lead red-teaming for prompt injection, tool misuse, and policy violation pre-release.
Yes. Trajectory-level preference pairs, Likert ratings, free-text rationale, and structured process labels — all programmatic via API with formats matched to DPO, reward-model, and RLAIF training pipelines. Use self-serve for fast iteration, managed services for calibrated production programmes.
Our task tooling and AI Task Builder are designed for whole-trajectory review — full tool-call sequences, conversation histories, and intermediate reasoning — with structured step-level and outcome-level judgement. Process reward models and long-horizon DPO are first-class use cases, not afterthoughts.
Yes. Domain Experts include licensed clinicians, engineers, lawyers, finance specialists, and researchers. For tasks that mix expertise with population scale — for example, clinical agent deployment requiring both specialist judgement and patient-population acceptance — use both pools in the same programme.
