3 in the library
Benchmarks
Model results, scored by human judgment
- Scoring
- Public, clinician or validated judge
- Uncertainty
- Intervals and n on every score
- Ranking
- Ties and overlapping intervals shown
5 in the library
Sample datasets
Research samples with their grading
- Contents
- Transcripts, trajectories and tasks
- Grading
- Experts, model judges or test harnesses
- Provenance
- From Prolific research
In development
Datasets & RL environments
Expert-built tasks with hidden tests
- Generation
- Fresh tasks on demand, never repeated
- Grading
- Deterministic hidden tests, offline
- Selection
- Kept only if it separates models
Evaluation library
Results to compare. Methods and tasks to explore.
8 resources across all domains
Overall preference (BTD score, 95% CI)
- 01Gemini 3.1 Pro (preview)35.4
- 02Gemini 3.5 Flash34.8
- 03Gemini 3 Pro33.8
- 04Claude Fable 533.5
HUMAINE Leaderboard
Multi-dimensional human preference, ranked with a hierarchical Bradley–Terry–Davidson model and broken down across 22 demographic groups.
Which models do people prefer, and does that change across demographic groups?
- Top result
- Gemini 3.1 Pro (preview), 73% probability of ranking first
- Measures
- Overall, task performance, communication, fluidity, trust & ethics
- Sample
- 253,849 comparisons, 54 models, 22 demographic groups
Clinicians: safer reply share
- Claude Fable 575%
- Qwen 3.7 Max65%
- Gemini 3.1 Pro50%
- DeepSeek V3.245%
- GPT-5.540%
- Mistral Large 325%
HUMAINE Health
Public preference on real health conversations, set against clinician safety review and an adversarial audit of where models give way under pressure.
Is the health advice people trust also the advice clinicians judge safe?
- Top result
- Claude Fable 5, safer reply in 75% of clinician comparisons
- Measures
- Public trust, clinician safety, risk of harm, adversarial failure
- Sample
- 17,168 conversations, 100 clinicians, 978 clinical judgements
Mean scenario score (clustered 95% CI)
- 01Claude 4.5 Sonnet4.66
- 02Claude 4.5 Opus4.65
- 03GPT-5.24.53
- 04Claude 4 Sonnet4.33
Alignment Leaderboard
Behavioural alignment across 904 realistic multi-turn scenarios, testing what models do under pressure rather than what they say they would do.
How do models behave when instructions conflict and pressure builds?
- Top result
- Claude 4.5 Sonnet, 90% of scenarios passed
- Measures
- Honesty, safety, non-manipulation, robustness, corrigibility, scheming
- Sample
- 904 scenarios, 24 models, judged by Claude Opus 4.5
Mechanically graded environments
- 5
- real open-source repositories
- 1
- hidden acceptance harness each
- 5
- parts shipped per environment
Elicitation‑sensitive coding environments
Coding environments that test whether an agent spots a hidden, underspecified stakeholder decision and asks about it, rather than guessing.
Does an agent ask before it guesses?
- Contents
- 5 coding environments
- Graded by
- Mechanical acceptance harness
- Source
- Real open-source repositories
Blind side-by-side
- 20
- contract lifecycle tasks
- 2
- anonymised model responses per task
- 1
- rubric, plus an overall preference
Legal contract side‑by‑side evaluation
A blind side-by-side evaluation of two anonymised models on 20 contract lifecycle tasks, rated by legal experts against a structured rubric.
Which model’s contract work do legal experts prefer, and why?
- Contents
- 20 contract tasks, 2 responses each
- Graded by
- Legal experts, blind
- Source
- Public SEC-filed agreements
Step-level verdicts
- 10
- financial-agent runs from BigFinanceBench
- 47
- steps, each judged independently
- 3
- model families, plus finance professionals
Financial agent process supervision
Every step of ten financial-agent runs, judged by finance professionals and by three model families, so human and model verdicts can be compared.
Where do model judges agree with finance professionals, step by step?
- Contents
- 10 agent runs, 47 steps
- Graded by
- Finance professionals and 3 model families
- Source
- BigFinanceBench
Episode-level comparison
- 2
- coding-agent trajectories per task
- 2
- measures: semantic diversity and outcome variance
- SxS
- rater prototype for human judgement
Coding trajectory side‑by‑side evaluation
Two coding-agent trajectories split into episodes and compared side by side, with comparison agents launched from the points that matter most.
Where do two coding agents’ trajectories diverge, and does it matter?
- Contents
- Paired agent trajectories, in episodes
- Graded by
- Automated judge
- Source
- Matched benchmark tasks
Curated failure candidates
- 6
- synthetic multi-turn conversations
- 81
- messages, in full transcripts
- 2
- lead cases flagged by more than one model family
Synthetic youth safety conversations
Synthetic multi-turn conversations that surface candidate youth-safety failures, each with its full transcript, severity evidence and realism caveats.
Where does a long conversation start to fail on youth safety?
- Contents
- 6 conversations, 81 messages
- Graded by
- Models from different families
- Source
- Synthetic multi-turn generation
Need to go beyond
an existing benchmark?
Start with the models, users and decisions that matter to you.
Talk to the team
