3 in the library
Benchmarks
Model results, scored by human judgment
- Scoring
- Public, clinician or validated judge
- Uncertainty
- Intervals and n on every score
- Ranking
- Ties and overlapping intervals shown
5 in the library
Sample datasets
Research samples with their grading
- Contents
- Transcripts, trajectories and tasks
- Grading
- Experts, model judges or test harnesses
- Provenance
- From Prolific research
In development
Datasets & RL environments
Expert-built tasks with hidden tests
- Generation
- Fresh tasks on demand, never repeated
- Grading
- Deterministic hidden tests, offline
- Selection
- Kept only if it separates models
Evaluation library
Results to compare. Methods and tasks to explore.
5 resources across all domains
Mechanically graded environments
- 5
- real open-source repositories
- 1
- hidden acceptance harness each
- 5
- parts shipped per environment
Elicitation‑sensitive coding environments
Coding environments that test whether an agent spots a hidden, underspecified stakeholder decision and asks about it, rather than guessing.
Does an agent ask before it guesses?
- Contents
- 5 coding environments
- Graded by
- Mechanical acceptance harness
- Source
- Real open-source repositories
Blind side-by-side
- 20
- contract lifecycle tasks
- 2
- anonymised model responses per task
- 1
- rubric, plus an overall preference
Legal contract side‑by‑side evaluation
A blind side-by-side evaluation of two anonymised models on 20 contract lifecycle tasks, rated by legal experts against a structured rubric.
Which model’s contract work do legal experts prefer, and why?
- Contents
- 20 contract tasks, 2 responses each
- Graded by
- Legal experts, blind
- Source
- Public SEC-filed agreements
Step-level verdicts
- 10
- financial-agent runs from BigFinanceBench
- 47
- steps, each judged independently
- 3
- model families, plus finance professionals
Financial agent process supervision
Every step of ten financial-agent runs, judged by finance professionals and by three model families, so human and model verdicts can be compared.
Where do model judges agree with finance professionals, step by step?
- Contents
- 10 agent runs, 47 steps
- Graded by
- Finance professionals and 3 model families
- Source
- BigFinanceBench
Episode-level comparison
- 2
- coding-agent trajectories per task
- 2
- measures: semantic diversity and outcome variance
- SxS
- rater prototype for human judgement
Coding trajectory side‑by‑side evaluation
Two coding-agent trajectories split into episodes and compared side by side, with comparison agents launched from the points that matter most.
Where do two coding agents’ trajectories diverge, and does it matter?
- Contents
- Paired agent trajectories, in episodes
- Graded by
- Automated judge
- Source
- Matched benchmark tasks
Curated failure candidates
- 6
- synthetic multi-turn conversations
- 81
- messages, in full transcripts
- 2
- lead cases flagged by more than one model family
Synthetic youth safety conversations
Synthetic multi-turn conversations that surface candidate youth-safety failures, each with its full transcript, severity evidence and realism caveats.
Where does a long conversation start to fail on youth safety?
- Contents
- 6 conversations, 81 messages
- Graded by
- Models from different families
- Source
- Synthetic multi-turn generation
Need to go beyond
an existing benchmark?
Start with the models, users and decisions that matter to you.
Talk to the team
