Financial agent process supervision

Ten financial-agent runs from BigFinanceBench. Finance professionals and three separate model families each judged the same 47 steps independently, which gives a matched human and model verdict for every step. The review instruments and provenance show where human and model judgement agree and where they diverge.

Contents: 10 agent runs, 47 steps
Source: BigFinanceBench
Graded by: Finance professionals and 3 model families
Output: Matched human and model verdicts per step

What’s included

Each step carries a human verdict and the model verdicts made against it.

01
Agent runs
Ten financial-agent trajectories from BigFinanceBench, 47 steps in all.
02
Expert verdicts
A verdict from finance professionals on every step.
03
Model verdicts
Verdicts on the same steps from three separate model families.
04
Matched pairs
Human and model verdicts paired step by step.
05
Review instruments
The instruments reviewers used, with the provenance of each run.

Capabilities tested

ComprehensivenessFalse premisesGrounding and citationsHedgingLong-horizon

Formats

Agent trajectoriesTool-use tracesTablesRubricsHuman and model evaluation

How it was built

Built from the research team’s catalogue record.

PeopleAutomated

1
Select runs
Ten financial-agent runs are taken from BigFinanceBench, 47 steps in all.
Captured
- Agent trajectories
- Tool-use traces
2
Expert review
People
Finance professionals judge every step independently.
Captured
- Human verdict per step
3
Model review
Automated
Models from three separate families judge the same 47 steps.
Captured
- Model verdicts per step
4
Match verdicts
Human and model verdicts are paired step by step, to show where they agree and where they diverge.
Captured
- Matched verdict pairs