Coding trajectory side-by-side evaluation

A working judge prototype that splits two coding-agent trajectories into episodes and launches comparison agents from high-leverage points. It measures semantic diversity and outcome variance across matched benchmark tasks. Review packets and a rater-facing prototype present the same episodes for human side-by-side judgement.

Contents: Paired agent trajectories, in episodes
Source: Matched benchmark tasks
Graded by: Automated judge
Output: Semantic diversity, outcome variance

What’s included

The judge, its outputs and the materials for human review of the same episodes.

01
Trajectories
Two coding-agent trajectories per task, split into episodes.
02
Comparison runs
Agents launched from high-leverage points in each trajectory.
03
Measures
Semantic diversity and outcome variance across matched benchmark tasks.
04
Review packets
The episodes packaged for review, with the methodology.
05
Rater prototype
A rater-facing tool for human side-by-side judgement on the same episodes.

Capabilities tested

Tool useGrounding and citationsMulti-step

Formats

Agent trajectoriesCodeTool-use tracesRubricsHuman evaluation

How it was built

Built from the research team’s catalogue record.

Automated

1
Capture trajectories
Automated
Coding agents attempt matched benchmark tasks, and their full trajectories are recorded.
Captured
- Agent trajectories
- Tool-use traces
2
Split into episodes
Automated
A judge prototype splits each trajectory into episodes.
Captured
- Episodes
3
Branch new runs
Automated
Comparison agents are launched from high-leverage points in a trajectory.
Captured
- Comparison runs
4
Measure
Automated
Semantic diversity and outcome variance are measured across the matched tasks.
Captured
- Semantic diversity
- Outcome variance
5
Prepare human review
Review packets and a rater-facing prototype present the same episodes for side-by-side judgement.
Captured
- Review packets
- Rater prototype