Elicitation-sensitive coding environments

Five coding environments that test whether an agent notices a hidden, underspecified stakeholder decision and resolves it by asking, rather than guessing. Each environment is a real open-source repository at a pinned pre-fix commit, graded mechanically by an acceptance harness outside the repository.

Contents: 5 coding environments
Source: Real open-source repositories
Graded by: Mechanical acceptance harness
Output: Pass or fail per run, full transcripts

Results

Pooled pass rate on a complete grid of 5 tasks and 4 models, with 3 runs per cell. Only the brief and who the agent can consult change.

Agents almost never ask on their own. Pass rates fall from 98% with the full brief to 5% when a stakeholder is reachable but not mentioned.

Full briefEvery requirement is given up front.
98%
Told to ask firstThe agent is explicitly instructed to consult the stakeholder before coding.
33%
Stakeholder reachableA stakeholder can be consulted, but the agent is not prompted to ask.
5%
No one to consultThe same ask, with nobody available to answer questions.
1%

What’s included

Every environment ships the same 5 parts, so results can be checked and rerun.

01
Real repository
An open-source repository checked out at a pinned commit from before the fix, so the agent works in a real codebase.
02
Requirement graph
Every required behaviour is tagged by who could have supplied it, separating what the code reveals from what only a stakeholder knows.
03
Acceptance harness
Tests sit outside the repository tree, where the agent cannot read or edit them. Grading is fully mechanical.
04
Gold patch and variants
A reference fix, plus a mutant and a structure-moving variant of it.
05
Graded transcripts
Complete, unedited transcripts of graded runs on both sides of the gap shown below.

Capabilities tested

Requirement elicitationPlanningFalse premisesArchitectural empathyMulti-step tool use

Formats

Task environmentsAgent trajectoriesRubricsCodeTool-use traces

How it was built

Built from the research team’s catalogue record.

Automated

1
Pin repository
An open-source repository is checked out at a commit from before a known fix.
Captured
- Repository and pinned commit
2
Generate task
Automated
A task is generated synthetically, withholding one decision that only a stakeholder could make.
Captured
- Task brief
- The withheld decision
3
Map requirements
Each required behaviour is tagged by who could have supplied it: the code or a stakeholder.
Captured
- Requirement graph
4
Run agents
Automated
Four models attempt each task under four conditions, with three runs per cell.
Captured
- Complete, unedited transcripts
- Tool-use traces
5
Grade
Automated
An acceptance harness outside the repository grades each run mechanically. The agent cannot read or edit it.
Captured
- Pass or fail per run
- Pooled pass rate per condition