Legal contract side-by-side evaluation

A completed blind side-by-side evaluation of two anonymised model responses across 20 contract lifecycle management tasks. The tasks are built from public SEC-filed agreements. Legal subject-matter experts rated each pair against a structured rubric and gave an overall preference.

Contents: 20 contract tasks, 2 responses each
Source: Public SEC-filed agreements
Graded by: Legal experts, blind
Output: Rubric ratings and overall preference

What’s included

The tasks ship with everything needed to read the ratings in context.

01
Contract tasks
20 contract lifecycle management tasks, each with its task context.
02
Sanitised contracts
Substitutes for the public SEC-filed agreements the tasks are built from.
03
Structured rubric
The criteria legal experts applied to each pair of responses.
04
Expert ratings
Rubric ratings and an overall preference for every pair, given blind to the model.

Capabilities tested

Grounding and citationsComprehensivenessMulti-step

Formats

Contract documentsTablesRubricsTask environmentsModel evaluation

How it was built

Built from the research team’s catalogue record.

PeopleAutomated

1
Build tasks
Contract lifecycle management tasks are built from public SEC-filed agreements, with sanitised substitutes for the contracts.
Captured
- Task context
- Sanitised contracts
2
Collect responses
Automated
Two models answer every task, and their responses are anonymised.
Captured
- Paired responses
3
Blind review
People
Legal subject-matter experts rate each pair against a structured rubric, without knowing which model wrote which response.
Captured
- Rubric ratings per pair
4
State a preference
People
For each pair, the expert gives an overall preference.
Captured
- Overall preference per pair