Behind every clean redline lies dozens of nuanced judgment calls. Before a single word of a contract is actually edited, the attorney must make decisions like what the change should be, how to escalate it, and whether a change is necessary at all. Historically, there have been no benchmarks to measure just how well a contracts AI tool can perform this layered and sophisticated work.
That’s why we’ve created ContractBench, a benchmark built in partnership with micro1 to make these processes visible and measurable. We’re publishing ContractBench along with our methodology, the public dataset, and the results of running it against seven frontier models.
Why we built a benchmark
Other legal benchmarks simply grade the edits a model makes. While this metric is technically reasonable, it doesn’t fully measure the effectiveness of a model. For example, an attorney who identifies every limitation of liability clause in an agreement and redlines them all has produced a technically thorough review—just at the cost of true efficacy and efficiency.
What makes a review truly effective comes down to the decisions made before the physical edits. These involve determinations like which issues are worth the negotiating capital, when acceptable language should be left alone, when a standard position should be adjusted based on the specifics of this deal, and when a call should be escalated to someone more senior.
Models are increasingly able to handle more of the end-to-end review process, including these context-dense judgment calls. As models evolve from suggesting changes to executing them, legal teams need to know which decisions a model can handle reliably and which need a human lawyer’s judgment. A single accuracy metric cannot determine that capability, and neither can a benchmark that only looks at edits.
How we built it
ContractBench contains 350 contract-review tasks across 35 scenarios and five agreement types: MSAs, IP licensing agreements, research partnerships, commercial leases, and marketing agreements. Half the tasks are opening redlines on a clean agreement, and the other half are responses to a counterparty’s markup.
Everything used in building ContractBench was written by contract attorney experts: the contracts, the playbooks, the commercial context for each deal, and the judgment criteria. The generated scenarios vary with regards to whether a playbook exists and how detailed it is, which allowed us to measure how much explicit company rules affect a model’s performance.
Each model receives the Word document, the representing party’s playbook, the commercial context, and a brief explaining which party the model represents and what decisions will require approval. Every judgment call made then falls into one of four action categories:
- Edit: make a substantive change to the provision.
- Preserve: leave acceptable language alone.
- Escalate: flag the issue for internal approval.
- Comment: communicate with the counterparty without changing the text.
When responding to a counterparty redline, the Edit action further branches into a choice of either rejecting a change, countering it, or introducing a new point, while the Preserve action provides a choice of either accepting a proposed change or leaving the language untouched.
The machinations of the Escalation action required a different process, given that whether an escalation is not represented in the final, physical document. The benchmark criteria thus evaluate the efficacy of an escalation by looking at whether a) an escalation was required and b) who should receive it. A model thus earns full credit on this metric by identifying both in a comment, and how the issue is ultimately resolved is not scored.
Combining each rubric item’s action, priority, and grounding produces five dimensions of contract-review judgment upon which the models are scored:
- Issue Prioritization: whether the reviewer prioritizes the right issues
- Negotiation Restraint: correctly knowing when to leave an issue alone
- Situational Judgment: adapting to the circumstances of the deal
- Escalation Judgment: recognizing the limits of the model’s given authority
- Standard Position Execution: correctly executes the company’s standard position
Separating these metrics into discrete fields is crucial for the integrity of the evaluation. A model can be, for example, strong at restraint and weak at prioritization, and neither can be improved unless they are evaluated separately.
Lastly, five attorneys reviewed each scenario independently, so every model is graded against five expert views rather than one. We determined this to be a necessary safeguard in a field where experienced practitioners can reasonably disagree.
What we found
Models are capable reviewers. On the opening redline, they catch the issues an experienced attorney would, and apply playbook positions correctly when such positions are provided.
However, distinct gaps still exist. In situations where there was a playbook rule given, the model often failed to recognize when a particular deal called for a deviation from that position.
Other notable results are the following:
Restraint is the strongest skill. Leaving acceptable language alone is the highest-scoring action for every model we tested. The other actions leave considerable room to improve.
Reconciling a playbook with the facts of a deal is the most interesting open question. Models apply an unconditional playbook rule well, but score consistently lower when the right position depends on the particulars of the transaction. Every model failed at this to some degree. The commercial context was available to the models the whole time, so the research question now is how that information gets connected to the relevant decision.
Escalation is the largest untapped skill. Models had approval boundaries, written instructions on when to escalate, and a dedicated tool for doing it. However, they largely did not do so. Knowing a rule exists and recognizing the moment you have reached its edge are different skills, and this result tells us that more training is needed in this area.
Edits land in broad strokes. Attorneys make 64% of their changes inline, averaging 93 characters. Models tend to rewrite whole blocks. This is less a question of quality of judgment and more one of how that judgment is applied differently between human and AI reviewers. Human editors tend to send the majority of the language back to the counterparty as written, while models will rewrite large swathes even when only a small, surgical edit was technically necessary.
Where ContractBench goes next
We designed ContractBench to outlast the models it ranks today. ContractBench measures specific judgment calls rather than performance against a fixed level of difficulty, so it will stay meaningful as capabilities improve. Ivo will continue to use it in our own model development, and we invite others to do the same.
Keeping substantive judgment as the leaderboard signal keeps the benchmark a clean target for post-training, which we believe should remain the priority. Attorneys should likewise remain in the loop, as practitioners need to know in what respects a model’s output is ready to send and where the output requires a second look.
We purposefully designed ContractBench to report substance and style of redlines separately. “Substance” here refers to whether a model’s edits identify the right issues and reach the right legal positions; “style” measures whether it expresses those positions in a redline an attorney would actually send, given the business context. This is important because metric that ranks on substance alone can potentially reward a model that reaches the right positions, but through, for example, extensive rewrites that no attorney would actually send.
As models handle more of a redline end to end, the benchmark should reward a redline that is right on both substance and style, and is ready to send as written. This is what Ivo is working towards with ContractBench.
What this means for models, contracting, and Ivo
Models will undoubtedly keep getting better at standard positions. The next gains will come from the application of nuanced judgment: reading a deal’s context, knowing when to hold back, and knowing what to ask and when. These calls depend on layers of complex context that no single document holds, such as the history with this counterparty, the business reason behind a deadline, and the person who needs to sign off.
A benchmark that measures those skills gives the field something to build against, and we intend to help lead that work. Better models made everything Ivo does today possible, so every advance in judgment strengthens what we offer our customers. By publishing ContractBench openly, we accelerate progress that shows up directly in their redlines.
For legal teams, that progress means a model that can handle the brunt of the work accurately, but also accurately knows when to call upon an attorney’s judgment. ContractBench gives the field a precise way to measure how close we are to that goal, and provides a blueprint for how to improve models for contract work.
Read the full methodology and results and explore the public dataset here.
