Scoring at a glance

Grok 4.7 leads a close top tier, but completion, cost, and speed separate the leaders. Grok leads on weighted rubric pass rate at 58.3%, followed by Opus 5.5 at 58.1% and Kimi K3 at 57.1%. Completion rates range from 80.8% for Opus to 99.5% for Grok and 100% for Kimi. Kimi costs 38% less per completed rollout and finishes 13% faster than Grok.
Models favor passive decisions, while Astra stands apart on escalation. All seven models score higher on opening redlines than on responses to counterparty markups. On Turn 2, models pass more than 96% of criteria requiring them to leave unchanged language alone, compared with 15.1% to 36.5% of criteria requiring a new edit. Astra scores 70.8% on issues requiring internal escalation, more than twice the rate of any other model.
Models struggle with high-priority issues and context-dense judgment. Six of seven models score lowest on issues attorneys consider most important. Every model scores 9 to 15 percentage points lower when company rules must be interpreted against deal-specific facts than when those rules apply unconditionally. Six of seven also score lowest without a company playbook.
Models score higher where attorney redlines overlap. In a controlled study of initial contract reviews, five attorneys independently redline the same agreement, demonstrating multiple viable approaches. Grok scores 67.0% on paragraphs flagged by four or five attorneys, compared with 39.5% on those flagged by only one. Opus scores 65.5% versus 30.3%.‍
No model matches attorneys' precision in redlining. Attorneys score 63.5% on the surgicality index, which measures the share of targeted inline edits, compared with 36.6% for Opus 5.5, the highest-scoring model. On initial reviews, attorneys make an average of 125 edits per task, each averaging 93 characters, while models make 41 edits averaging 300 characters.

Introduction

Contract review is a judgment problem before it is an editing problem. The hard part is not changing language; it is deciding when to intervene at all. A finished redline hides that reasoning, compressing a series of legal, commercial, and authority judgments into the final document. We've built ContractBench to make those judgments visible and measurable at scale.

ContractBench is a fully expert-authored benchmark of 350 contract-review tasks across 35 scenarios and five major agreement types: Master Services Agreement ("MSA"), IP Licensing, Research Partnership, Commercial Lease, and Marketing Agreement. It covers both the first pass of a clean agreement ("Turn 1") and the response to a counterparty markup ("Turn 2"), with variation in commercial context, jurisdiction, and the availability and complexity of enterprise playbooks.

The core innovation is a rubric taxonomy that reconstructs the judgment behind the redline. Each decision is captured as one of four observable actions, Edit, Preserve, Comment, or Escalate, then interpreted together with its priority and the governing playbook and transaction context. This decomposition makes five dimensions of negotiation judgment measurable: whether the reviewer prioritizes the right issues (Issue Prioritization), exercises restraint (Negotiation Restraint), recognizes when to escalate (Escalation Judgment), adapts to the circumstances of the deal (Situational Judgment), and correctly executes the company's standard position (Standard Position Execution).

ContractBench also models contract review as a multi-expert solution space. Instead of assuming that one attorney's redline defines the correct answer, the benchmark captures where experienced attorneys converge and where multiple reasonable approaches exist.

Substantive alignment, however, is only part of a production-grade redline. A model may identify the right issues while producing a document no attorney would send. We introduce a surgicality index and editing-volume measures to quantify how closely models' drafting behavior aligns with attorney practice, capturing differences that substantive rubric scores alone cannot reveal.

The differences in model performance across negotiation judgment, attorney consensus, and drafting behavior demonstrate this benchmark's ability to differentiate model capabilities and provide a durable basis for evaluating future model generations. To our knowledge, ContractBench is the largest and most comprehensive contract review benchmark to date.

Methodology

Benchmark construction

2.1

Table 1. Benchmark characteristics

Benchmark characteristic	Coverage
Scale	350 tasks across 35 scenarios, posed as 210 distinct inputs (630 rollouts per model)
Agreement types	MSA, IP Licensing, Research Partnership, Commercial Lease, Marketing Agreement (seven scenarios and 70 tasks each)
Negotiation stages	175 initial-redline tasks (Turn 1); 175 responses to counterparty redlines (Turn 2)
Jurisdiction	30 US scenarios; five UK
Playbook availability	30 scenarios with playbooks; five without (one per agreement type)
High complexity	11 scenarios with multi-contract playbooks: seven US, four UK

ContractBench contains 35 scenarios across five agreement types. Thirty use U.S. law and five use UK law. Within each agreement type, six scenarios include a company playbook and one does not. Across the 30 playbook scenarios, 19 use a single-agreement playbook and 11 use a higher-complexity multi-contract playbook.

Each scenario combines a contract with transaction-specific commercial context and, where applicable, the represented party's playbook. The playbook defines authorized positions, fallbacks, negotiation rules, and approval boundaries. Each scenario yields five Turn 1 opening-redline tasks and five Turn 2 response tasks, for 175 tasks at each turn. Turn 1 begins from a clean agreement; Turn 2 begins from an attorney-produced counterparty redline.

Rubric taxonomy

2.2

Attorneys complete the redline before writing their rubrics, then encode the most important judgments from the review as approximately 10 to 20 pass-or-fail items per task. Each rubric item identifies the relevant contract section, required action, priority, evaluation dimension, substantive condition required to pass, and a justification documenting the grounding for that judgment.

Required action

The required action taxonomy has four categories: Edit, Preserve, Comment, or Escalate. For Edit, Preserve, and Comment, the rubric maps directly to the attorney's golden redline. The required action is the action the attorney takes in the document. On Turn 2, Edit further distinguishes Reject, Counter, and New Edit, while Preserve distinguishes Accept, where a counterparty edit should be accepted, from Leave Untouched, where an unedited paragraph should remain unchanged.

Escalate is the one category where the rubric captures a judgment that may not appear in the attorney's golden redline. Attorneys may leave the provision unchanged or resolve the issue in the document, keeping the redline realistic as an outbound, counterparty-ready redline. The rubric separately records whether escalation was required and to whom. Models must identify both in a comment to receive full credit; the ultimate resolution is not scored.

Priority

Table 2. Rubric items by priority and turn

Priority	Turn 1 items	Turn 2 items	All items
High	1,713	1,592	3,305
Medium	1,045	1,057	2,102
Low	196	259	455

Rubric items are assigned High, Medium, or Low priority, weighted 3, 2, and 1. The benchmark is concentrated on higher-priority issues, producing 3,305 High-priority items, 2,102 Medium-priority items, and 455 Low-priority items. After weighting, High-priority items account for 68% of total rubric weight. This makes performance on the most consequential issues the dominant component of the overall leaderboard.

Evaluation dimension

Required action, priority, and grounding together determine the evaluation dimension. High-priority Edit and Comment items measure Issue Prioritization; Preserve items measure Negotiation Restraint; and Escalate items measure Escalation Judgment. Medium- and Low-priority Edit and Comment items measure Situational Judgment when the answer depends on transaction-specific facts, and Standard Position Execution when an unconditional playbook rule supplies the answer.

Multi-expert solution space

2.3

Turn 1 creates a controlled multi-expert solution space. Five attorneys independently review the same contract and grounding materials, each producing a separate redline and rubric set. This yields five expert solutions to the same contract-review problem.

We grade each Turn 1 model output against all five rubric sets. This captures both shared and individualized attorney judgments and lets us measure how model performance changes as attorney consensus increases.

Agent's environment

2.4

All models use the same document-editing environment. They receive the Word contract, the represented party's commercial context and applicable playbook, the task instructions, and the same redlining toolkit.

Turn 1 starts from an unmarked agreement. Turn 2 preserves the counterparty's tracked changes and comments. Models edit the .docx directly using tracked changes, accept or reject decisions, comments, and escalation flags.

Figure 3. The agent's environment

Agent's sandbox

/app/

contract.docx

grounding/

<Playbook>.docx | .pdf

<Commercial>.docx | .pdf

/skills/contract-redliner/

SKILL.md

references/anchors.md

scripts/

read_document.py

propose_edits.py

add_comment.py

mark_reserved.py

redline_engine/

doc_revisions/

instruction.md

the agent's working folder the Word contract under negotiation background for the represented side positions, fallbacks, approval rules deal facts and business priorities

one redlining toolkit for every task how to redline with the scripts edit format and anchoring rules command-line tools the agent runs shows the contract, paragraphs labeled tracked edits, accepts and rejects adds comments and internal escalations strikes a whole section as Reserved document parsing and anchoring library tracked-change support for Word files the task brief the agent receives

01 /app/contract.docx	The working file. Turn 1 supplies an unmarked draft; turn 2 supplies the counterparty's markup of it, tracked changes and comments included. Whatever the agent leaves in this file when it stops is what gets graded.
02 /app/grounding/	Background on the agent's own side only: the playbook and the commercial brief for the deal, in their original PDF or Word form. Nothing from the counterparty is included, and the no-playbook scenarios supply the brief alone.
03 /skills/contract-redliner/	A short guide and four command-line scripts for reading the contract and writing Word-native tracked changes, comments, and escalations. The same tools serve every party and turn.
04 instruction.md	The task brief: who the agent represents, what it may decide on its own and what needs approval, how to flag those points, how to address the counterparty, and what this turn asks for.

Model evaluation and grading

2.5

We evaluate seven models (Opus 5.5, GPT-6 Astra, Gemini 3.8 Flash, Grok 4.7, Kimi K3, GLM-5.3, and DeepSeek V4 Pro) through the same harness and skill, each at its provider's highest reasoning setting with a per-turn output limit of 128,000 tokens unless the model's documented maximum is lower. We run three independent rollouts for every model input. The 350 benchmark tasks correspond to 210 distinct inputs: 35 shared Turn 1 inputs and 175 distinct Turn 2 counterparty redlines. This produces 630 rollouts per model, including 105 on Turn 1 and 525 on Turn 2. Each Turn 1 rollout is graded against all five attorney rubric sets for the scenario.

For each completed rollout, the grader diffs the returned document against the input contract. A validity gate checks that every modification appears as a tracked change or a permitted accept/reject decision. The grader then creates a normalized change list covering the model's edits by section, the disposition of each counterparty change, its comments and escalations, and the resulting text of each section.

Three judges from different model providers independently grade every rubric item Pass or Fail from that record: Opus 4.8, GPT-5.5, and Gemini 3.5 Flash. A strict majority determines the final verdict. Task score equals earned rubric weight divided by total rubric weight, allowing scores to remain comparable across tasks with different numbers of rubric items.

We calculate rubric scores only for completed, gradeable redlines. A rollout that reaches the token limit or fails the validity gate receives no substantive score and instead counts against the model's completion rate.

Overall results

Grok 4.7 leads a tight top tier.

3.1

Three models form a clear leading group. Grok 4.7 scores highest on weighted rubric pass rate at 58.3%, followed closely by Opus 5.5 at 58.1% and Kimi K3 at 57.1%. Gemini 3.8 Flash and GLM-5.3 follow at roughly 55%, while DeepSeek V4 Pro and GPT-6 Astra score 48.4% and 45.6%.

Judge agreement is high, with little evidence of same-family self-preference. The three judges were unanimous on 83.4% of graded rubric items. For the three models with a same-family judge, the estimated self-preference margin is effectively zero: 0.0 points for Opus 5.5, -0.4 for GPT-6 Astra, and +0.2 for Gemini 3.8 Flash.

Completion rate further differentiates the leading models. Grok 4.7 completes 99.5% of its rollouts and Kimi K3 all of them, while Opus 5.5 completes 80.8%. Grok 4.7 therefore combines the highest weighted rubric score with near-complete benchmark coverage.

The completion gap is driven primarily by token-limit failures. Across the benchmark, 152 of the 4,410 rollouts ended before producing a gradeable redline because the model exhausted its output budget, compared with only six failures from validity-gate rejection. Opus 5.5 accounts for 121 of those token-limit stops, affecting 19.2% of its rollouts; GLM-5.3 is a distant second at 29 stops, or 4.6%. The other five models almost never hit the token limit.

Table 3. Failure modes (rollouts without a gradeable redline)

Failure mode	What happens	Occurrences
Token limit	The model spends its whole per-turn output budget reasoning, and the turn ends before it writes an edit.	152
Gate rejection	The validity gate rejects the deliverable because an edit was made outside tracked changes or the counterparty's markup was altered.	6

Cost and speed sharpen the differences between closely matched models.

3.2

Cost and latency create much wider separation than rubric performance among the leading models. Kimi K3 scores only 1.2 points below Grok 4.7, but costs 38% less per completed rollout and finishes 13% faster. Compared with Opus 5.5, Kimi scores one point lower while costing 62% less and finishing 36% faster.

The tradeoff is even more pronounced just outside the leading group. Gemini 3.8 Flash scores 3.3 points below Grok, but costs 59% less and completes a rollout 52% faster. For enterprise contract review, where the same workflow may be run across large volumes of agreements, these differences can become material. When model scores cluster within a few percentage points, cost and throughput can meaningfully change the practical comparison.

Decomposing model performance

The leaderboard already differentiates models at the aggregate level. To understand where this differentiation arises, we further decompose performance along the core axes of our rubric taxonomy (negotiation stage, required action, priority, and evaluation dimension), while also assessing the robustness of these findings along other dimensions of the benchmark design: playbook complexity, agreement type, and jurisdiction. These analyses corroborate one another, yield meaningful separation in model capabilities, and demonstrate that the ContractBench taxonomy serves as a durable methodology for assessing future models.

By negotiation stage

4.1

Models perform better on the opening redline than the response.

Every model scores higher when marking up a clean template than when responding to a counterparty redline. The decline ranges from 1.4 points for Kimi K3 and 2.3 for GLM-5.3 to 8.8 for Grok 4.7. Grok posts the highest Turn 1 score at 62.7%, followed by Opus 5.5 at 60.9%, while the remaining models range from 49% to 58%.

Turn 2 compresses much of the field. Five models score between 52% and 56%, with only DeepSeek V4 Pro and GPT-6 Astra falling below that range at 44.2% and 42.6%. The opening redline therefore produces clearer separation among the higher-scoring models, while responding to an existing markup appears to be a more uniformly difficult task.

By required action

4.2

Models score best on passive decisions, while Astra stands apart on escalation.

Across the four required actions defined in Section 2.2.1, Preserve is the highest-scoring action for every model, with pass rates ranging from 72.7% to 87.9%. Comment is the weakest, ranging from 12.4% to 27.7%. Edit performance falls in between and creates greater separation: Grok 4.7 leads at 61.2%, followed by Opus 5.5 at 56.4% and Kimi K3 at 54.7%, while GPT-6 Astra scores 31.3%.

Astra has a markedly different profile on escalation. It passes 70.8% of escalation rubrics, more than twice the rate of every other model, while the remaining six models range from 19.2% to 34.9%. This strength contrasts with Astra's weaker Edit performance and overall leaderboard position, suggesting that Astra is comparatively better at recognizing the boundaries of its decision authority and when an issue should be escalated.

Turn 2 makes the passivity pattern explicit: models are nearly perfect when doing nothing is the right answer, but their performance drops sharply when they must actively negotiate.

Table 5. Turn 2 weighted pass rate by sub-action

Model	New Edit	Counter	Reject	Accept	Leave Untouched
Opus 5.5	27.0%	46.2%	60.4%	71.0%	98.9%
GPT-6 Astra	15.1%	21.9%	24.1%	68.5%	99.6%
Gemini 3.8 Flash	27.1%	41.3%	58.8%	77.1%	97.0%
Grok 4.7	36.5%	47.6%	66.4%	60.2%	96.6%
Kimi K3	23.9%	43.9%	68.3%	79.2%	97.8%
GLM-5.3	22.8%	37.6%	61.3%	81.3%	98.7%
DeepSeek V4 Pro	16.7%	25.5%	40.9%	80.7%	98.5%

The Turn 2 sub-action breakdown reveals a sharp difference between passive and active decisions. All models pass more than 96% of Leave Untouched rubrics, where the correct move is simply not to reopen language the counterparty did not change. Performance falls when Preserve requires an affirmative decision to Accept, which ranges from 60.2% to 81.3%. The more interventionist responses are harder still: Reject ranges from 24.1% to 68.3%, Counter from 21.9% to 47.6%, and New Edit from 15.1% to 36.5%.

By priority

4.3

Models score lowest on the highest-priority issues.

Models generally perform worse on the issues experts consider most important. Six of seven models score lowest on High-priority rubrics and highest on Low-priority rubrics; Grok 4.7 is the only exception, with nearly identical High and Medium performance. The gap is substantial for several models: DeepSeek V4 Pro rises from 45.4% on High-priority items to 75.3% on Low-priority items, while GPT-6 Astra rises from 42.8% to 67.3%.

By evaluation dimension

4.4

Context-sensitive judgment remains harder than standard playbook execution.

The evaluation-dimension results largely reinforce the patterns above. Negotiation Restraint is the strongest dimension for every model, while Astra's 70.9% score on Escalation Judgment reflects its distinctive strength in recognizing the boundaries of its authority. Issue Prioritization also tracks the earlier weakness on High-priority issues: Grok leads at 62.7%, while Astra scores 31.1%.

The gap between Standard Position Execution and Situational Judgment points to a specific weakness: reconciling playbook rules with deal context. Standard Position Execution tests whether models follow unconditional playbook rules that apply regardless of deal facts. Situational Judgment tests the conditional rules, those that apply only when particular deal facts are present, and so requires models to reconcile the facts and commercial circumstances of the transaction with the playbook before deciding which rule governs. Every model scores 9 to 15 points lower on the latter. Grok, for example, falls from 62.6% on Standard Position Execution to 47.7% on Situational Judgment, while Gemini drops from 54.2% to 40.3%. The consistent gap underscores the difficulty of translating enterprise playbooks into context-sensitive negotiation decisions.

By playbook complexity

4.5

Models benefit from explicit rules, even in complex playbooks.

No-playbook scenarios place the greatest demands on contextual judgment. Models must determine appropriate negotiation positions from the commercial facts alone, without explicit company rules. Six of seven models score lowest in these scenarios. Most models score highest with single-agreement playbooks, while even multi-contract playbooks outperform no playbook despite their additional rule-selection complexity. These results reinforce the Situational Judgment findings: models struggle with context-sensitive decisions, but playbooks help orient their judgment in complex negotiations.

GPT-6 Astra is the exception. Its score rises from 43% on multi-contract playbooks to 46% on single-agreement playbooks and 50% with no playbook. That reversal does not line up neatly with its weaker Situational Judgment score, suggesting that Astra responds differently to explicit playbook instructions than the rest of the field.

By agreement type and jurisdiction

4.6

The leaderboard is not driven by one contract type or jurisdiction.

Performance varies across agreement types, but the overall model rankings remain broadly consistent. Commercial Lease produces the highest scores for six of seven models, with Astra scoring slightly higher on MSA. This consistent advantage suggests that some agreement types are easier for models to navigate, although we cannot isolate whether the difference comes from contract structure, commercial context, or other factors. No single agreement type appears to drive the overall leaderboard.

We also find no systematic U.S. advantage. Opus 5.5 and Gemini 3.8 Flash score essentially the same across jurisdictions, while Grok 4.7 and Kimi K3 differ by only one point. Astra shows the largest difference, scoring six points higher on UK scenarios. The results suggest that the overall leaderboard is not obviously driven by U.S.-specific legal familiarity, although the smaller UK sample limits the strength of that conclusion.

The multi-expert solution space

We designed Turn 1 as a controlled test to quantify the extent of overlap in attorney judgment while capturing multiple viable approaches to opening negotiation. This multi-expert solution space avoids treating a single attorney's opening move as the only correct answer. We measure the overlap in where attorneys intervene (location consensus) and what actions they choose (action consensus).

Attorney consensus and variation

5.1

Attorneys show substantial overlap in where and how they act.

Table 9. Attorney consensus on location and action, by paragraph and section

Measure	Paragraph	Section
Locations flagged for action	895	516
Flagged by four or five attorneys	22.3%	39.1%
Flagged by two or three	34.7%	33.5%
Flagged by one attorney	42.9%	27.3%
Overlap between two attorneys (Jaccard)	0.37	0.52
Share of an attorney's locations a given colleague also flags	55.1%	70.3%
Disputed: another attorney requires no change	14.4%	22.7%
Same action where all attorneys act on the same location	73.4%	63.5%

At the section level, 39.1% of flagged sections are identified by four or five attorneys and 33.5% by two or three, so about 73% are flagged by at least two attorneys. In pairwise comparisons, one attorney flags on average 70.3% of the sections identified by a given colleague.

Action consensus is also substantial. Among attorneys who act at the same location, 73.4% choose the same action at the paragraph level and 63.5% at the section level. Direct disagreement is less common: another attorney requires no change in 14.4% of flagged paragraphs and 22.7% of flagged sections.

We reviewed cases where attorneys chose different actions at the same location. These differences often arose in definitions, miscellaneous provisions, and attached schedules. Some attorneys made direct edits, others used comments, while others left these sections untouched and concentrated their redlines on substantive provisions. These choices therefore represent different viable approaches to the same review. By scoring model redlines against all five attorney rubric sets in Turn 1, we reduce the influence of any individual attorney's preferences while accounting for variation in professional judgment.

Model performance by attorney consensus

5.2

Models perform better on judgments shared by more attorneys.

Table 10. Turn 1 weighted pass rate by attorney consensus

Model	Four or five attorneys	Two or three	One attorney	Spread across rubric sets (points)
Opus 5.5	65.5%	53.9%	30.3%	15.5
GPT-6 Astra	42.2%	42.1%	35.0%	13.1
Gemini 3.8 Flash	62.0%	50.1%	31.0%	14.8
Grok 4.7	67.0%	60.1%	39.5%	14.6
Kimi K3	59.9%	46.0%	27.9%	13.8
GLM-5.3	58.1%	42.3%	26.6%	13.5
DeepSeek V4 Pro	53.4%	36.8%	20.1%	13.2

Each column groups a model's Turn 1 Edit, Comment and Escalate rubrics by how many of the scenario's five attorneys flag the paragraph the rubric applies to. Spread: the standard deviation, in points, of the same deliverable's score across the scenario's five rubric sets, averaged.

Model performance rises with attorney consensus. We map each rubric to the paragraph where its required action applies, then group rubric scores by how many attorneys independently target that paragraph. Grok 4.7 scores 67.0% when four or five attorneys flag the paragraph, compared with 39.5% when only one attorney does. Opus 5.5 drops from 65.5% to 30.3%, and Gemini 3.8 Flash from 62.0% to 31.0%. The same pattern holds across the model set: models score substantially higher on the shared core of attorney review than on issues identified by a single attorney.

Surgicality and drafting quality

The analyses above measure how well models align with attorneys on substantive negotiation decisions. They do not capture whether the resulting document looks like a redline an attorney would actually send. Even a redline that addresses every substantive issue may be unsuitable for the counterparty if it effectively rewrites the agreement. We introduce surgicality to measure the precision of edits, examine differences in editing volume, and test how these behaviors relate to rubric scores.

Surgicality

6.1

Attorneys make most edits inline, while every model relies predominantly on block rewrites.

Every tracked-change event $e$ in a deliverable is classified as inline or block. For an event in paragraph $p$ , where a paired deletion and insertion at one anchor counts as one event, let $d_{e}$ and $i_{e}$ be the characters it deletes and inserts, and $u_{p}$ the characters of $p$ outside any tracked change. $τ$ is the inline threshold: an event counts as inline when its size, the larger of what it deletes and inserts, is less than $τ$ times $u_{p}$ .

Here $C$ is the set of the 35 Turn 1 contracts and $E_{a, c}$ the set of events in actor $a$ 's deliverables on contract $c$ , so the surgicality score is the share of an actor's edits that are inline, averaged over the contracts. Redlined passages per task are the contiguous runs of paragraphs an actor touches in a deliverable. Edits per task are its events, inline and block. Edit volume is the sum of $size (e)$ over its events, in characters, and the typical insertion is the median insertion length.

We classify each tracked-change event as inline or block based on its size relative to the surrounding paragraph. An edit counts as inline when the larger of its insertion and deletion is less than 30% of the unchanged text in that paragraph. We then average the inline share across the 35 Turn 1 contracts.

Attorneys make 63.5% of their edits inline. Opus 5.5 leads the models at 36.6%, followed by GPT-6 Astra at 31.0%. The other five models range from 14.1% to 20.8%. Attorneys make most of their changes inline; every model makes most of its changes as block rewrites.

Editing volume

6.2

Attorneys cover similar ground through more frequent, smaller, and more targeted edits.

Table 11. Editing volume, attorneys versus models (mean and 95% CI)

Metric	Attorneys, mean (95% CI)	Models, mean (95% CI)
Redlined passages per task	18.5 (15.9 to 21.2)	19.0 (16.3 to 21.8)
Edits per task	125 (102 to 148)	41 (31 to 51)
Average edit length (characters)	93 (80 to 106)	300 (270 to 329)
Total edit volume (characters)	10,675 (8,821 to 12,529)	11,718 (9,462 to 13,974)
Inline share	63.5% (58.5% to 68.6%)	22.0% (17.8% to 26.3%)
Edits per Turn 2 task	63 (53 to 74)	9 (8 to 10)

On Turn 1, attorneys and models redline almost the same number of passages per task, 18.5 versus 19.0, and change a similar amount of text overall, 10,675 versus 11,718 characters. The structure of those edits differs sharply. Attorneys make 125 tracked edits per task at an average length of 93 characters. Models make 41 edits at an average length of 300 characters.

Models therefore cover a similar document footprint with roughly one-third as many edits, each more than three times longer. On Turn 2, the gap widens further: attorneys make 63 tracked edits per task on average, compared with nine for models. That pattern matches the earlier Turn 2 results, where models score lower when responding to an existing redline.

Editing behavior and rubric reward

6.3

Redlining more passages loosely correlates with reward; longer edits do not.

Table 12. Correlation of rubric reward with editing behavior (Pearson r)

Turn	r(reward, redlined passages)	r(reward, insertion length)
Turn 1 (pooled over tasks)	+0.29	+0.14
Turn 2 (pooled over tasks)	+0.23	+0.05

The number of redlined passages has a modest positive relationship with rubric reward, at r = 0.29 on Turn 1 and r = 0.23 on Turn 2. Median insertion length shows little relationship, at r = 0.14 and r = 0.05. Models therefore gain more from identifying additional relevant issues than from making individual edits longer. This suggests the rubric taxonomy is robust to a simple reward-hacking strategy of inflating edits through large block rewrites.

Conclusion

Contract review is a judgment problem, and ContractBench makes the judgment behind the markup measurable. Even the leading models satisfy fewer than 60% of weighted attorney-defined criteria on completed redlines. Models perform best when preserving acceptable language, but struggle with high-priority issues, positions that depend on the facts of the deal, and responding to counterparty redlines. GPT-6 Astra's strength in escalation, despite its lower overall ranking, demonstrates why aggregate scores alone cannot capture the full range of model capabilities.

Attorneys redline with a scalpel, making frequent, small, inline edits; every model we tested relies predominantly on block rewrites. ContractBench measures this difference through surgicality and editing volume, capturing a dimension of production readiness that substantive scores alone miss.

ContractBench sets a standard that will outlast the models it currently ranks. It scores substance and style separately because attorneys still sit in the loop, and reporting the two apart tells practitioners where a model can be trusted and where it needs supervision. We also believe substantive judgment should remain the priority for post-training, and a leaderboard ranked on substance keeps that signal clean. The limitation is that such a leaderboard can reward the right positions reached through rewrites no attorney would send. The direction for ContractBench v2 is to fold surgicality into the overall leaderboard score, so that as models take on more of the redline end to end, the benchmark rewards a redline that is right on the substance and ready to send as written.

Explore the Ivo-micro1 ContractBench

Hugging Face

Ivo-micro1 ContractBench, October 13, 2026.

Ivo-micro1

ContractBench