Contract Review Comparison
An independent evaluation of AI-powered contract review tools
Introduction
Contract review makes up the majority of the work done by in-house legal departments; enterprise legal teams review thousands of contracts per year. In order to streamline the laborious manual contract review process, AI-powered contract review and analysis providers have created an extremely large and valuable software category. There are hundreds of companies offering these services and the category is expected to grow at a 29.4% CAGR through 2030.
This presents the question: How do these tools actually perform at typical contract review tasks? Do legal teams need to invest in a specialized legal tool or will a generic AI tool like Claude work just as well?
To find out, we assembled a panel of five senior attorneys to judge the contract review output of Ivo, Claude Opus 4.6 for Word, and an experienced human attorney.
Ivo outperformed Claude by a significant margin. In addition, its performance was comparable to a senior practicing commercial attorney at an AmLaw 25 law firm.
Methodology
Ivo conducted an benchmarking study comparing a purpose-built legal AI tool to a general AI tool and a human for redlining tasks
The Participants
- Ivo's latest release
- Claude for Word (Opus 4.6)
- A practicing Special Counsel at an Amlaw 25 firm with 8 years experience
The Contracts
20 real, anonymized contracts were reviewed, spanning NDAs, MSAs, DPAs, and other types of commercial agreements.
The Judges
The outputs were judged by five senior attorneys from the Fenwick FLEX program. Every output was stripped of identifying information and scored blind across five different criteria for contract review.
The Scoring
Outputs were judged on a scale of 1-10. Final scores represent the mean across all five judges.
Overall scores
Ivo’s performance was nearly indistinguishable from a human lawyer and significantly outperformed Claude on the five judging criteria.
Ivo
Human attorney
Claude
Key learnings
Purpose-built legal AI cannot be replicated by general AI.
Specialist teams have spent years crafting the prompts, logic, and outputs that create redlines comparable to top performing senior lawyers. General AI tools cannot yet compare to the performance.
The largest score delta was in surgical redlining and judgment.
Ivo’s team has spent a great deal of time perfecting its surgical redlining abilities. In addition, Ivo excelled at selecting which position is best for the client. Ivo also excelled at analyzing complex contracts with complicated transactions.
Ivo’s output is broadly comparable to a senior practicing attorney at a highly regarded law firm.
The human attorney and Ivo had very similar scores, suggesting Ivo’s output was comparable to a high-performing senior lawyer. However, the attorney completed their redlining tasks in 10 hours, whereas the AI tools took around 10 minutes.
Issue spotting test analysis
Did the author spot all the issues within the scope of the playbook? Did the author over-spot issues that may not apply to this contract?
The playbook prescribes California as the preferred governing law, with Delaware and New York as acceptable alternatives. Binding arbitration (JAMS or AAA) is the preferred dispute mechanism, preceded by a 15–30 day good-faith escalation step. Governing law should never be silent, and litigation in the counterparty's home jurisdiction should be avoided.
15.11 Governing Law; Submission to Jurisdiction Dispute Resolution. This Agreement is governed by and shall be construed in accordance with the internal laws of the State of Delaware without giving effect to any choice or conflict of law provision or rule that would require or permit the application of the laws of any other jurisdiction. Any Action dispute, controversy or claim arising out of or related to this Agreement, the licenses granted hereunder or the transactions contemplated hereby shall be instituted exclusively in the federal courts of the United States of America or the courts of the State of Utah, in each case located in Salt Lake City, and each party irrevocably submits to the exclusive jurisdiction of such courts in any such Action first be submitted to the senior executives of each party for resolution through good-faith negotiation for a period of fifteen (15) days following written notice of such dispute. If the dispute is not resolved within such period, either party may submit the dispute to final and binding arbitration administered by JAMS in accordance with its Comprehensive Arbitration Rules and Procedures. The arbitration shall be conducted by a single arbitrator and shall take place in Wilmington, Delaware. The arbitrator's award shall be final and binding and may be entered as a judgment in any court of competent jurisdiction. Notwithstanding the foregoing, either party may seek equitable relief in any court of competent jurisdiction as provided in Section 15.13. In the event of any Action arising out of or related to this Agreement, the licenses granted hereunder or the transactions contemplated hereby, the prevailing party thereto shall be entitled to, in addition to any other damages assessed, its reasonable attorneys' fees and all other costs and expenses incurred in connection therewith, including, without limitation, cost of collection and enforcement and in pursuit of insurance claims; provided that any obligation by Ivo AI hereunder remains subject to Section 13.2.
Evaluation
The playbook prescribes California as the preferred governing law, with Delaware and New York as acceptable alternatives. Binding arbitration (JAMS or AAA) is the preferred dispute mechanism, preceded by a 15–30 day good-faith escalation step. Governing law should never be silent, and litigation in the counterparty's home jurisdiction should be avoided.
15.11 Governing Law; Submission to Jurisdiction. This Agreement shall be subject to and construed, interpreted, and applied in accordance with the laws of the State of California, United States of America. Any dispute, claim or controversy arising out of or relating to this Agreement or the breach, termination, enforcement, interpretation or validity thereof, shall be determined by arbitration in San Francisco California USA before a single arbitrator. The arbitration shall be administered by JAMS pursuant to its Comprehensive Arbitration Rules and Procedures. The language of the arbitration shall be English. The arbitral award, which shall be final and binding on both parties, may be enforced in any court having jurisdiction thereof. Any discovery as part of the arbitration process shall include the right to subpoena. Parties acknowledge and agree that the U.N. Convention on Contracts for the International Sale of Goods and the Uniform Computer Information Transactions Act shall not apply to this Agreement. This Agreement is governed by and shall be construed in accordance with the internal laws of the State of Delaware without giving effect to any choice or conflict of law provision or rule that would require or permit the application of the laws of any other jurisdiction. Any Action arising out of or related to this Agreement, the licenses granted hereunder or the transactions contemplated hereby shall be instituted exclusively in the federal courts of the United States of America or the courts of the State of Utah, in each case located in Salt Lake City, and each party irrevocably submits to the exclusive jurisdiction of such courts in any such Action. In the event of any Action arising out of or related to this Agreement, the licenses granted hereunder or the transactions contemplated hereby, the prevailing party thereto shall be entitled to, in addition to any other damages assessed, its reasonable attorneys' fees and all other costs and expenses incurred in connection therewith, including, without limitation, cost of collection and enforcement and in pursuit of insurance claims; provided that any obligation by Ivo AI hereunder remains subject to Section 13.2.
Evaluation
This revision changed to California law with JAMS arbitration in San Francisco but did not include a pre-arbitration negotiation step.
15.11 Governing Law; Submission to Jurisdiction. This Agreement is governed by and shall be construed in accordance with the internal laws of the State of California without giving effect to any choice or conflict of law provision or rule that would require or permit the application of the laws of any other jurisdiction. Any Action arising out of or related to this Agreement, the licenses granted hereunder or the transactions contemplated hereby shall be instituted exclusively in the federal courts of the United States of America or the courts of the State of California, in each case located in San Francisco, This Agreement is governed by and shall be construed in accordance with the internal laws of the State of Delaware without giving effect to any choice or conflict of law provision or rule that would require or permit the application of the laws of any other jurisdiction. Any Action arising out of or related to this Agreement, the licenses granted hereunder or the transactions contemplated hereby shall be instituted exclusively in the federal courts of the United States of America or the courts of the State of Utah, in each case located in Salt Lake City, and each party irrevocably submits to the exclusive jurisdiction of such courts in any such Action. In the event of any Action arising out of or related to this Agreement, the licenses granted hereunder or the transactions contemplated hereby, the prevailing party thereto shall be entitled to, in addition to any other damages assessed, its reasonable attorneys' fees and all other costs and expenses incurred in connection therewith, including, without limitation, cost of collection and enforcement and in pursuit of insurance claims; provided that any obligation by Ivo AI hereunder remains subject to Section 13.2.
Evaluation
This revision changed it to California law with litigation in San Francisco courts. It addresses the playbook's preference for California but uses litigation rather than the playbook's preferred arbitration mechanism.
Redlining test analysis
Did the author make minimal, precise changes to fix the issue, or did they aggressively rewrite the whole paragraph?
Neither party should assign without consent, except to affiliates or in connection with M&A. Require 30-day advance written notice. Restrict assignment to direct competitors.
11.5 Assignment; Delegation. Neither this Agreement nor any right or obligation hereunder may be assigned or otherwise transferred (whether voluntarily, by operation of law or otherwise), without the prior express written consent of the other party; provided, however, that either party may, without such consent, assign this Agreement and its rights and obligations hereunder (a) to an affiliate of such party, or (b) in connection with the transfer or sale of all or substantially all of its business or assets related to this Agreement, or in the event of its merger, consolidation, change in control or other similar transaction, provided that in each case the assigning party provides at least thirty (30) days' advance written notice to the other party and the assignee is not a direct competitor of the other party. Any permitted assignee shall assume all obligations of its assignor under this Agreement. Any purported assignment or transfer in violation of this Section will be null, void and of no effect.
Evaluation
Made two precise insertions into the existing sentence without deleting or restructuring any original text: (1) inserted the affiliate assignment right the playbook calls for, and (2) appended the written notice requirement and the direct competitor restriction as a trailing proviso. The original sentence structure, the M&A carve-out language, and all existing terms were preserved exactly as written.
11.5 Assignment; Delegation. Neither this Agreement nor any right or obligation hereunder may be assigned or otherwise transferred (whether voluntarily, by operation of law or otherwise), without the prior express written consent of the other party; provided, however, that either party may, without such consent, assign this Agreement and its rights and obligations hereunder in connection with the transfer or sale of all or substantially all of its business or assets related to this Agreement, or in the event of its merger, consolidation, change in control or other similar transaction, provided, further, that Customer shall provide Ivo AI with 30 days' prior written notice with respect to any such assignment, and that, notwithstanding the foregoing, Customer shall not assign this Agreement to any direct competitor of Ivo AI without Ivo AI's prior written consent. Any permitted assignee shall assume all obligations of its assignor under this Agreement. Any purported assignment or transfer in violation of this Section will be null, void and of no effect.
Evaluation
Clean insertion of the written notice requirement and the direct competitor restriction. However, some drafters may deem it as redundant. In addition, it missed the affiliate carveout.
11.5 Assignment; Delegation. Neither this Agreement nor any right or obligation hereunder may be assigned or otherwise transferred (whether voluntarily, by operation of law or otherwise), without the prior express written consent of the other party; provided, however, that either party may, without such consent, assign this Agreement and its rights and obligations hereunder in connection with the transfer or sale of all or substantially all of its business or assets related to this Agreement, or in the event of its merger, consolidation, change in control or other similar transaction, subject to thirty (30) days' advance written notice to the other party; provided, further, that neither party may assign this Agreement to a direct competitor of the other party without such other party's prior written consent. provided, however, that either party may, without such consent, assign this Agreement and its rights and obligations hereunder in connection with the transfer or sale of all or substantially all of its business or assets related to this Agreement, or in the event of its merger, consolidation, change in control or other similar transaction. Any permitted assignee shall assume all obligations of its assignor under this Agreement. Any purported assignment or transfer in violation of this Section will be null, void and of no effect.
Evaluation
This revision completely deleted the existing M&A carve-out clause and replaced it with a re-written version that adds the competitor restriction and 30-day notice. It also missed the affiliate carveout.
Judgment test analysis
Did the author pick the right position for each issue when there are different fallback options? When some playbook rules have ambiguity and a strict compliance may harm the party’s interest, did they make judgments that are the best for the party?
Playbook prefers Delaware or California as governing laws, but also accepts any reasonable US state where the counterparty has a nexus. Playbook also prefers litigation over arbitration and accepts counterparty proposed single-arbitrator arbitration under AAA rules (domestic) or ICC rules (international). Counterparty's preferred arbitration forum is acceptable if it's reasonable and commercially standard
16. Miscellaneous.
(a) Governing Law. The validity, interpretation, construction and performance of this Agreement, and all acts and transactions pursuant hereto and the rights and obligations of the parties hereto shall be governed, construed and interpreted in accordance with the laws of the state of Wisconsin, without giving effect to principles of conflicts of law.
Evaluation
Ivo left the Wisconsin governing law clause entirely untouched. This reflects correct application of the playbook, which permits any reasonable US state where the counterparty has a nexus, and the counterparty in this agreement is a Wisconsin company.
14. Miscellaneous.
(a) Governing Law. This Agreement shall be subject to and construed, interpreted, and applied in accordance with the laws of the State of California, United States of America. Any dispute, claim or controversy arising out of or relating to this Agreement or the breach, termination, enforcement, interpretation or validity thereof, shall be determined by arbitration in San Francisco California USA before a single arbitrator. The arbitration shall be administered pursuant to [AAA rules]. The language of the arbitration shall be English. The arbitral award, which shall be final and binding on both parties, may be enforced in any court having jurisdiction thereof. Any discovery as part of the arbitration process shall include the right to subpoena. Parties acknowledge and agree that the U.N. Convention on Contracts for the International Sale of Goods and the Uniform Computer Information Transactions Act shall not apply to this Agreement. The validity, interpretation, construction and performance of this Agreement, and all acts and transactions pursuant hereto and the rights and obligations of the parties hereto shall be governed, construed and interpreted in accordance with the laws of the state of Wisconsin, without giving effect to principles of conflicts of law.
Evaluation
This revision changed to California law with JAMS arbitration in San Francisco but did not include a pre-arbitration negotiation step.
15. Miscellaneous.
(a) Governing Law. The validity, interpretation, construction and performance of this Agreement, and all acts and transactions pursuant hereto and the rights and obligations of the parties hereto shall be governed, construed and interpreted in accordance with the laws of the State of Delaware state of Wisconsin, without giving effect to principles of conflicts of law.
Evaluation
A clean word-level swap, mechanically precise. But substantively unnecessary — the playbook allows the counterparty's home-state law when a nexus exists. This spends negotiating capital on a non-critical point.
Formatting test analysis
When the author makes changes, did they maintain font, spacing, paragraph numbering, and cross-references flawlessly in Word? Did they respect and correctly capitalize defined terms specific to this document?
13. Injunctive Relief. Each party hereby acknowledges and agrees that in the event of any breach of this Agreement by such party, including, without limitation, the actual or threatened disclosure or unauthorized use of Confidential Information that no remedy at law would adequately protect or appropriately compensate the other party for such injury. Accordingly, each party agrees that in such an event, the other party shall be entitled to seek equitable relief.
Limitation of Liability. IN NO EVENT WILL EITHER PARTY BE LIABLE TO THE OTHER PARTY FOR ANY SPECIAL, INDIRECT, PUNITIVE, INCIDENTAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES OR LOSSES, INCLUDING LOST PROFITS OR LOSS OF BUSINESS OPPORTUNITY, ARISING OUT OF OR RELATING TO THIS AGREEMENT.
Evaluation
It is acceptable, and preferred by some drafters, to break the consequential damages waiver out as its own numbered section, as shown in this redline. However, it strips the bold formatting of the section header and does not add a section number.
8. No Warranties. The Confidential Information is provided "as is."THE DISCLOSING PARTY EXPRESSLY DISCLAIMS ANY AND ALL WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT WILL EITHER PARTY BE LIABLE UNDER ANY CIRCUMSTANCES TO THE OTHER PARTY FOR SPECIAL, INDIRECT, PUNITIVE, INCIDENTAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES OR LOSSES, INCLUDING LOST PROFITS, LOSS OF BUSINESS OPPORTUNITY OR OTHER SIMILAR DAMAGES RESULTING FROM OR ARISING OUT OF OR RELATING TO THIS AGREEMENT. Without limiting the generality of the foregoing, the Disclosing Party makes no warranty as to the accuracy or completeness of the Confidential Information. Neither the Disclosing Party nor any of its Representatives will have any liability to the Receiving Party or to any of the Receiving Party's Representatives relating to or resulting from the use of any of the Disclosing Party's Confidential Information or any inaccuracies or errors therein or omissions therefrom, except as may be provided in any additional definitive written agreement entered into by the parties.
Evaluation
This revision inserted the consequential damages waiver immediately after the warranties disclaimer and before the "Without limiting the generality of the foregoing…" sentence about accuracy/completeness. This creates a natural logical flow: (1) warranty disclaimer → (2) damages limitation → (3) no liability for inaccuracies.
8. No Warranties. The Confidential Information is provided "as is."THE DISCLOSING PARTY EXPRESSLY DISCLAIMS ANY AND ALL WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. Without limiting the generality of the foregoing, the Disclosing Party makes no warranty as to the accuracy or completeness of the Confidential Information. Neither the Disclosing Party nor any of its Representatives will have any liability to the Receiving Party or to any of the Receiving Party's Representatives relating to or resulting from the use of any of the Disclosing Party's Confidential Information or any inaccuracies or errors therein or omissions therefrom, except as may be provided in any additional definitive written agreement entered into by the parties. IN NO EVENT WILL EITHER PARTY BE LIABLE UNDER ANY CIRCUMSTANCES TO THE OTHER PARTY FOR SPECIAL, INDIRECT, PUNITIVE, INCIDENTAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES OR LOSSES, INCLUDING LOST PROFITS, LOSS OF BUSINESS OPPORTUNITY OR OTHER SIMILAR DAMAGES RESULTING FROM OR ARISING OUT OF OR RELATING TO THIS AGREEMENT.
Evaluation
This revision inserted the consequential damages waiver in the no warranties section, but unlike the revision by the human attorney, it inserted the redline at the very end. This makes the clause feel tacked on as an afterthought, disconnected from the related warranty disclaimer language above it. In addition, it strips the bold formatting.
Commenting test analysis
Did the author adhere to the playbook rules and added all the approved comments necessary?
Recipient hereby agrees to indemnify, defend and hold Discloser harmless from any and all liabilities, expenses, causes of action, claims and liabilities arising out of or resulting, to any degree, from the unauthorized disclosure of Confidential Information of Discloser.
The obligations set forth in this Agreement shall continue for a period of five years following disclosure of such Confidential Information to Recipient. Either Party may terminate this Agreement for convenience upon thirty (30) days' prior written notice to the other Party. The confidentiality obligations set forth herein shall survive any expiration or termination of this Agreement.
Evaluation
Ivo attached the standard language for external comment to the appropriate location. The tone is professional and the message is clear.
21.2. Disputes in General
The Parties will resolve all Disputes in accordance with the procedures described in the Dispute Resolution Procedures Schedule.
22. GENERAL
22.1. Relationship of Parties
(a) No Joint Venture
The Agreement shall not be construed as constituting either Party as partner, joint venture or fiduciary of the other Party or to create any other form of legal association that would impose liability upon one Party for the act or failure to act of the other Party, or as providing either Party with the right, power or authority (express or implied) to create any duty or obligation of the other Party.
Evaluation
Comments show clear understanding of the playbook, but they usually mix internal comments with external comments and may review internal negotiation strategy.
Recipient hereby agrees to indemnify, defend and hold Discloser harmless from any and all liabilities, expenses, causes of action, claims and liabilities arising out of or resulting, to any degree, from the unauthorized disclosure of Confidential Information of Discloser.
Evaluation
Claude made the correct revision, but, contrary to the playbook instruction, did not attach the standard language for external comment as instructed by the playbook.
Conclusion
Ivo’s performance on contract redlining was comparable to a practicing senior attorney at a highly regarded law firm, and done in a fraction of the time.
This study aimed to provide a transparent data point on whether domain experts on legal AI perform legal tasks better than general AI tools. There are very rarely objective studies on how AI tools perform; this is a way to benchmark the performance of AI tools on typical legal tasks.
FAQs
Why did Ivo conduct this study?
Objective performance data for AI contract review tools is rare. We wanted to create a rigorous, transparent benchmark that gives legal teams a real basis for evaluating tools, including ours.
Who judged the outputs?
Five senior attorneys from the Fenwick FLEX program. All outputs were stripped of identifying information before scoring, so judges evaluated the work blind across five criteria.
How were the tools tested?
All three participants: Ivo, Claude for Word (Opus 4.6), and a practicing Special Counsel at an AmLaw 25 firm reviewed the same 20 real, anonymized contracts. These spanned NDAs, MSAs, DPAs, and other commercial agreements.
How were outputs scored?
Judges scored each output on a scale of 1–10 across five criteria for contract review: Issue spotting, Surgical redlining, Formatting retention, Judgment, and Comments. Final scores represent the mean across all five judges.
What were the results?
Ivo scored 4.38. The human attorney scored 4.37. Claude for Word scored 3.59. Ivo and the human attorney were nearly indistinguishable; both outperformed Claude by a significant margin.
Where did Ivo succeed?
The largest gap was in surgical redlining and legal judgment; specifically, selecting the right position for the client in context. This is an area where Ivo's team has invested years of product development.
Does this mean Ivo is as good as a lawyer?
On these tasks, Ivo's output was comparable to a senior practicing attorney at a highly regarded law firm. The attorney completed the same work in roughly 10 hours; Ivo took about 10 minutes.
What are the limitations of this study?
This benchmark reflects a single point in time (April 2026) with one human baseline, one prompt configuration for Claude, and 20 contracts across six commercial types. The playbook was provided as system configuration for Ivo and as a user prompt for Claude — which reflects realistic deployment conditions, not a controlled model-to-model comparison.
It is against Ivo AI standard procedure to agree to indemnify or defend at the NDA phase. We do not believe it is the right time for either party to take on indemnification costs before any real value exchange. Without this indemnity provision, the wronged party still has a remedy as it can sue for damages and collect after proving its case.