Contract Review Comparison
An independent evaluation of AI-powered contract review tools
Introduction
Contract review dominates the workload of in-house legal departments — enterprise teams process thousands per year. Legal leaders know that AI could help ease the burden. The AI-powered contract analysis market is now worth $4.3 billion and projected to reach $12.06 billion by 2030. But the category is now so crowded that evaluating vendors has become difficult. Our research found that over 50% of respondents struggle to objectively compare AI tools across vendors.
We created an independent benchmarking study to answer three questions:
- What's the right way to objectively measure the output of AI tools?
- Can AI tools' claims be independently verified?
- Why invest in a purpose-built legal product like Ivo when lawyers are already using general AI tools like Claude and Microsoft Copilot?
We assembled a panel of three senior attorneys to evaluate contract review output from Ivo, Claude for Word (Opus 4.6), and a Practicing Special Counsel at an Am Law 25 firm.
Ivo outperformed Claude by a significant margin, and its performance was comparable to the accomplished human attorney.
Methodology
Ivo conducted a benchmarking study comparing a purpose-built legal AI tool to a general AI tool and a human for redlining tasks.
The Participants
- Ivo's latest release
- Claude for Word (Opus 4.6)
- A practicing Special Counsel at an Am Law 25 firm with 8 years experience
The Contracts
19 real, anonymized contracts were reviewed, spanning NDAs, MSAs, and DPAs.
The Judges
The outputs were judged by three attorneys with recent experience either working with an Am Law 100 firm or serving as in-house counsel at technology companies. Every output was stripped of identifying information and scored blind across five different criteria for contract review.
The Scoring
Outputs were judged on a scale of 1-10. Final scores represent the mean across all three judges.
Overall scores
Ivo’s performance was nearly indistinguishable from a human lawyer and significantly outperformed Claude on the five judging criteria.
Ivo
Claude for Word
Human Attorney
Key learnings
Purpose-built legal AI cannot be replicated by general AI.
Specialist teams have spent years crafting the prompts, logic, and outputs that create redlines comparable to top performing senior lawyers. General AI tools cannot yet compare to the performance.
The largest score delta was in Surgical Redlining and Judgment.
Ivo’s team has spent a great deal of time perfecting its surgical redlining abilities. In addition, Ivo achieved the highest scores in legal judgment, even higher than the human attorney. Finally, Ivo excelled at analyzing complex contracts with complicated transactions.
Ivo’s output is broadly comparable to a senior practicing attorney at a highly regarded law firm.
The human attorney and Ivo had very similar scores, suggesting Ivo’s output was comparable to a high-performing senior lawyer. However, the attorney completed their redlining tasks in 10 hours, whereas Ivo’s average performance was 2 minutes and 45 seconds and Claude for Word’s was 4 minutes and 53 seconds.
Average time for each participant to finish reviewing a document
Ivo
Claude for Word
Human Attorney
Issue Spotting analysis
Did the author spot all the issues within the scope of the playbook? Did the author over-spot issues that may not apply to this contract?
The playbook prescribes California as the preferred governing law, with Delaware and New York as acceptable alternatives. Binding arbitration (JAMS or AAA) is the preferred dispute mechanism, preceded by a 15–30 day good-faith escalation step. Governing law should never be silent, and litigation in the counterparty's home jurisdiction should be avoided.
15.11 Governing Law; Submission to JurisdictionDispute Resolution. This Agreement is governed by and shall be construed in accordance with the internal laws of the State of Delaware without giving effect to any choice or conflict of law provision or rule that would require or permit the application of the laws of any other jurisdiction. Any Action dispute, controversy or claim arising out of or related to this Agreement, the licenses granted hereunder or the transactions contemplated hereby shall be instituted exclusively in the federal courts of the United States of America or the courts of the State of Utah, in each case located in Salt Lake City, and each party irrevocably submits to the exclusive jurisdiction of such courts in any such Action first be submitted to the senior executives of each party for resolution through good-faith negotiation for a period of fifteen (15) days following written notice of such dispute. If the dispute is not resolved within such period, either party may submit the dispute to final and binding arbitration administered by JAMS in accordance with its Comprehensive Arbitration Rules and Procedures. The arbitration shall be conducted by a single arbitrator and shall take place in Wilmington, Delaware. The arbitrator's award shall be final and binding and may be entered as a judgment in any court of competent jurisdiction. Notwithstanding the foregoing, either party may seek equitable relief in any court of competent jurisdiction as provided in Section 15.13. In the event of any Action arising out of or related to this Agreement, the licenses granted hereunder or the transactions contemplated hereby, the prevailing party thereto shall be entitled to, in addition to any other damages assessed, its reasonable attorneys' fees and all other costs and expenses incurred in connection therewith, including, without limitation, cost of collection and enforcement and in pursuit of insurance claims; provided that any obligation by Party A hereunder remains subject to Section 13.2.
Evaluation
This redline kept Delaware law but replaced litigation with JAMS arbitration in Wilmington, Delaware, added a 15-day good-faith negotiation step, and preserved an equitable relief carveout. This most precisely matches the playbook's preferred structure of binding arbitration with a senior-leadership escalation step before arbitration.
15.11 Governing Law; Submission to Jurisdiction. This Agreement is governed by and shall be construed in accordance with the internal laws of the State of California without giving effect to any choice or conflict of law provision or rule that would require or permit the application of the laws of any other jurisdiction. Any Action arising out of or related to this Agreement, the licenses granted hereunder or the transactions contemplated hereby shall be instituted exclusively in the federal courts of the United States of America or the courts of the State of California, in each case located in San Francisco, This Agreement is governed by and shall be construed in accordance with the internal laws of the State of Delaware without giving effect to any choice or conflict of law provision or rule that would require or permit the application of the laws of any other jurisdiction. Any Action arising out of or related to this Agreement, the licenses granted hereunder or the transactions contemplated hereby shall be instituted exclusively in the federal courts of the United States of America or the courts of the State of Utah, in each case located in Salt Lake City, and each party irrevocably submits to the exclusive jurisdiction of such courts in any such Action. In the event of any Action arising out of or related to this Agreement, the licenses granted hereunder or the transactions contemplated hereby, the prevailing party thereto shall be entitled to, in addition to any other damages assessed, its reasonable attorneys' fees and all other costs and expenses incurred in connection therewith, including, without limitation, cost of collection and enforcement and in pursuit of insurance claims; provided that any obligation by Party A hereunder remains subject to Section 13.2.
Evaluation
This revision changed it to California law with litigation in San Francisco courts. It addresses the playbook's preference for California but uses litigation rather than the playbook's preferred arbitration mechanism.
15.11 Governing Law; Submission to Jurisdiction. This Agreement shall be subject to and construed, interpreted, and applied in accordance with the laws of the State of California, United States of America. Any dispute, claim or controversy arising out of or relating to this Agreement or the breach, termination, enforcement, interpretation or validity thereof, shall be determined by arbitration in San Francisco California USA before a single arbitrator. The arbitration shall be administered by JAMS pursuant to its Comprehensive Arbitration Rules and Procedures. The language of the arbitration shall be English. The arbitral award, which shall be final and binding on both parties, may be enforced in any court having jurisdiction thereof Any discovery as part of the arbitration process shall include the right to subpoena. Parties acknowledge and agree that the U.N. Convention on Contracts for the International Sale of Goods and the Uniform Computer Information Transactions Act shall not apply to this Agreement. This Agreement is governed by and shall be construed in accordance with the internal laws of the State of Delaware without giving effect to any choice or conflict of law provision or rule that would require or permit the application of the laws of any other jurisdiction. Any Action arising out of or related to this Agreement, the licenses granted hereunder or the transactions contemplated hereby shall be instituted exclusively in the federal courts of the United States of America or the courts of the State of Utah, in each case located in Salt Lake City, and each party irrevocably submits to the exclusive jurisdiction of such courts in any such Action. In the event of any Action arising out of or related to this Agreement, the licenses granted hereunder or the transactions contemplated hereby, the prevailing party thereto shall be entitled to, in addition to any other damages assessed, its reasonable attorneys' fees and all other costs and expenses incurred in connection therewith, including, without limitation, cost of collection and enforcement and in pursuit of insurance claims; provided that any obligation by Party A hereunder remains subject to Section 13.2.
Evaluation
This revision changed to California law with JAMS arbitration in San Francisco but did not include a pre-arbitration negotiation step.
Surgical Redlining analysis
Did the author make minimal, precise changes to fix the issue, or did they aggressively rewrite the whole paragraph?
Neither party should assign without consent, except to affiliates or in connection with M&A. Require 30-day advance written notice. Restrict assignment to direct competitors.
11.5 Assignment; Delegation. Neither this Agreement nor any right or obligation hereunder may be assigned or otherwise transferred (whether voluntarily, by operation of law or otherwise), without the prior express written consent of the other party; provided, however, that either party may, without such consent, assign this Agreement and its rights and obligations hereunder (a) to an affiliate of such party, or (b) in connection with the transfer or sale of all or substantially all of its business or assets related to this Agreement, or in the event of its merger, consolidation, change in control or other similar transaction, provided that in each case the assigning party provides at least thirty (30) days' advance written notice to the other party and the assignee is not a direct competitor of the other party. Any permitted assignee shall assume all obligations of its assignor under this Agreement. Any purported assignment or transfer in violation of this Section will be null, void and of no effect.
Evaluation
Made two precise insertions into the existing sentence without deleting or restructuring any original text: (1) inserted the affiliate assignment right the playbook calls for, and (2) appended the written notice requirement and the direct competitor restriction as a trailing proviso. The original sentence structure, the M&A carve-out language, and all existing terms were preserved exactly as written.
11.5 Assignment; Delegation. Neither this Agreement nor any right or obligation hereunder may be assigned or otherwise transferred (whether voluntarily, by operation of law or otherwise), without the prior express written consent of the other party; provided, however, that either party may, without such consent, assign this Agreement and its rights and obligations hereunder in connection with the transfer or sale of all or substantially all of its business or assets related to this Agreement, or in the event of its merger, consolidation, change in control or other similar transaction, subject to thirty (30) days' advance written notice to the other party; provided, further, that neither party may assign this Agreement to a direct competitor of the other party without such other party's prior written consent. provided, however, that either party may, without such consent, assign this Agreement and its rights and obligations hereunder in connection with the transfer or sale of all or substantially all of its business or assets related to this Agreement, or in the event of its merger, consolidation, change in control or other similar transaction. Any permitted assignee shall assume all obligations of its assignor under this Agreement. Any purported assignment or transfer in violation of this Section will be null, void and of no effect.
Evaluation
This revision completely deleted the existing M&A carve-out clause and replaced it with a re-written version that adds the competitor restriction and 30-day notice. It also missed the affiliate carveout.
11.5 Assignment; Delegation. Neither this Agreement nor any right or obligation hereunder may be assigned or otherwise transferred (whether voluntarily, by operation of law or otherwise), without the prior express written consent of the other party; provided, however, that either party may, without such consent, assign this Agreement and its rights and obligations hereunder in connection with the transfer or sale of all or substantially all of its business or assets related to this Agreement, or in the event of its merger, consolidation, change in control or other similar transaction, provided, further, that Customer shall provide Party A with 30 days' prior written notice with respect to any such assignment, and that, notwithstanding the foregoing, Customer shall not assign this Agreement to any direct competitor of Party A without Party A’s prior written consent. Any permitted assignee shall assume all obligations of its assignor under this Agreement. Any purported assignment or transfer in violation of this Section will be null, void and of no effect.
Evaluation
Clean insertion of the written notice requirement and the direct competitor restriction. However, some drafters may deem it as redundant. In addition, it missed the affiliate carveout.
Formatting Retention analysis
When the author makes changes, did they maintain font, spacing, paragraph numbering, and cross-references flawlessly in Word? Did they respect and correctly capitalize defined terms specific to this document?
Limited Warranty: The counterparty should warrant authority to enter into the agreement, that services will be performed in a professional and workmanlike manner, that deliverables will conform to specifications, and that the counterparty's performance will not violate applicable laws or third-party rights. For software and SaaS, the counterparty should also warrant that the product will perform materially in accordance with its documentation during the subscription term.
CCPA: Playbook requires that whenever CCPA/CPRA applies, the DPA must include the statutory contractual terms for service providers or contractors. The processor must certify it understands and will comply with its obligations, must not sell or share personal information, must not retain or use personal information outside the direct business relationship, and must notify Party A if it can no longer meet its obligations. These provisions should be included even when the DPA is primarily GDPR-focused, to ensure cross-jurisdictional coverage.
CCPA/CPRA-SPECIFIC PROVISIONS
To the extent the CCPA applies to the processing of Personal Data under this Data Processing Agreement, Processor (as a "service provider" or "contractor" as defined under the CCPA) certifies that it understands and will comply with its obligations under the CCPA and shall:
not sell or share Personal Data;
not retain, use, or disclose Personal Data for any purpose other than for the specific purposes of performing the Services under the Agreement, or as otherwise permitted by the CCPA;
not retain, use, or disclose Personal Data outside of the direct business relationship between Controller and Processor; and
promptly notify Controller if it determines that it can no longer meet its obligations under the CCPA.
Evaluation
This redline inserts the CCPA provision as a standalone clause, structured with sub-clauses. It preserves the document's formatting conventions such as the section heading being bold and in all caps, and the body text being left-aligned. However, it omits a section or sub-section number.
14. CCPA/CPRA PROVISIONS
14.1. To the extent the CCPA applies to the processing of Personal Data under this Data Processing Agreement, Processor (as a "service provider" under the CCPA) shall: (a) not sell or share Personal Data; (b) not retain, use, or disclose Personal Data for any purpose other than for the specific purpose of performing the Services, including not retaining, using, or disclosing Personal Data for a commercial purpose other than providing the Services; (c) not combine Personal Data received from or on behalf of Controller with personal information that Processor receives from or on behalf of another person or collects from its own interaction with data subjects, except as permitted by the CCPA; and (d) promptly notify Controller if Processor determines that it can no longer meet its obligations under the CCPA.
Evaluation
This revision inserts the CCPA provisions as a standalone clause. The section heading is bold and in all caps and it includes section numbers, but the entire text is unnecessarily bolded and center-aligned.
10. Limited Warranty. Party B represents and warrants that (a) it will perform Services (i) in accordance with (i) the terms of this Agreement, and (ii) in accordance with all applicable industry standards and all applicable laws, regulations, rules, published guidelines and, if applicable, generally accepted standards of good clinical practice and SOPs (collectively, the "Applicable Laws"), and (iii) in a professional and workmanlike manner; (b) Party B and Party B Personnel or any other person used by Party B to perform Services has not been (i) debarred, convicted, or is subject to a pending debarment
Evaluation
This revision adjusted the clause numbers when making redline edits and shows clear understanding of the different layers of the sub-clauses. It did not introduce any formatting that conflicts with the rest of the agreement.
Commenting analysis
Did the author adhere to the playbook rules and add all the approved comments necessary?
Indemnification: Do not agree to indemnification of any kind at the NDA stage. This is a firm Party A position. Indemnification is an inappropriate shifting of risk before any real value exchange. Without an indemnity provision, the wronged party still has a remedy—it can sue for damages and collect after proving its case.
Fallback: Escalate to senior legal counsel. In certain civil law jurisdictions (France, Italy, Germany, Spain, Netherlands, Belgium), accept reasonable indemnification limited to direct damages in a final, non-appealable order.
Comment language for counterparty markup:“It is against Party A standard procedure to agree to indemnify or defend at the NDA phase. We do not believe it is the right time for either party to take on indemnification costs before any real value exchange. Without this indemnity provision, the wronged party still has a remedy as it can sue for damages and collect after proving its case.”
One-Sided vs. Mutual NDA: Party A’s strong preference is always mutual. If a counterparty proposes a one-way NDA that only requires Party A to protect the counterparty's information, you should push back and request it be converted to a mutual agreement — or at minimum ensure Party A’s confidential information receives equivalent protection.The standard comment language for this is: "Please note this is a one-way NDA requiring only Party A to protect the counterparty's information. As Party A will also be sharing confidential information, we request that this be converted to a mutual NDA or that equivalent protections be added for Party A's confidential information."
Recipient hereby agrees to indemnify, defend and hold Discloser harmless from any and all liabilities, expenses, causes of action, claims and liabilities arising out of or resulting, to any degree, from the unauthorized disclosure of Confidential Information of Discloser.
The obligations set forth in this Agreement shall continue for a period of five years following disclosure of such Confidential Information to Recipient. Either Party may terminate this Agreement for convenience upon thirty (30) days' prior written notice to the other Party. The confidentiality obligations set forth herein shall survive any expiration or termination of this Agreement.
Evaluation
Ivo attached the standard language for external comment to the appropriate location. The tone is professional and the message is clear.
Recipient hereby agrees to indemnify, defend and hold Discloser harmless from any and all liabilities, expenses, causes of action, claims and liabilities arising out of or resulting, to any degree, from the unauthorized disclosure of Confidential Information of Discloser.
Evaluation
Claude made the correct revision, but, contrary to the playbook instruction, did not attach the standard language for external comment as instructed by the playbook.
THIS NON-DISCLOSURE AGREEMENT (this "Agreement") is made as of __________ (the "Effective Date") by and between Party A, Inc. ("Party A" and together with its direct and indirect subsidiariesAffiliates, "Disclosing Party") and Party B ("Recipient"). For the purposes of this Agreement, each of Recipient and Disclosing Party is sometimes referred to, individually, as a "Party" and Recipient and Disclosing Party are sometimes referred to as, together, the "Parties".
The playbook says "Party A's strong preference is always to enter into a mutual NDA," but only requires escalation in the event of a one-way NDA that only requires Party A to bear confidentiality obligations. A one-way NDA where only Party A is disclosing appears acceptable.
Evaluation
Comments show clear understanding of the playbook. However, the reviewer needs to review all comments and manually delete all internal comments.
Judgment analysis
Did the author pick the right position for each issue when there are different fallback options? When some playbook rules are ambiguous and strict compliance may harm the party’s interest, did they make judgments that are the best for the party?
Playbook prefers Delaware or California as governing laws, but also accepts any reasonable US state where the counterparty has a nexus. Playbook also prefers litigation over arbitration and accepts counterparty proposed single-arbitrator arbitration under AAA rules (domestic) or ICC rules (international). Counterparty's preferred arbitration forum is acceptable if it's reasonable and commercially standard
16. Miscellaneous.
(a) Governing Law. The validity, interpretation, construction and performance of this Agreement, and all acts and transactions pursuant hereto and the rights and obligations of the parties hereto shall be governed, construed and interpreted in accordance with the laws of the state of Wisconsin, without giving effect to principles of conflicts of law.
Evaluation
Ivo left the Wisconsin governing law clause entirely untouched. This reflects correct application of the playbook, which permits any reasonable US state where the counterparty has a nexus, and the counterparty in this agreement is a Wisconsin company.
15. Miscellaneous.
(a) Governing Law. The validity, interpretation, construction and performance of this Agreement, and all acts and transactions pursuant hereto and the rights and obligations of the parties hereto shall be governed, construed and interpreted in accordance with the laws of the State of Delawarestate of Wisconsin, without giving effect to principles of conflicts of law.
Evaluation
A clean word-level swap, mechanically precise. But substantively unnecessary — the playbook allows the counterparty's home-state law when a nexus exists. This spends negotiating capital on a non-critical point.
14. Miscellaneous.
(a) Governing Law. This Agreement shall be subject to and construed, interpreted, and applied in accordance with the laws of the State of California, United States of America. Any dispute, claim or controversy arising out of or relating to this Agreement or the breach, termination, enforcement, interpretation or validity thereof, shall be determined by arbitration in San Francisco California USA before a single arbitrator. The arbitration shall be administered pursuant to [AAA rules]. The language of the arbitration shall be English. The arbitral award, which shall be final and binding on both parties, may be enforced in any court having jurisdiction thereof. Any discovery as part of the arbitration process shall include the right to subpoena. Parties acknowledge and agree that the U.N. Convention on Contracts for the International Sale of Goods and the Uniform Computer Information Transactions Act shall not apply to this Agreement.The validity, interpretation, construction and performance of this Agreement, and all acts and transactions pursuant hereto and the rights and obligations of the parties hereto shall be governed, construed and interpreted in accordance with the laws of the state of Wisconsin, without giving effect to principles of conflicts of law.
Evaluation
The entire paragraph was deleted and replaced with a multi-sentence California law and AAA arbitration framework. The playbook's default is litigation, not arbitration — arbitration is only accepted if proposed by the counterparty. Introducing it unprompted is both a substantive overreach and a structural over-edit.
Conclusion
Ivo performed comparably with a very accomplished human lawyer and outperformed Claude for Word on every metric. Underneath the headline are some interesting patterns. It’s exciting that the scores achieved by both Ivo and the human are so close. It took the human lawyer approximately 10 hours to complete the review of all the documents whereas Ivo’s average time to review a contract was 2 minutes and 45 seconds. Ivo performs a deeply manual and lengthy task nearly identically to a human lawyer in a fraction of the time.
Like humans, Ivo excels in certain areas and underperforms in others. A weak area for Ivo is Formatting Retention (e.g. the degree to which the reviewer retains the integrity of the Word document), whereas stronger areas include Surgical Redlining and Judgment. We were not surprised by the strength of Ivo’s performance in Surgical Redlining; our team has spent years perfecting its systems to create precise, accurate redlines. Ivo’s strength in Judgment (e.g. selecting the best outcome for the client) suggested that familiarity with the context of business agreements helps Ivo select the right option when there is more than one choice.
Ivo also significantly outperformed Claude for Word on every metric. The closest score was in Issue Spotting (e.g. identifying problematic provisions in agreements according to a playbook), but it was weak in all other areas. particularly Surgical redlining and Judgment. The score delta was much larger. Ivo’s specialization in legal work comes into play here; it is designed and built to work like a lawyer does. A general AI tool does not have that level of specialization and is therefore not capable of performing the tasks in the same way.
FAQs
Why did Ivo conduct this study?
Objective performance data for AI contract review tools is rare. We wanted to create a rigorous, transparent benchmark that gives legal teams a real basis for evaluating tools, including ours.
Who judged the outputs?
Three attorneys with recent experience either working with an Am Law100 firm or serving as in-house counsel at technology companies. All outputs were stripped of identifying information before scoring, so judges evaluated the work blind across five criteria.
How were the tools tested?
All three participants—Ivo, Claude for Word, and a practicing Special Counsel at an Am Law 25 firm—reviewed the same 19 real, anonymized contracts. These spanned NDAs, MSAs, DPAs, and other commercial agreements.
How were outputs scored?
Judges scored each output on a scale of 1–10 across five criteria for contract review: Issue Spotting, Surgical Redlining, Formatting Retention, Judgment, and Comments. Final scores represent the mean across all five judges.
What were the results?
Ivo scored 4.52. The human attorney scored 4.56. Claude for Word scored 3.50. Ivo and the human attorney were nearly indistinguishable; both outperformed Claude by a significant margin.
Where did Ivo succeed?
The largest gap was in Surgical Redlining and Judgment; specifically, selecting the right position for the client in context. This is an area where Ivo's team has invested years of product development.
Does this mean Ivo is as good as a lawyer?
On these tasks, Ivo's output was comparable to a Special Counsel at an Am Law 25 firm. The attorney completed the same work in roughly 10 hours; Ivo’s average time was 2 minutes and 45 seconds.
What are the limitations of this study?
This benchmark reflects a single point in time (April 2026) with one human baseline, one prompt configuration for Claude for Word, and 19 contracts across six commercial types. The playbook was provided as system configuration for Ivo and as a user prompt for Claude—which reflects realistic deployment conditions, not a controlled model-to-model comparison.
It is against Ivo AI standard procedure to agree to indemnify or defend at the NDA phase. We do not believe it is the right time for either party to take on indemnification costs before any real value exchange. Without this indemnity provision, the wronged party still has a remedy as it can sue for damages and collect after proving its case.