arrow_back Back to Blog

How is GPT-4 at Contract Analysis?

Noah Waisberg • March 16, 2023 • 11 minute read

Where Generative AI Is Good in Contract Data Extraction … And Where It Isn’t

TL;DR: GPT-4 is impressive overall, but-on contract review tasks-it’s inconsistent and makes mistakes; probably not yet ready as a standalone approach if predictable accuracy matters.

Back in 2011, when we started Kira Systems, other contract review software companies were using rules- or comparison-based approaches to finding data in contracts. We used supervised machine learning to find information in contracts, and did well against our competitors. Eventually, most of the world’s leading law and audit/consulting firms (and a bunch of corporates) became Kira customers. In recent months, Generative AI solutions have become all the rage. Are they going to supplant other machine learning approaches in contract analysis, just as machine learning approaches beat out rules?

We thought it would be useful to drill into this. Our thoughts on the subject come in three parts:

  1. How do Large Language Models perform on contract analysis tasks. Given GPT-4’s easy accessibility, we’ll discuss this with examples from it.
  2. What are advantages and disadvantages of LLMs versus other tech.
  3. Where we think LLMs can be especially helpful in contract review, and where contracts AI tech is headed.

Throughout, we use “Large Language Models,” “LLMs,” and “Generative AI” interchangeably. We recognize that this isn’t a great equivalence. Generative AI uses Large Language Models but is not the only form of Large Language Model. Nonetheless, we think it works well enough here that we do it.

This piece will cover how LLMs do at contract analysis. Specifically, we tested GPT-4 on a number of contract analysis challenges.

This is a long piece, so here’s a table of contents in case you would like to skip parts:

Why We’re (Not?) Worth Reading On This Topic

How GPT-4 Does At Identifying Data In Contracts

Before we get to our evaluation, let’s cover why you might find our perspective on this helpful.

Why We’re (Not?) Worth Reading On This Topic

On the “worth reading” side, we have a lot of experience in contracts AI, and ⅔ of us have computer science PhDs.

  • Noah was a corporate lawyer, then co-founded leading machine learning contract analysis company Kira Systems in 2011, and was its CEO until its sale to Litera in 2021. He’s now Zuva’s CEO. He has been involved in contract analysis AI longer than most.
  • Adam joined Kira in 2016 as a Research Scientist, having finished his PhD in computer science at the University of Waterloo (where he worked with Professor Gordon Cormack, a longtime leader in eDiscovery). At Zuva, he leads our Research and Product Development teams.
  • Sam joined Kira in 2018, after finishing his computer science postdoctoral work. He is currently a Senior Research Scientist at Zuva. His work has included a focus on differential privacy.

On the “not worth reading” side, perhaps:

  • Our experience prevents us from understanding a completely different present and future. We would like to think we’re open minded, but this is possible.
  • We are biased because we have a horse in the race: Zuva sells contracts AI software, and maybe we are trying to defend how we do things, as opposed to looking at the space clearly.

At Zuva, our mission is to make it dead easy to use the world’s best contracts AI. Frankly, we don’t really care about the technical approach we take to get there. Supervised ML, LLMs, even rules if appropriate - whatever. We just would like to (1) build great contracts AI and (2) make it dead easy to use. We think doing those things have the potential to get us to a good outcome. Also, even if we are biased, bias can sometimes drive crisper thinking on an issue. This is basically how the adversarial legal system works.

How GPT-4 Does At Identifying Data In Contracts

Adam, Noah, and Dr. Alexander Hudek (Kira and Zuva’s co-founder) ran a number of contract review tests on ChatGPT’s performance on contract analysis tasks beginning with ChatGPTs release, but we tarried in writing up our findings. With GPT-4’s release, we thought the time was right to share what we have learned.

When we first started playing with ChatGPT, we were pretty wowed.

GPT-3.5 was very impressive, and GPT-4 (on limited testing) seems even better. We think these can be really helpful for the right use cases. However, we now have a more nuanced view of how GPT-4 performs on contract review tasks.

welcome, GPT-4! how accurate are you at identifying contract clauses? As an Al language model, I can help identify and analyze contract clauses to some extent, but my accuracy depends on various factors such as the clarity and complexity of the contract language, the context in which the clauses appear, and the specific legal jurisdiction. Please note that I am not a licensed attorney, and my assistance should not be considered legal advice. It is always recommended to consult with a qualified legal professional for any contract-related matters. If you have a specific contract clause you’d like me to analyze, please provide the text, and I’II do my best to help you understand it.

Due to (1) the amount of attention on GPT-4 and (2) how easy it is to try ChatGPT, we are going to discuss LLM performance with examples from GPT-4. We know there are other Large Language Models out there, including some that have been specifically trained on legal documents. It’s very likely they perform differently, but it’s hard to say whether that means they are better or worse.

Let’s get into some examples.

Header Detection?

One really tricky thing about doing accurate post-signature contract analysis is that wordings can be non-standard. Sometimes documents come in the form of poor quality scans. At other times, they are drafted in atypical ways.

To measure how GPT-4 performed on slightly altered wording, we first gave GPT-4 some contract clauses (copied from a contract filed on Edgar) and asked it to identify what clauses were there.

thank you can you please identity the clauses in this contract excerpt: 9.4Export Control. Distributor understands and acknowledges that VMware is subject to regulation by agencies of the United States, including, but not limited to, the U.S. Denartment of Commerce which prohibit export or diversion or certain broducts and technology to certain countries. Any and all obligations of VMware to provide any technical information or assistance shall be subiect in all respects to such laws and regulations as shall from time to time govern the license and delivery of technology and products abroad by persons subject to the jurisdiction of the United States, including without limitation the U.S. Export administration Act of 1979, as amended, any successor legislation, and the Export Administration Regulations issued by the US Department of Commerce, Bureau of Export Administration, Distributor agrees to cooperate with VMware including without limitation, providing required documentation, in order to obtain export icenses or exemptions theretrom. Distributor warrants that it will comply with the S Export Administration Regulations and other U.S. and foreign laws and regulations governing exports and imports in ettect from time to time 9.5 Foreign Corrupt Practices Act; Compliance with Law. In conformity with the United States Foreign Corrupt Practices Act, Distributor and its employees and agents shall not directly or indirectly make an otter, payment, promise to pay, or authorize payment, or offer a gift, promise to give, or authorize the giving of anything of value for the purpose of influencing an act or decision of an official of any government within the Territory or the United States Government (including a decision not to act) or inducing such a person to use his influence to affect any such governmental act or decision in order to assist VMware in obtaining, retaining or directing any such business. Distributor will at all times comply with all laws and regulations which apply from time to time to the promotion, sale and support of the Products in the Territory. 9.6 Language. This Agreement is in the English language only, which language shall be controlling in all respects, and all versions hereot in any other language shall not be binding on the parties hereto. All communications and notices to be madeorgiven pursuant to this Agreement shall be in the English language 9.7 Currency. All Prices and other amounts payable by Distributor hereunder shall be paid in United States Dollars. All references to “dollars” “U.S.” or “$”. shall mean United States collars 9.8 Notices. Any notice under this Agreement must be in writing and either delivered in person, sent by fax, or first class mall it within the United States or sent by air courier to the address set forth on Exhibit c, addressed to the contact for Business issues, with a copy of the notice to the contact for Contract/Legal issues, Notices will be considered to have been given at the time of actual delivery in person or by tax, five 5) business days after deposit in the mail or two misiness Cavs atter delivery to an air courer service 9.9 Relationship of the Parties. The relationship of VMware and Distributor established by this Agreement is that of independent contractors. Nothing contained herein shall constitute either party the agent of the other party, or otherwise grant either party the authority to bind the other party to any obligation, or constitute the parties as partners or joint venturers and neither party shall hold itself out as being an agent, having such authority, or being a partner or joint venturer of the other.

Here are the clauses identitied in the contract excerpt providese 1. 9.4 Export Control: This clause addresses the distributor’s understanding and acknowledgment that VMware is subject to U.S. export regulations. It outlines the distributor’s agreement to cooperate with VMware in obtaining export licenses or exemptions and to comply with relevant U.S. and foreign laws and regulations governing exoors and imports 2. 9.5 Foreign Corrupt Practices Act; Compliance with Law: This clause states that the distributor and its employees and agents must not engage in any corrupt practices as defined by the US. Foreian Corrupt Practices Act. It also recuires the distributor to comoly win al aop icaolelaws and regulatons concerning the promorion. sale. and support of the products in the territory. 3. 9.6 Lanquage: This clause establishes that the agreement is in Enalish and that the English lanquage version is controlling. A communications and notices related to the agreement must be in English 4. 9.7 Currency: This clause specifies that all prices and amounts payable by the distrouror muscoe paid in unmeo stales volars 5. 9.8 Notices: This clause outlines the requirements for delivering notices under the agreement. incuoing me acceo able memoos o deliverv ane he umerrames for wen a norce is consideres to nave been given h. as Relationshin of the Partios: This cause clarities that the relationshin between yMware and the distributor is that of independent contractors. Neither party has the author o one me oter and me parties are nor consicered parers or oint venturers Please note that this analysis is for informational purposes only and should not be considered legal advice. Always consult a qualified legal professional for contract-related matters.

While—for this use-case—I could have done without the summary, GPT-4 accurately identified the clauses. And the summary was well done (and much more impressive than the GPT-3.5 one).

But that’s a pretty easy test. To spice things up, we ran some new contract text and changed the headers to incorrect contract terms. Here is the original contract segment:

  • 7.1 Indemnification. VMware shall, at its expense, defend Distributor against and pay all costs and damages including reasonable attorneys’ fees made in settlement or finally awarded against Distributor resulting from any claim, action or allegation brought against Distributor, that a Software Product infringes any copyright or trademark of a third party in the United States, Japan or European Community (“Infringement Claim”); provided that, as conditions of VMware’s obligation so to defend and pay, Distributor: (a) promptly notifies VMware in writing of any such Infringement Claim; (b) gives VMware sole control of the defense of any such Infringement Claim and any related negotiations or settlement; and (c) gives VMware the information and assistance necessary to settle or defend such Infringement Claim. If it is adjudicatively determined, or if VMware reasonably believes that any Software Product infringes any third party copyright or trademark, then VMware may, at its option and expense: (i) modify the Software Product or infringing part thereof to be reasonably equivalent and non-infringing; (ii) procure for Distributor a license to continue distributing the Software Product or infringing part thereof; (iii) replace the Software Product or infringing part thereof with other comparable products; or (iv) terminate Distributor’s rights hereunder with respect to the infringing Software Product.
  1. LIMITATION OF LIABILITY.VMWARE’S LIABILITY UNDER THIS AGREEMENT, REGARDLESS OF THE FORM OF ACTION, WILL NOT EXCEED ONE HUNDRED PERCENT (100%) OF THE AMOUNTS PAID UNDER THIS AGREEMENT BY DISTRIBUTOR TO VMWARE DURING THE PREVIOUS TWELVE (12) MONTH PERIOD FOR THE PRODUCTS GIVING RISE TO THE LIABILITY. NEITHER PARTY WILL BE LIABLE FOR ANY SPECIAL, INDIRECT, CONSEQUENTIAL OR INCIDENTAL DAMAGES ARISING OUT OF THIS AGREEMENT, WHETHER OR NOT SUCH PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES, AND NOTWITHSTANDING ANY FAILURE OF ESSENTIAL PURPOSE OF ANY LIMITED REMEDY.
  • 9.1 Governing Law. The rights and obligations of the parties under this Agreement shall not be governed by the 1980 U N Convention on Contracts for the International Sale of Goods. This Agreement will be governed by the laws of the State of California and the United States of America, without regard to conflict of law principles. The parties hereby consent to the non-exclusive jurisdiction of the state and federal courts located in Santa Clara County, California, for resolution of any disputes arising out of this Agreement. Either party may seek injunctions to prevent and/or stop any breach of, and otherwise enforce, the provisions of Section 6, and VMware may seek injunctions to prevent and/or stop any infringement of, and otherwise enforce, its intellectual property rights of whatever nature, in the courts of any country, state or other territory which accepts jurisdiction.
  • 9.2 Assignment.This Agreement and any rights or obligations of Distributor hereunder may not be assigned, sub-contracted or otherwise transferred by Distributor without VMware’s prior written consent. Subject to the preceding sentence, this Agreement shall be binding upon and inure to the benefit of the parties’ permitted successors and assigns.
  • 9.3 Indemnification by Distributor. If VMware should incur any liability to a third party caused by the non-performance of Distributor of any of its obligations under this Agreement, or resulting from any act or omission of Distributor, or if VMware incurs any liability to a third party by reason of acts of Distributor in marketing or distributing the Products, Distributor agrees to indemnify and hold VMware free and harmless from any such liability, and from all loss, claims, costs, demands, debts, and causes of action in connection therewith, including reasonable attorney’s fees.

Here’s what we pasted into GPT-4, and what we got in return:

thank you. can you please identity the clauses in the following contract excerpt. no need to describe what the clauses are about - simply identitying them is all need: 7.1 Exclusivity. VMware shall, at its expense, defend Distributor against and pay all costs and damages including reasonable attorneys’ fees made in settlement or finally awarded against Distributor resulting from any claim, action or allegation brought against Distributor, that a Software Product infringes any copyright or trademark of a third party in the United States, Japan or European Community (“Infringement Claim”); provided that, as conditions of VMware’s obligation so to defend and pay, Distributor: (a) promptly notifies VMware in writing of any such Infringement Claim; (b) gives VMware sole control of the defense of any such Infringement Claim and any related negotiations or settlement; and c) gives VMware the information and assistance necessary to settle or defend such Infringement Claim. If it is adjudicatively determined, or if VMware reasonably believes that any Software Product infringes any third party copyright or trademark, then VMware may, at its option and expense: (i) modify the Software Product or infringing part thereof to be reasonably equivalent and non-infringing; (ii) procure for Distributor a license to continue distributing the Software Product or infringing part thereof; (ili) replace the Software Product or infringing part thereof with other comparable products; or (iv) terminate Distributor’s rights hereunder with respect to the intringing Sottware Product. 8. CONFIDENTIALITY. VMWARE’S LIABILITY UNDER THIS AGREEMENT, REGARDLESS OF THE FORM OF ACTION, WILL NOT EXCEED ONE HUNDRED PERCENT (100%) OF THE AMOUNTS PAID UNDER THIS AGREEMENT BY DISTRIBUTOR TO VMWARE DURING THE PREVIOUS TWELVE (12) MONTH PERIOD FOR THE PRODUCTS GIVING RISE TO THE LIABILITY. NEITHER PARTY WILL BE LIABLE FOR ANY SPECIAL, INDIRECT. CONSEQUENTIAL OR INCIDENTAL DAMAGES ARISING OUT OF THIS AGREEMENT, WHETHER OR NOT SUCH PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES, AND NOTWITHSTANDING ANY FAILURE OF ESSENTIAL PURPOSE OF ANY LIMITED REMEDY. 9.1 Compliance With Law. The rights and obligations of the parties under this Agreement shall not be governed by the 1980 U N Convention on Contracts for the International Sale of Goods. This Agreement will be governed by the laws of the state of California and the United States of America, without regard to conflict of law principles. The parties hereby consent to the non-exclusive minsdiction of the state and tocera courts located in Santa Clara County, California, for resolution of any disputes arising out of this Agreement. Either party may seek injunctions to prevent and/or stop any breach of, and otherwise enforce, the provisions of Section 6, and VMware may seek injunctions to prevent and/or stop any infringement of, and otherwise enforce, its intellectual property rights of whatever nature, in the courts of any country, state or other territory which accepts jurisdiction. 9.2 Non-Disparagement. This Agreement and any rights or obligations of Distributor hereunder may not be assigned, sub-contracted or otherwise transferred by Distributor without VMware’s prior written consent. Subject to the preceding sentence, this Agreement shall be binding upon and inure to the benefit of the parties permitted successors and assigns 9.3 Non-Solicitation. If VMware should incur any liability to a third party caused by the non-performance of Distributor of any of its obligations under this Agreement, or resulting from any act or omission of Distributor, or if VMware incurs any liability to a third party by reason of acts of Distributor in marketing or distributing the Products, Distributor agrees to indemnify and hold VMware free and harmless from any such liability, and from all loss, claims, costs, demands, debts, and causes of action in connection therewith, including reasonable attorneys fees Here are the identitied clauses in the provided contract excerpt: 1. 7.1 Exclusivity 2. 8. Confidentiality c. a compliance with rav 4. 9.2 Non-Disparagement 5. 9.3 Non-Solicitation

From the look of it, it seems like GPT-4 identified contract clauses with a header-detection approach. This might work sometimes, but didn’t here. We would not recommend it if you need accurate contract clause identification. Headers only sometimes describe clause contents.

While we thought the previous prompt was pretty reasonable, we decided to try one additional prompt:

Thank you can you identity any indemnity clauses in the text I provided? Yes, there are two indemnity clauses in the provided text: 1. In the first excerpt, the indemnity clause is found within section 7.1 Exclusivity, where VMware agrees to defend and indemnity Distributor against certain copyright or tracemark intrngement c aims 2. In the third excerpt, the indemnity clause is found within section 9.3 Non-Solicitation, where Distributor agrees to indemnify and hold VMware harmless from any liability arising from Distributor’s non-performance, acts, or omissions related to marketing or distributing the Products.

GPT-4 correctly identified both (pretty-standard, apart from the changed headers) indemnity clauses. This is an improvement from GPT-3.5, which found one and missed one.

are there any indemnity clauses in the text I provided? Yes, there is an indemnity clause in the contract text you provided. Section 9.3 states that it Distributor tails to perform its obligations under the agreement, or it its acts or omissions result in VMware incurring liability to a third party, then Distributor agrees to indemnity and hold VMware harmless from any such liability and related costs, including reasonable attorneys’ fees.

Overall, this is inconsistent performance. These tests weren’t particularly hard. GPT-4 was right sometimes, wrong other times. On the one hand, it’s very cool that GPT-4 was this good out of the box. On the other hand, I might have reservations about using it if I was in a situation where I needed more consistent and predictable accuracy.

Differently-Phrased Clauses

Contract clauses can be worded a lot of different ways. Poor quality scans, and contracts written in different jurisdictions (sometimes by non-English-first-language, non-lawyer authors) can contribute to even more disparate wordings.

Change of control clauses are among the most important to identify in connection with M&A contract review (aka, due diligence). At the time we sold Kira, 18 of the top 25 global M&A law firms were Kira customers, so we have a fair amount of experience with this use case. 30–60% of a typical M&A legal bill goes to due diligence. Not all of this is contract review, and not all the contract review segment of due diligence is about finding change of control clauses, but (accurately!) finding change of control clauses tends to be a big part of this work. Companies who are buying other companies literally spend millions of dollars with Biglaw firms on a regular basis to get this done right. Not only are change of control clauses important to get right, but there are a lot of ways to word them. As with other areas of contract review, this problem can be exacerbated by atypical drafting and poor quality scans.

We decided to test how GPT-4 would do if we ran some typical change of control clauses through. We took it relatively easy here, only giving pretty standard change of control language and not, say, introducing the distortion of a poor quality scan or non-English-first-language drafted clauses.

thank you can you identify any change of control clauses in the following text: 2 Term. This Agreement will commence on the Effective Date and will continue for a term of one (1) year, unless sooner terminated in accordance with this Section 5. This Agreement will automatically renew for successive one (1) year terms unless terminated by either party upon no less than thirty (30) days written notice prior to the termination of the initial or any renewal term. 3 Termination. Either party may terminate this Agreement at any time, for any reason or no reason, upon twelve (12) months prior written notice to the other party. This Agreement may be terminated for material breach of any provision of this Agreement by either party, provided that written notice of the breach has been given to the breaching party and the breaching party has not cured the breach within thirty (30) days after delivery of the notice. This Agreement may be terminated for any direct or indirect assignment. The provisions of Sections 1.4, 27, 2.10, 12, 4.2, 5.2, 5.3, 6, 7, 8 and 9 will survive expiration or termination of this Agreement. Expiration or termination shall be without prejudice to the accrued rights and liabilities if the parties. 5 Assignment. This Agreement and any rights or obligations of Distributor hereunder may not be assigned, sub-contracted or otherwise transferred by Distributor without VMware’s prior written consent. Subject to the preceding sentence, this Agreement shall be binding upon and inure to the benefit of the parties’ permitted successors and assigns 6. Consolidation or Merger. Notwithstanding any other provision herein to the contrary, in case of any consolidation or merger to which the Guarantor is a party other than a merger or consolidation in which the Guarantor is the continuing corporation. or in case of an sale or conveyance to another corporation of the property of the Guarantor as an entirety or substantially as an entirety, or in the case of any statutory exchange of securities with another corporation (including any exchange effected in connection with a merger of a third corporation into the Guarantor), there shall be no adjustments under Section 10.05 but the Holder of each Security then outstanding shall have the right thereafter to convert such Security into the kind and amount of securities, cash or other property which he would have owned or have been entitled to receive immediately after such consolidation, merger, statutory exchange, sale or conveyance had such Security been converted immediately prior to the effective date of such consolidation, merger, statutory exchange, sale or conveyance and in any such case, if necessary, appropriate adjustment shall be made in the application of the provisions set forth in this Article 10 with respect to the rights and interests thereafter of the Holders of the Securities, to the end that the provisions set forth in this Article 10 shall thereafter correspondingly be made applicable, as nearly as may reasonably be, in relation to any shares of stock or other securities or property thereafter deliverable on the conversion of the Securities. 7 Assignment. Any assignment, whether direct or indirect, requires consent. 8 Limitation of Liability upon Termination. WITHOUT PREJUDICE TO SECTION 8, NEITHER VMWARE NOR DISTRIBUTOR WILL BE LIABLE TO THE OTHER FOR DAMAGE OF ANY KIND, INCLUDING LOSS OF PROFIT, LOSS OF CUSTOMERS, LOSS OF BUSINESS OPPORTUNITY AND INDIRECT, INCIDENTAL OR CONSEQUENTIAL DAMAGES, ON ACCOUNT OF THE TERMINATION OR EXPIRATION OF THIS AGREEMENT IN ACCORDANCE WITH THIS SECTION 5 DISTRIBUTOR WAIVES ANY RIGHT IT MAY HAVE TO RECEIVE ANY COMPENSATION OR REPARATIONS ON TERMINATION OR EXPIRATION OF THIS AGREEMENT UNDER THE LAW OF THE TERRITORY OR OTHERWISE, OTHER THAN AS EXPRESSLY PROVIDED IN THIS AGREEMENT. 9 1940 Act Assignment. Any assignment, as such term is defined in the Investment Advisers Act of 1940, is void without consent. In the provided text, there isn’t a clause explicitly labeled as a “change of control” clause. However, section 6, “Consolidation or Merger.” addresses some aspects of a change of contro situation. such as consolidation. merger. sale, or convevance of the company’s property. This clause outlines the rights of the security holders in case of such events and the necessary adjustments to be made.

All of 3, 6, 7, and 9 are change of control clauses. GPT-4 showed real room for improvement, though at least it got 6.

One objection to this test is that GPT-4 (or another Large Language Model) could be trained to find variations of change of control clauses, just as other contracts AI was heavily trained to find clauses. Zuva has a not-small machine learning research team, and we are always interested in how to get our tech to perform better. When we have tested using LLMs to find information in contracts, we have found we can get comparable accuracies. The catch is that they are orders of magnitude more expensive to train and use. For example, in one of our papers we compared a (non-generative) LLM to our current tech for finding named entities. The LLM was 4% more accurate, which is not nothing, but cost 10,000 times more than our baseline to achieve that¹. We’ll delve deeper into costs in part 2.

Sensitivity To Prompts

In the course of testing GPT-4 on summarizing text and question answering (which we’ll discuss in more detail in part 3), we noticed that it was sensitive to prompts. This is fairly unsurprising, given what we’ve seen with other LLMs. Be mindful of this if using an LLM in your work.

thank you Answer yes or no only to the following, is consent required to assign the contract based on the following clause: The BL shall not, without the prior written consent of the Supplier (not to be unreasonably withheld and whether by operation of law or otherwise, assign, transfer, charge, subcontract or deal in any other manner with all or any of its rights or obligations under this Agreement, including all Order Forms.20.2 The Supplier shall not, without the prior written consent of the BL (not to be unreasonably withheld and whether by operation of law or otherwise, assign, transfer, charge, sub-contract or deal in any other manner with all or any of its rights or obligations under this Agreement, including all Order Forms. 20.3 Notwithstanding the foregoing in 20.1 and 20.2, either party may assign this Agreement in its entirety (including all Order Forms), without consent of the other party, to its Affiliate or in connection with a merger, acquisition, corporate reorganization, or sale of all or substantially all of its assets not involving a direct competitor of the other party. No thank you. based on the following contract clause, is consent required to assign the contract. please answer yes or no only The BL shall not, without the prior written consent of the Supplier (not to be unreasonably withheld and whether by operation of law or otherwise, assign, transfer, charge, subcontract or deal in any other manner with all or any of its rights or obligations under this Agreement, including all Order Forms.20.2 The Supplier shall not, without the prior written consent of the BL (not to be unreasonably withheld and whether by operation of law or otherwise, assign, transfer, charge, sub-contract or deal in any other manner with all or any of its rights or obligations under this Agreement, including all Order Forms.20.3 Notwithstanding the foregoing in 20.1 and 20.2, either party may assign this Agreement in its entirety (including all Order Forms), without consent of the other party, to its Affiliate or in connection with a merger, acquisition, corporate reorganization, or sale of all or substantially all of its assets not involving a direct competitor of the other party. Yes

Summary

GPT-4 has generally really impressed us. It appears able to do a lot very well. That said, we are unconvinced that it yet offers predictable accuracy on contract analysis tasks. (And we have not yet really pushed it, for example on poor quality scans and less standard wordings, though perhaps it would do well on these.) If you use Generative AI to help you find contracts with a change of control clause, it will identify change of control clauses in contracts. If, however, you need it to find all (or nearly all) change of control clauses over a group of contracts, we wouldn’t yet count on it². Still, this technology is improving, and our view may change as we test further iterations. Also, as we’ll discuss in a further installment, we think LLMs offer significant benefits today when combined with other machine learning contract analysis technologies. Exciting!

¹ In our experience, Zuva’s ML is far more effective on clause detection than finding contextually-relevant entities. Accordingly, we didn’t expect to “win” this comparison. But we think it’s important to test and see where we can improve.

² In situations where accurate contract review really matters, we recommend using contract analysis AI in conjunction with a user interface that includes a document viewer. A document viewer increases the odds that a human working with the AI might catch mistakes the AI makes.