How to Evaluate Contract AI Against Changing Terms and Incomplete Evidence
A 12-case synthetic contract test pack for checking amendment dates, service scope, signature evidence and citation applicability in AI answers.
When evaluating contract AI for changing terms and incomplete evidence, it is crucial to connect clauses to the right service, date, and document status. To test a contract AI system, teams can use 12 synthetic contract families with two questions each. The examples are concise enough to be inspected manually, aiming to help teams assess and test the contract AI with clear, traceable evidence.
In this walkthrough, the AI is presented with a fictional document family involving a 60-day notice period for an Analytics renewal. If the renewal date is 1 August 2026, the AI should return 90 days, as the amendment signed on 1 March changes the notice period to 90 days for renewals on or after 1 April. However, if the renewal date is 15 March, the answer should be 60 days, as the amendment's effective date has not yet arrived.
The proposal dated 1 May, suggesting a 30-day notice period, does not take effect without both parties' signatures.
To verify the correctness of the AI's answer, teams should check three things separately: (1) the existence of the cited passage, (2) whether the passage supports the statement being made, and (3) the answer's accounting for the evidence determining whether the term applies. The answer may properly cite an older clause to explain why it no longer governs the question.
Written by urgent.news from HackerNoon's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
