{
  "id": 14322616,
  "title": "Test document extraction before an agent acts on it",
  "url": "https://urgent.news/2026/10/13/test-document-extraction-before-an-agent-acts-on-it",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-13T23:45:54.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/conorbronsdon/test-document-extraction-before-an-agent-acts-on-it-ab0"
  },
  "original_language": "en",
  "account": "An agent can follow its instructions perfectly, yet still provide incorrect results due to a parser mishandling a table in an invoice. The agent does not view the original PDF; it only receives the extracted data. This initial error propagates through the entire workflow, making all downstream results unreliable. Jerry Liu, co-founder and CEO of LlamaIndex, highlighted this in a June 2026 episode of his podcast. If the parser fails to accurately represent a table or generates significant hallucinations, the workflow accuracy drops below acceptable levels, causing disruption in the agent's subsequent tasks. Liu advocates for a high accuracy threshold, aiming for 95% or more, particularly in sensitive use cases where human review is required on some extracted data. He emphasizes the importance of setting individual accuracy bars for each field based on the potential cost of a wrong value. To achieve this, Liu suggests treating extraction as a pipeline stage with its own testing, measured specifically on the fields that the agent consumes. He recommends labeling a small set of real documents, including both well-formatted and messy layouts, and keeping some back to evaluate prompts or rules without interference. Fields should be labeled individually rather than as whole documents. When a field is missing from the document, it should be assigned a value of None, while any failure to return a field should be considered an error. Liu advises comparing values in a canonical form by normalizing dates and currency, although he warns that stripping punctuation from an ID can inadvertently merge different IDs into a single string. He presents a hypothetical function, `extract_fields(doc)`, which returns a dictionary. Liu then provides a Python function, `evaluate(cases, extract_fields)`, which calculates scores for each case based on the extracted fields compared to the expected values. The scores report the number of correct predictions, the total number of tests, and any fields that missed the expected values. Liu emphasizes the importance of reporting accuracy per document type and field, rather than relying on an overall score, as a perfect score on a few invoices does not guarantee reliability. He stresses the significance of keeping track of the missed fields to pinpoint and fix the issues. Liu suggests using the evaluation results to establish a review gate. For documents without evaluation coverage, they should be held for human review. If the measured accuracy of a field falls below its defined bar, the field should be reviewed for that particular document type. If a required field is missing or its format is incorrect, the actions dependent on that field should be blocked. Additionally, if values violate any business rules, such as line items not summing to the total, they should be flagged for review. In cases where the extracted values break business rules, reviewers should see the values alongside the source passage or image region. The corrections should be stored and fed back into the labeled dataset, ensuring a separate held-out set remains untouched for future evaluations. Liu also addresses the issue of storing extracted facts in memory, suggesting a simple approach of using files as persistent context. He advises keeping candidates and accepted facts in separate places, with each record including its source reference, review status, and the extractor version that produced it. When a reviewer corrects a value, the old one should be marked as superseded instead of overwritten, allowing for traceability of the data's origin. In summary, before an agent acts on the extracted data, it is crucial to label real documents per type, define comparison rules for each field, report per-field accuracy with sample counts and retain the missed fields. Uncovered document types and weak fields should be routed to human review. Extracted facts must be stored with their source, status, and extractor version to maintain transparency and facilitate future corrections.",
  "summary": "An agent can follow its instructions perfectly and still pay the wrong amount, because the parser misread a table in the invoice. The agent never sees the PDF, only the extraction. If that step is wrong, everything downstream is confidently wrong too. Jerry Liu, co-founder and CEO of LlamaIndex, made this point in a June 2026 episode. A parser that represents a table wrong, or hallucinates at a…",
  "key_points": [
    "Agent relies on extracted data, not original PDF",
    "Parser errors can cause workflow failure",
    "Accuracy threshold advised at 95%"
  ],
  "editors_take": "Treating document extraction as a discrete, testable pipeline stage prevents parsing errors from corrupting downstream agent workflows and ensures sensitive data meets necessary accuracy thresholds through targeted human review.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}