PO Matching Looked Like It Halved Unsafe Invoice Approvals Until I Checked Why
A LangGraph agent for Japanese invoices, killed with SIGKILL, pre-registered, and gated in CI. The headline gain from its last check turned out to be mostly routing, not detection. Scope note: everything here is measured on synthetic invoices , a synthetic registry and synthetic purchase orders (fictitious companies, generated from the true documents with a fixed scenario mix). One extractor (the…
A LangGraph agent for Japanese invoices, which was killed with SIGKILL, performed well in detecting unsafe invoice approvals. The headline gain from its last check showed a halved unsafe auto-approval rate, but further analysis revealed the difference was mostly due to routing, not detection. The agent underwent three versions: v1 trusted the extraction, v2 added verification, and v3 added duplicate checks and purchase order (PO) matching.
The primary metric was the unsafe auto-approval rate, which decreased from 17.0% to 4.0% after verification and further to 2.0% after duplicate checks and PO matching. Verification caught most of the errors, such as issue date, invoice number, and one item line. PO matching only stopped one wrong extraction out of six invoices with wrong PO agreement.
The analysis showed that PO matching works well when extractor's mistakes are in fields a PO can see, but adds little when mistakes are not. The study also highlighted the importance of state durability in LangGraph, as poor handling of interruptions led to lost checkpoints and incorrect feedback in LangSmith experiments.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
