{
  "id": 2951126,
  "title": "What Broke When We Turned Government Open Data Into Agent-Callable Evidence",
  "url": "https://urgent.news/2026/08/24/what-broke-when-we-turned-government-open-data-into-agent-callable",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-24T05:25:11.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/civicdataforge/what-broke-when-we-turned-government-open-data-into-agent-callable-evidence-1m81"
  },
  "original_language": "en",
  "account": "Government open data may appear simple at first glance. To successfully integrate it into a production system, one must consider the complexities that arise once it reaches a software system or AI agent. The post outlines the critical factors that must be in place to ensure the data qualifies as evidence.\n\nOne important consideration is that a successful HTTP response does not equate to a complete dataset. Government data platforms often impose limits on the number of records returned per request. This can lead to discrepancies between the number of records reported and the actual number available. The collection rule outlined in the article is to discover the publisher's page limit, request all pages in a stable order, track the page count, observed row count, and source-reported totals, and reject partial or inconsistent results.\n\nAnother key point is that the absence of a match does not automatically imply a negative legal conclusion. When an address lookup returns no permit record, it is better to return a fail-closed decision vocabulary, such as EVIDENCE_FOUND, NO_PUBLISHED_MATCH, REVIEW_REQUIRED, or SOURCE_UNAVAILABLE. This approach avoids manufacturing certainty and allows downstream workflows to map evidence states to appropriate actions while preserving the original state for review.\n\nNormalization is necessary when working with government data, but it can also lead to the loss of the record's identity. The article recommends retaining enough lineage during normalization to answer questions about the authority that published the record, the endpoint or dataset it came from, the query and scope used to retrieve it, the retrieval time, and the transformation applied to the original data. A minimal evidence envelope should include the source information, scope, record details, and decision state, with a hash to ensure the integrity of the record.\n\nSource health and data freshness are distinct dimensions. A source can be reachable but provide outdated data. It may have a stable schema but still return incomplete data. The article suggests monitoring multiple dimensions of health, including availability, schema compatibility, record-count behavior, source date, and freshness. If freshness cannot be measured, the status should be labeled as \"freshness_not_measured\" to prevent a green check from implying that the data is current.\n\nChange monitoring for AI agents requires a more robust approach than simple alerts. It is essential to understand whether the previous snapshot was complete, if the new snapshot is complete, whether the change originated from the source or the collector, and how the event can be delivered and replayed. Idempotency keys should be used to ensure that a webhook is treated as a notification about an evidence object rather than the object itself. The durable dataset, record hashes, scope, and decision receipt should remain the source of truth for inspection and analysis.",
  "summary": "Government open data looks deceptively easy from a distance. Find an endpoint. Make a request. Normalize the JSON. Ship an API. That is enough for a demo. It is not enough for a production system that has to answer a consequential question next month, after the publisher changes a field, caps a response, moves an endpoint, or returns an empty page that looks exactly like “there are no records.”…",
  "key_points": [
    "HTTP response limits can cause discrepancies between reported and actual records",
    "Fail-closed decision vocabulary recommended for address lookups with no permit records",
    "Evidence envelope should include source info, scope, record details, decision state"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}