{
  "id": 12130923,
  "title": "When Skipping an API Error Can Delete Valid Data: A Cartography Fix",
  "url": "https://urgent.news/2026/10/05/when-skipping-an-api-error-can-delete-valid-data-a-cartography-fix",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-05T10:45:38.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/vanshsharma27/when-skipping-an-api-error-can-delete-valid-data-a-cartography-fix-1d6g"
  },
  "original_language": "en",
  "account": "When correcting a BigQuery synchronization issue in the Cartography project, I discovered that allowing skipped API errors could inadvertently delete valid data. Cartography, a CNCF sandbox project written in Python, gathers infrastructure assets and their relationships and stores them in a Neo4j graph. The project's BigQuery ingestion process lists resources, fetches additional details, loads the results, and performs cleanup. One problematic scenario was an external table with invalid Delta Lake files causing BigQuery to return HTTP 400 with the reason invalidQuery. When this error was caught by the original code, a sync failure occurred. I fixed this by changing the error tolerance policy for BigQuery handlers. My first patch allowed more API errors to be skipped, but an automated reviewer flagged a risk: a failed list request could result in the cleanup deleting valid inventory nodes. I traced this issue through the sync code and adjusted which errors the patch would tolerate. A single malformed table could stop the Cartography GCP sync. The fix involved identifying the cause and ensuring the appropriate handlers behaved correctly. Handlers for table details, datasets, and lists had specific behaviors, with table details returning None to preserve the listed table and datasets handling the error differently to maintain the listed dataset. This approach avoided incomplete data appearing as resource removal, preserving the inventory's trustworthiness.",
  "summary": "TL;DR: I fixed a BigQuery sync failure in Cartography, then narrowed the error tolerance after review showed that skipping failed enumeration could feed false deletions into graph cleanup. One malformed BigQuery table could stop a Cartography GCP sync. My first patch let more API errors be skipped, but an automated reviewer caught a risk I had missed: a failed list request could leave cleanup…",
  "key_points": [
    "Skipping API errors can delete valid Cartography data",
    "Fixed error tolerance policy to prevent data loss",
    "Adjusted handlers for table details, datasets, and lists"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}