When Skipping an API Error Can Delete Valid Data: A Cartography Fix

When correcting a BigQuery synchronization issue in the Cartography project, I discovered that allowing skipped API errors could inadvertently delete valid data. Cartography, a CNCF sandbox project written in Python, gathers infrastructure assets and their relationships and stores them in a Neo4j graph. The project's BigQuery ingestion process lists resources, fetches additional details, loads the results, and performs cleanup.

One problematic scenario was an external table with invalid Delta Lake files causing BigQuery to return HTTP 400 with the reason invalidQuery. When this error was caught by the original code, a sync failure occurred. I fixed this by changing the error tolerance policy for BigQuery handlers. My first patch allowed more API errors to be skipped, but an automated reviewer flagged a risk: a failed list request could result in the cleanup deleting valid inventory nodes.

I traced this issue through the sync code and adjusted which errors the patch would tolerate. A single malformed table could stop the Cartography GCP sync. The fix involved identifying the cause and ensuring the appropriate handlers behaved correctly. Handlers for table details, datasets, and lists had specific behaviors, with table details returning None to preserve the listed table and datasets handling the error differently to maintain the listed dataset.

This approach avoided incomplete data appearing as resource removal, preserving the inventory's trustworthiness.