{
  "id": 6114074,
  "title": "From Detection to Production: A PII-Safe Pipeline in Python and DuckDB",
  "url": "https://urgent.news/2026/09/07/from-detection-to-production-a-pii-safe-pipeline-in-python-and-duckdb",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-07T07:21:54.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/nbaubek/from-detection-to-production-a-pii-safe-pipeline-in-python-and-duckdb-1638"
  },
  "original_language": "en",
  "account": "This article describes a process for handling Personally Identifiable Information (PII) in a Python and DuckDB pipeline. The pipeline is divided into three zones: raw, curated, and vault. Raw zone stores unmasked data, while curated zone contains pseudonymized and masked data, and vault holds tokens for authorized reversal. The pipeline reads a classifications.yaml file to apply policies and transforms data based on its tier. Protection is implemented at the boundary out of raw zone, rather than eventually in a dashboard. The serving layer uses DuckDB roles to simulate masking policies, and erasure requests are processed by deleting data from all zones.",
  "summary": "Part 1 gave you a map: every PII-bearing column, tier-tagged. Part 2 armed you: HMAC for direct identifiers, tokens for what must be reversible, masking for display, generalization for quasi-identifiers. Now the part where articles usually wave their hands and your actual weekend disappears: putting it together. Everything in this post runs from the companion repo with uv sync — no warehouse, no…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}