From Detection to Production: A PII-Safe Pipeline in Python and DuckDB
Part 1 gave you a map: every PII-bearing column, tier-tagged. Part 2 armed you: HMAC for direct identifiers, tokens for what must be reversible, masking for display, generalization for quasi-identifiers. Now the part where articles usually wave their hands and your actual weekend disappears: putting it together. Everything in this post runs from the companion repo with uv sync — no warehouse, no…
This article describes a process for handling Personally Identifiable Information (PII) in a Python and DuckDB pipeline. The pipeline is divided into three zones: raw, curated, and vault. Raw zone stores unmasked data, while curated zone contains pseudonymized and masked data, and vault holds tokens for authorized reversal. The pipeline reads a classifications.yaml file to apply policies and transforms data based on its tier.
Protection is implemented at the boundary out of raw zone, rather than eventually in a dashboard. The serving layer uses DuckDB roles to simulate masking policies, and erasure requests are processed by deleting data from all zones.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
