Mask, Hash, Tokenize, or Encrypt? Choosing the Right PII Protection for Your Pipeline

In Part 1, a data map was created: every column holding personal information (PII) was tagged with a tier. The next logical question arose: should we hash or mask the emails? Hashing is almost always the wrong approach because the question should be, what do we still need this data to do? This article explores that question deliberately. All code can be found in the companion repository, with every claim backed by a runnable file.

The key takeaway is that hashing or other techniques that can recover the original value still count as personal data. Encryption, tokenization, masking, and generalization fall on a spectrum of how much original value can be recovered, ranging from irreversible to potentially precomputable.

Masking is appropriate when nobody needs the original value; it's a display logic that makes the data display-safe but irreversible. For example, masking an email address (e.g., jane.doe@example.com → j***@example.com). This is great for support agents verifying identity by last-4 digits or for building test data, but it's not suitable for anything that needs to join, count distinct, or reach back out to the person.

Hashing, on the other hand, feels like privacy alchemy but is not suitable for PII due to its deterministic nature and precomputable input space. While hashing is irreversible and join-friendly, hashing small input spaces like phone numbers or SSNs leaves them vulnerable to precomputation attacks. Using HMAC-SHA256 with a secret key can provide a deterministic and irreversible output while still maintaining join-friendliness, but it introduces the responsibility of key management.

Tokenization is the solution when the business needs to reach back out to the person. It swaps the PII for a token and stores the mapping in a vault, keeping the original data inaccessible to the analytics pipeline. The pipeline only sees the token, and reversal is a vault lookup behind an authorization boundary. Tokenization can be deterministic or random, and it is the preferred method for PCI-DSS compliance for card data.

Encryption is essential for access control but not for anonymizing data. If a key can decrypt the column, it still counts as personal data. Guarding, auditing, and backing up the encryption keys like they contain everyone's names is crucial, as compromising them can lead to a full breach of personal data.