When is an aggregate really anonymous? Differencing attacks on AI query layers
A sum is not anonymization. Whether a number over special-category personal data (Art. 9 GDPR) says something about one person does not depend on its shape but on the population behind it. If fewer than k people stand behind a number, the aggregate is a single-person disclosure in table form. A worked example with invented numbers An HR department analyses union membership. The minimum group size…
An aggregate is not automatically anonymized, even if it appears to obscure individual identities. The level of anonymity within a number depends on the population behind it. If fewer than k people are represented by a number, that aggregate constitutes a disclosure of information about a single individual. To illustrate, imagine an HR department analyzing union membership, with a minimum group size set at 5.
Queries A and B ask about union membership in purchasing. Query A returns 6 union members, while Query B returns 5. The difference between the two queries reveals that one person - the one who joined the purchasing department in 2026 - is a union member. Despite both queries meeting the k-anonymity threshold independently, the difference between their results discloses sensitive information.
Two consequences arise from this scenario. First, a rule attached to the shape of a query (requiring at least k rows) can be circumvented by filtering the population down to a single person. Second, k-anonymity is merely a minimum condition, not a comprehensive solution. At Nowl, a self-hosted semantic access layer for existing business software, the system automatically applies a group-size condition to grouped queries, counting the population before execution and rejecting the query if the count falls below k.
The rejection does not provide any information about the data, except whether the count is below k. While k is customizable per project, it cannot be set to 1. Moreover, the system logs every request, including rejections. However, this approach does not prevent differencing attacks across multiple queries, nor does it provide l-diversity guarantees.
Additionally, it does not address whether all sensitive fields have been properly classified. Currently, this system is in the pilot phase, tested on synthetic data, and has not been validated on a live customer dataset. To ensure the proper handling of sensitive data in AI systems, vendors should be asked about several key factors.
First, who determines which fields are special categories, and is this decision confirmed by a human? Second, does the query itself contain the sensitive statement? Third, is the threshold determined by the shape of the query or the population behind it? Fourth, who sets the k value, and can it be adjusted to disable the check? Fifth, does the system count individuals or joined rows?
Sixth, what happens when the k-anonymity check fails? Seventh, does a rejection leak any information about the data? And finally, is there any tracking of queries or a noted gap in this area?
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
