Why Access Control Cannot Enforce Purpose Limitation in Data Pipelines

In May 2022, the Federal Trade Commission (FTC) and the U.S. Department of Justice fined Twitter $150 million. Twitter had collected personal data from over 140 million users for account security purposes, but they did not restrict which systems could access those fields. The collected data included phone numbers and email addresses, which advertisers later used to target ads.

While users were informed of the collection purpose, the system did not limit which components could read the data. In most organizations, policy texts and consent databases store the permissible uses of each record. Access control systems enforce access at the table, column, or row level, evaluating the principal and resource but not the purpose of the job or the allowed purposes of each record.

After the read operation, the record undergoes various transformations and training sets without carrying the collection purpose as metadata. Downstream jobs lack information about the data's origin. To enforce purpose limitation, each record or field should carry its allowed purposes. Data joins, transformations, and copies should preserve these labels.

Each pipeline and training job should declare its purpose. At the point of use, the system should compare the job's purpose with the data labels, removing or blocking records that are not allowed. Pre-deployment testing is crucial, as it can identify violations and unintended drops of allowed data. When a user withdraws consent for a purpose, systems must receive the change promptly, and it should be possible to trace which training sets and models used a given dataset.

The implementation of this control should be automated, not reliant on manual processes or documents.