A 125M model beat a 14B LLM at de-identifying medical text 40 faster, on CPU

localscrub, a local-first PHI de-identification cascade, achieved a significant improvement in speed and accuracy when compared to a standard baseline. This local solution runs entirely on the user's CPU, eliminating the need to send sensitive data to cloud APIs, thus maintaining privacy and control over the data. The project developed a two-stage cascade: the first stage employs rule-based and Named Entity Recognition (NER) methods to identify well-formatted identifiers such as phone numbers, emails, dates, and account numbers, while the second stage utilizes a locally served LLM to handle the more ambiguous portions of the text.

This approach results in a 125M parameter model that outperformed a 14B parameter model by 40 times faster on a single consumer laptop. The developers created a synthetic clinical note dataset using the localscrub synth tool, allowing for accurate evaluation of the de-identification process without relying on real patient data. The evaluation harness calculates three key metrics: relaxed, strict, and redaction recall.

These metrics help quantify the effectiveness of the de-identification process and ensure that the output is genuinely safe. The authors emphasize the importance of verifying the accuracy of privacy tools, as using unverifiable claims could lead to potential liabilities. The localscrub project demonstrates a comprehensive and rigorous approach to de-identifying medical text, setting a new benchmark for local solutions and paving the way for further improvements in this critical area.