I built an epistemic gate to stop LLM data poisoning during fine-tuning. Tested across 5 architectures, orchestrated on a 2006 Toshiba laptop for $0.

In recent years, the AI industry has promoted the notion that advanced safety measures, data poisoning detection, and model alignment necessitate large-scale computing resources and substantial financial investment. This report outlines a study that challenges this belief, demonstrating that these critical components can be achieved with a modest budget and minimal hardware requirements.

The researchers conducted a 16-experiment series, ranging from EXP08 to EXP16, on a 2006 Toshiba Satellite laptop equipped with 2GB RAM and free Kaggle T4 GPUs. The primary objective was to investigate the phenomenon of epistemic collapse and data poisoning in large language models (LLMs).

One of the key challenges addressed by this research is malicious data poisoning during fine-tuning. Traditional metrics, such as standard perplexity, often fail to detect such covert attacks. Attackers can effectively corrupt specific facts, even to a very precise threshold, while simultaneously maintaining the appearance of improved global metrics due to the introduction of fluent prose during fine-tuning.

To counter this threat, the researchers introduced Beatriz, an epistemic gate that functions as a lightweight defensive proxy with a negligible decision latency (approximately 0.1ms). Positioned between the generative source and the training model, Beatriz acts as a screening mechanism, verifying the generated text against an immutable anchor corpus using cosine similarity in embedding spaces.

This verification process is executed through a composite loss function, consisting of two components: LceLce, which ensures linguistic fluency, and LcontrastivaLcontrastiva (Softplus), which maintains a constant pressure on the model to anchor to truth even after the cross-entropy term reaches its saturation point.

Throughout the experiments, Beatriz was tested across five different LLM architectures, including GPT-2 (124M), Qwen-2.5-0.5B, TinyLlama-1.1B, Pythia-1.4B, and Phi-3-mini-4k-instruct (3.8B). An additional ablation study demonstrated that the pure filtering approach (GATE_ONLY) provided 65% of the defensive benefit, with the contrastive term contributing the remaining 35%.

A more extensive evaluation was conducted in EXP16, where the model was evaluated on a rigorous 30-domain unattacked facts benchmark. The results showcased a robust truth margin of +4.19 ± 0.08, accompanied by a precision of 0.93 and recall of 0.80.

The findings presented in this report indicate that Beatriz offers an effective, low-cost solution to mitigate the risk of data poisoning during LLM fine-tuning. By providing a lightweight and non-invasive defensive proxy, Beatriz empowers small teams and startups to safeguard their local models against malicious attacks, thereby enhancing the overall security and reliability of AI-powered systems.