I benchmarked AI decision APIs against XGBoost for fraud detection. XGBoost won, by a lot.

Between September and October, three companies - TypeSafe, OpenAI, and Cloudflare - launched decision model APIs for fraud detection. The author of the story wanted to compare these AI options with the commonly used XGBoost model. To do this, they built an open-source benchmark that runs offline in about five seconds.

The models were evaluated using the same transactions and fields, with scores ranked by each model. The comparison included XGBoost, trained on labeled history, a hand-written rules checklist, TypeSafe Jev, OpenAI Decisions, Cloudflare Clef and Clef-flash (a general LLM forced to return a strict JSON score), and a general LLM (gpt-6-luna).

The metric used was the review queue, specifically how many fraud cases would be flagged by each model in the top 100 reviews, and the top 1% and 5%. The results showed that XGBoost caught the highest percentage of fraud in the first 100 reviews, while the best AI approach only caught 46.2%. However, the author notes that the AI approaches were run zero-shot (with no examples), which is one way to use LLMs in fraud work. Few-shot prompts, richer context or fine-tuning could potentially change the results.

The benchmark also looked at speed and cost, finding that the AI APIs were fast and cheap ($0.016-$0.10 per 1,000 synthetic transactions). However, the most crucial aspect is that XGBoost was trained on labeled data, while the AI approaches were zero-shot. The author suggests that this difference could impact the accuracy of the AI models in real-world production environments.