Coinbase's Fraud Agent Never Decides Anything. That's Why It Works.

Coinbase's fraud detection system, the Fraud Agent, operates without making final decisions on transactions. Instead, it assigns a risk level, and application code ultimately decides whether to allow or block the transaction. This architecture assigns asymmetric authority to the Fraud Agent. It can block fraudulent transactions that passed all upstream checks, but it cannot reverse an allowed transaction.

In an A/B test, adding the Fraud Agent reduced fraudulent transactions by 30% and cut fraud value by 22%. However, the benchmark used for testing consisted of only 16,140 transactions (7,293 users, 813 confirmed fraud cases). Moreover, the performance of newer models, such as Opus, Sonnet, and GPT releases, actually declined in recall and F1 scores compared to their predecessors.

This suggests that the general capabilities of newer models do not necessarily translate into improved specialized judgment needed for fraud detection.

Coinbase addressed this issue by fine-tuning Qwen3.5-9B using reinforcement learning on its own proprietary fraud data. This process, SkyRL on Anyscale, significantly improved the model's performance compared to Opus 4.5. The fine-tuned model achieved higher precision, recall, and F1 scores, with margins ranging from 7.5 to 35.4 points.

Its median latency was also much lower, at 0.683 seconds compared to 1.515 seconds for Opus 4.5. The training cost for this fine-tuning was under $100, using just one NVIDIA RTX PRO 6000 96GB GPU.

The Fraud Agent's design allows it to safely advise human reviewers on transactions deemed risky but not automatically block them. For instance, a $4,800 purchase from a brand-new bank account after two failed authentication attempts received a 0.65 risk level, triggering a human review. Similarly, a routine $120 USDC transaction made by a long-term user at an expected time received the same advisory.

These examples highlight the Fraud Agent's inability to discriminate between patterned fraud attempts and regular, legitimate transactions, underscoring the importance of Coinbase's fine-tuned solution in maintaining effective fraud detection.