Newer AI models missed more payment fraud in Coinbase’s benchmark

Updated large language models from three major AI families detected a smaller share of fraudulent payments than the earlier versions they replaced, according to a controlled replay of Coinbase Onramp activity reported by CryptoSlate. Coinbase reported the results on Oct. 7, 2026. The evaluation covered 16,140 transactions involving 7,293 users, including 813 confirmed fraudulent transactions, and each model was tested against the same transaction group, contextual-review process and risk-to-decision policy.

Across the comparisons, Opus 5, Sonnet 5 and GPT-5.6 (sol) posted lower recall, F1 and dollar-weighted recall than their predecessors. Recall measures the share of fraudulent transactions detected, while dollar-weighted recall measures the share of fraudulent transaction value identified. Opus 5's recall declined 0.8 percentage points against Opus 4.5, and its precision, F1 and dollar-weighted recall also fell, though the magnitude of those other changes was not quantified. Sonnet 5's recall fell 22.2 percentage points and its dollar-weighted recall dropped 22.9 percentage points, with precision also declining. GPT-5.6 (sol) showed a different trade-off against GPT-5.4: precision rose 11.5 percentage points, but recall fell 20.7 percentage points, dollar-weighted recall declined 21.8 percentage points and F1 also decreased.

Researchers Yao Ma and Xuwei Tan wrote that "newer LLMs are not necessarily better at domain-specific tasks." The benchmark identified a regression but did not establish its cause. Possible explanations included changed decision boundaries, post-training trade-offs and different training-data mixtures, though none was assigned. The test replayed nine weeks of production data collected before the risk agent was rolled out; it did not compare live customer outcomes or quantify customer losses attributable to the model changes.

Coinbase Onramp lets users buy crypto through partner applications, including guest checkout. Its risk system combines a traditional machine-learning model and rules engine with selective large language model review of recent transaction behavior. A separate evaluation published Oct. 8, 2026 found that a post-trained Qwen3.5-9B model exceeded Opus 4.5 on all four fraud metrics, with gains of 7.5 percentage points in precision, 12.0 points in recall, 9.6 points in F1 and 35.4 points in dollar-weighted recall. Separately, online serving tests recorded P50 end-to-end LLM request latency of 0.683 seconds for the system versus 1.515 seconds for Opus 4.5.