I benchmarked the NEW Sonnet 5. The results shocked me.
TL;DR
Sonnet 5 underperformed expectations in blind testing, landing second-to-last in the creator's personal ranking despite Anthropic's claims of near-Opus performance at lower cost.
Gemini 3 Pro unexpectedly topped the AI-judged leaderboard, tied with Sonnet 5, surprising the creator who had forgotten Gemini was even in the test set.
Human taste diverged sharply from AI judges, with the creator preferring Sonnet 4.6 while AI judges scored it poorly, revealing that automated evals miss qualitative "vibe" factors.
Different models excel at different tasks, with GPT 5.5 recommended for PRDs, Sonnet 4.6 for prototyping and conversation, and Opus 4.8 for complex UI work.
The benchmark cost roughly $2 per million input tokens at launch pricing, making it affordable to run comprehensive evals before prices increase after summer.
Summary
Claude Sonnet 5 landed at the bottom of a blind taste test ranking, beaten by Sonnet 4.6, Gemini 3 Pro, and GPT 5.5 in the creator's own benchmark. The How I AI bench revealed a striking split between AI judges and human taste, with models agreeing on quality but the human preferring completely different outputs.
Was This Useful?
Share
Keep Reading
Make Alcreon Yours
Tune your feedFive quick questions, and the feed ranks what matters to you first.Or just get notified
The weekly Echo. Signal worth keeping in your inbox.
Every new piece, announced on X.
Read Next
See all
Playbook
The Retirement Email Isn't a Warning
Model retirements now arrive every few weeks; the config-eval-rehearsal loop turns each deprecation email from a fire drill into an afternoon swap.
Playbook
The Cheapest Model That Passes
OpenRouter lists 400 models behind one API. The fix for choosing isn't a better leaderboard, it's a four-step protocol that ends in a real eval.
Playbook
Cheap Models, Hard Tasks
Most agent workflows route every step to the frontier model by default. The bill scales with how chatty the agent gets, even when most steps don't need that brain.