The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI

TL;DR

• 1

  The eval gap is now bigger than the capability gap: Chen argues that enterprises in finance, insurance, and healthcare hesitate not because agents are useless, but because measurement has not kept up with what the models can already do.

• 2

  Good benchmarks start with obsessive task quality: He praises GPQA's multi-reviewer, adversarial quality control process, including expert adjudication, revision loops, and payout incentives tied to agreement.

• 3

  Distribution and headroom matter as much as raw score: MMLU worked because its 57-domain taxonomy was intentional, and ARC-AGI stayed valuable because it remained unsaturated and still launched ARC-AGI-3 with frontier models under 1 percent.

• 4

  Robust evals should measure the thing that actually matters in practice: Tau-Bench is his example because it scores not just task completion for multi-turn agents, but policy adherence, so booking the right flight still fails if it breaks fare rules.

• 5

  The best benchmarks make a directional bet on the field: Terminal Bench bet early that the CLI would become a core interface for general-purpose agents, and Chen says that thesis now looks prescient given Claude, Codex, and enterprise agent workflows.

• 6

  Researcher UX is an underrated adoption driver: Benchmarks like HELM and Terminal Bench 2.0 with Harbor succeeded in part because they gave researchers a modular harness, easy model runs, and a practical path to extending tasks and training loops.