20 days of compute vs 7 hours: rethinking what state-of-the-art means — Bertrand Charpentier, Pruna

TL;DR

• 1

  Public leaderboards disagree more than people admit: Bertrand compares Arena, Design Arena, and Artificial Analysis and shows that image editing rankings shift a lot across them, with models like Hume landing around rank 5 on one board and rank 10 on another.

• 2

  Aggregated scores hide the model you actually need: On task-specific image editing leaderboards such as object removal, background changes, and text edits, ChatGPT Image is not consistently number one, because different models are better at different jobs.

• 3

  Manual inspection is doubly biased: His live audience image voting demo shows people prefer different outputs, and some even change their minds across examples, which is why a handful of prompts plus one evaluator is a weak way to pick a model.

• 4

  Generic automated metrics can be noisy and misleading: He shows CLIP score rankings shifting across datasets with tiny score differences, then contrasts that with task-specific text rendering metrics that produce clearer separation and more stable rankings.

• 5

  Compute efficiency changes the meaning of state of the art: Generating 26,000 ChatGPT Image samples at 62 seconds each takes about 20 days of compute, versus 7 hours and roughly $265 for Pruna's faster model, so quality gains have to be weighed against latency, price, and energy.

• 6

  The right framing is Pareto optimality, not one winner: Bertrand recommends plotting quality against latency or cost and choosing among the models on the Pareto front, especially with task-specific metrics, because the best practical model is often a smaller performance-tuned one rather than a huge foundation model.