Cooking with OpenAI’s Research Chief: AGI, o1, Evals, and Scaling Laws — Mark Chen

TL;DR

• 1

  Scaling laws still hold: Chen firmly believes in the exponential, arguing that every time people have said pre-training can't scale further, research and engineering breakthroughs have proven them wrong.

• 2

  Reasoning was a hard internal sell: Even at OpenAI, the o1 reasoning project faced skepticism because the pre-training + post-training paradigm was working so well. It took conviction from Ilya Sutskever and others to push it forward.

• 3

  The eval crisis is real: Chen says there are too few gold-standard benchmarks and teams must separate eval creation from model optimization to avoid "benchmaxing" or overfitting to test distributions.

• 4

  Research taste comes from replication: To develop good research taste, Chen recommends fully replicating papers you admire, trying to match exact training curves teaches techniques authors don't write down.

• 5

  Vibe research is becoming real: Researchers increasingly act as idea generators while models handle implementation. Chen predicts models will develop research taste within a three-year horizon.

• 6

  High-risk bets are OpenAI's alpha: The lab consciously takes risky bets that often fail, but management avoids delusion by cutting losses. One mega hit can justify many misses, like a trading mentality.