Braintrust CEO: Evals are the new PRD for AI products

TL;DR

• 1

  Evals are the new PRD: Goyal says AI shifts programming from defining the how to defining the what, so the core product job becomes writing quantifiable success criteria and examples instead of prose-only specs.

• 2

  Agents can handle serious systems work, not just CRUD apps: At Braintrust, agents run continuous experiments on database internals like Bloom filters, Tantivy column stores, EC2 to S3 latency, and query patterns across billions of traces over 90 days.

• 3

  The real quality gain is rigor, not magic code generation: Goyal argues no human staff engineer will manually run as many benchmarks, compare as many algorithms, or keep testing edge cases as relentlessly as an agent guided by strong evals.

• 4

  Taste scales when you encode it: Braintrust uses evals to capture designer David's judgment, then applies his quality bar across more outputs so his taste matters more, not less.

• 5

  Maker time matters more in the agent era: Goyal keeps mornings for meetings, afternoons for coding, and runs roughly four to six foreground agents in parallel through tmux sessions, with heavier experiments offloaded to remote compute.

• 6

  If the agent is flailing, fix the eval and restart: His default recovery move is not to argue with the model but to close the session, improve the scoring setup, and try again from scratch, especially after watching a vibe-coded 3,000-line eval script turn into junk.