Benchmarking semantic code retrieval on Claude Code — Kuba Rogut, Turbopuffer

TL;DR

• 1

  Cursor's production results are strong: Kuba cites Cursor's blog showing a 24% relative improvement in answer accuracy for Composer, plus a 2.6% increase in code retention and a 2.2% drop in dissatisfied requests on large codebases.

• 2

  Embeddings act like cached compute: Instead of re-grepping the same codebase every session, semantic indexing pays an upfront chunk-embed-index cost so multiple agents can query stored meaning later with fewer tokens and less repeated work.

• 3

  Turbo Grep raised precision sharply on Claude Code: In Kuba's 50-task ContextBench-style evaluation, file precision improved from about 65% baseline to 87% with windowed grep plus semantic search, meaning wasted file reads fell from roughly 1 in 3 to 1 in 8.

• 4

  Semantic search did not automatically improve recall: Raw Claude Code still led on file recall because it aggressively explores many files, while semantic search and windowed grep ended up with similar recall despite better targeting.

• 5

  The tool choice depends on the task shape: Semantic search won when files were behaviorally related but did not share obvious keywords, such as logic spread across multiple ORMs and libraries, while grep won when the task was basically tracing imports from an early keyword hit.

• 6

  Inline comments make semantic retrieval better: Kuba says messy code is harder, and repositories with strong inline documentation performed noticeably better because the embedding model could infer the chunk's meaning more accurately.