Frontier AI at Home — Alex Cheema, EXO Labs

TL;DR

• Cheema thinks there’s still a 100x price-to-performance gain left in local AI — his thesis is that kernels, harnesses, model design, orchestration, and hardware co-design will compound enough that a roughly $5,000 box can deliver near-frontier local inference within 18 months to 2 years.

• Inference is a memory problem, not a FLOPs problem — unlike training, local decode is dominated by fitting models in memory, memory bandwidth, and energy per byte, which is why Apple Silicon’s huge RAM pools matter and why phones still fail on battery life and heat.

• EXO found embarrassingly large software inefficiencies in today’s stack — Cheema says a look at Qwen 3.5 on Apple Silicon showed performance was about 50% below theory, and simple kernel fusion work alone improved inference speed by 30%.

• Cheema’s local-vs-cloud case is philosophical and practical — citing Karpathy’s “not your weights, not your brain,” he argues that if AI becomes an exocortex, relying on centralized APIs means privacy risk, lockouts, censorship, and token costs that some teams already experience at “thousands of dollars a day.”

• The live demo centered on heterogeneous inference, not just Mac clusters — EXO auto-discovered devices, sharded a model across four Thunderbolt-connected Mac Studios, and then showed a MacBook plus Nvidia Spark split where prefill ran on the higher-compute Spark and decode on the higher-bandwidth MacBook, yielding about a 2x speedup on a large prompt.

• He thinks cloud batching advantages may shrink if agentic and continual-learning workloads win — multi-agent systems like Grok, test-time scaling, and especially test-time training would all increase effective local batch sizes or break cloud batching entirely because every user’s model weights would diverge.

The Breakdown