Local AI Coding Bottleneck: Massive System Prompts Cripple Laptop Performance
New benchmarks reveal that AI coding harnesses optimized for data centers create 'unbearable' latency on consumer hardware.
The gap between data center compute and local hardware is creating a critical performance bottleneck for AI coding agents. While frontier models are increasingly capable, the frameworks used to steer them—known as harnesses—are often designed with the assumption of near-infinite compute, leaving local users facing massive delays.
In a recent benchmark conducted by Nathan Sutton, nine AI coding harnesses were tested on an M4 MacBook Pro with 24GB of RAM. Running a 3-bit quantization of the Qwen 3.8 27B model via llama.cpp, Sutton found that the 'prefill' stage—where the model processes the initial prompt—became a primary point of failure for responsiveness. The most extreme example was Opencode, whose 18,046-token system prompt and tool schema resulted in a 225.7-second wait before the first token was even generated. Sutton described this disparity between data center and laptop performance as "unbearable."
The Prefill Problem
Modern AI coding assistants rely on extensive system prompts and complex tool definitions to guide the model's behavior. In a data center environment, high-end GPUs handle these large prompts almost instantaneously. However, on consumer hardware, the prefill stage is significantly slower than the actual generation of code.
Sutton categorized the tested harnesses into three distinct tiers based on their efficiency. "Lean and stable" tools included pi, mini-swe-agent, and chad. A middle tier of "Heavy but disciplined" tools consisted of dsh, cline, codex, and goose. Finally, crush and opencode were labeled "Heavy to start," characterized by the massive prompts that trigger extreme latency on local machines.
Engineering for Local Silicon
To address these inefficiencies, Sutton introduced 'chad,' a harness specifically optimized for Apple silicon. By utilizing a minimal tool surface and integrating the server and agent loop into the same process via an MLX engine, the harness significantly reduced overhead. This architectural shift, specifically when paired with the DFlash2 drafter, increased the experienced speed from 7.9 to 17.4 tokens per second.
Why It Matters
This research highlights a fundamental oversight in the development of AI agent frameworks: the assumption that prefill is virtually free. For developers prioritizing privacy and local execution, the choice of harness is now as critical as the choice of the underlying model. Without optimization for local prefill and efficient cache reuse, the promise of a responsive, private AI coding assistant remains out of reach for most consumer hardware.
What's Next
As the industry moves toward more autonomous agents, the tension between complex tool schemas and local performance will likely intensify. The success of the MLX-integrated approach suggests that the next generation of local AI tools must move away from generic API-style wrappers and toward hardware-specific integrations to remain viable.