Runta's FrontierHarness Eval Reveals 17x Cost Variance in AI Agent Frameworks
A new benchmark shows that the harness managing an LLM's environment can drastically alter the cost and success rate of software engineering tasks.
Runta has released FrontierHarness Eval, a benchmark demonstrating that the framework managing an AI agent's interaction with its environment can change operational costs by up to 17 times. By isolating the "harness" variable, the study proves that the wrapper around a model is not neutral and significantly impacts both financial efficiency and task success.
The evaluation tested nine different harnesses across 12 configurations, utilizing the Kimi K3 model on Runta agent runtimes. To ensure accuracy and prevent warm-cache bias, the researchers used identical cold starts via fresh checkpoint restores for every test. The results revealed a massive disparity in spending: the Exo Harness emerged as the most cost-effective option at $1.05 per task, while Claude Code was the most expensive, costing $18.34 per task.
The Performance Trade-off
While cost varied wildly, the study also tracked quality and speed. Codex achieved the highest overall quality, recording a 66.7% pass rate with a median cost of $3.47 per task. For those prioritizing velocity, DSH Minimal led the group in speed with a median runtime of 5 minutes and 41 seconds and a 56.7% pass rate. These metrics highlight a critical tension in agent architecture: the most affordable frameworks are not necessarily the most capable, and the fastest are not always the most accurate.
Why the Harness Matters
As AI agents evolve from simple chat interfaces into autonomous software engineering tools, the harness—which handles tool use, loop management, and environment interaction—becomes a critical architectural component. This research demonstrates that choosing the wrong harness can make the same underlying model prohibitively expensive or significantly less reliable.
For enterprises deploying AI agents at scale, these findings suggest that optimizing the harness is as critical as selecting the LLM itself. A mismatch between the model and its management framework can lead to massive waste in compute spend without a corresponding increase in the pass rate for complex engineering tasks. This shift in focus suggests that the "intelligence" of an agent is a composite of the model's reasoning and the harness's execution efficiency.
Future Outlook
The FrontierHarness Eval provides a baseline for understanding how different agentic loops affect the bottom line. As more frameworks emerge to compete with established tools, the industry will likely shift toward more transparent reporting on the cost-per-successful-task. The primary remaining question for developers is whether the highest-quality harnesses can be optimized to match the cost-efficiency of leaner configurations like Exo, or if a permanent premium on quality is inevitable in autonomous software engineering.