The Convergence Problem: Why AI Agents Struggle to Know When They Are Done
A new analysis from a16z argues that completion is a systemic judgment, not a property of the task itself.
Autonomous AI agents are transitioning from simple prompt-response interactions to iterative loops where they plan, execute, and refine their own work. However, a critical failure point has emerged: these agents often struggle to recognize when a task is actually finished.
In a recent analysis titled "Knowing When to Stop: The Art of Making a Loop Converge," Andreessen Horowitz (a16z) explores the challenge of convergence in AI agent loops. The piece argues that "done" is rarely an inherent property of the work itself, but is instead a judgment produced by the system surrounding the work. Without a clear convergence criterion, agents risk looping indefinitely or stopping prematurely, resulting in systemic inefficiency or total failure.
The Human Signal Patchwork
To solve this, a16z examines how humans determine completion. Unlike a binary switch, human professional work relies on a complex patchwork of signals to signal the end of a project. These include formal tests, technical specifications, historical precedent, and direct approval from supervisors.
Beyond these explicit markers, humans also weigh external pressures and internal logic. Deadlines and risk assessments often dictate the stopping point, as does the recognition of diminishing returns—the point where further effort no longer significantly improves the outcome. The analysis suggests that for AI to converge, it must be designed to mimic these human signals.
Why Convergence Matters
Solving the convergence problem is essential for the reliability of autonomous AI. Currently, the inability to independently determine if a goal has been met limits the utility of agents in professional environments. If an agent cannot judge when further refinement is wasteful or when a specification has been fully satisfied, it cannot be fully trusted with complex, open-ended tasks.
For the industry, this means the focus must shift from simply improving the agent's ability to execute a step to improving the system's ability to judge the result. Reliability in autonomy depends not on the agent's effort, but on the accuracy of the "stop" signal.
The Path Forward
As agents move toward greater autonomy, the industry must develop more robust frameworks for convergence. This involves building surrounding systems that can provide the same types of signals humans use—such as automated testing suites and risk-based thresholds—to guide the agent toward a logical conclusion. What remains to be seen is how these systems will handle subjective tasks where "done" is a matter of taste or shifting requirements rather than a hard specification.