AI Coding Agents Fail Over 60% of Tasks in New Enterprise Benchmark
Specific Labs' Real-SWE benchmark reveals a steep performance drop when AI agents move from public repositories to private production codebases.
The industry's leading AI coding agents struggle to navigate the complexity of private enterprise software, failing the majority of real-world tasks in a new industry benchmark. The results suggest a significant gap between solving isolated coding puzzles and performing professional software engineering in unfamiliar environments.
Specific Labs, a Y Combinator-backed AI data company, released the Real-SWE benchmark to test how agents handle proprietary codebases rather than public GitHub repositories. Claude Fable 5.1, operating via Claude Code, emerged as the top performer but achieved a success rate of only 38.8%. Other frontier models fared similarly, with GPT-6 Astra (via Codex CLI) scoring 33.8% and Gemini 3.8 Flash (via Gemini CLI) scoring 31.2%.
The Contamination Problem
Most existing coding benchmarks rely on public repositories. Because these repositories are often included in the training sets of frontier large language models (LLMs), the results are frequently skewed by data contamination—essentially, the AI is recalling a solution it has already seen.
To eliminate this bias, Real-SWE utilizes proprietary code licensed from actual businesses. The test environment includes a consumer fintech platform and a product serving over 200,000 users. By forcing agents to operate in these private, unfamiliar production environments, Specific Labs aimed to simulate the actual conditions a human engineer faces when joining a new project or maintaining a legacy system.
Implications for the Industry
These findings challenge the prevailing narrative that AI agents are ready to replace human programmers in professional settings. The high failure rate indicates that the ability to generate syntactically correct code is fundamentally different from the ability to navigate, understand, and modify a large, private enterprise codebase.
Solving a coding problem is very different from finding one's way through an unfamiliar production codebase. The results suggest that current 'frontier' agents lack the architectural reasoning and discovery skills required to handle the scale and opacity of real-world corporate software.
What's Next
The Real-SWE results highlight a critical need for new training methodologies that prioritize codebase navigation and system-wide reasoning over simple pattern matching. Developers and enterprise leaders will likely look for agents that can demonstrate reliability in private environments before integrating them into mission-critical production pipelines. Whether future model iterations can bridge this 60% failure gap remains the primary question for the next generation of AI software engineering tools.