AI Cannot Hack Alone: Why Human Expertise Remains the Critical Link

The push toward fully autonomous cybersecurity tools is hitting a wall of complexity. While Large Language Models (LLMs) can execute scripts with speed, they currently lack the strategic intuition required to breach sophisticated targets without human guidance.

James Kettle, Director of Research at PortSwigger—the company behind the industry-standard Burp Suite—has spent his recent research pushing AI to its absolute limits to identify exactly where the technology fails. According to Kettle, the most effective and dangerous attacks are not the product of standalone AI, but rather a synergy where a human expert provides the strategic direction and interprets complex results that the AI cannot yet grasp. He argues that the most potent threats emerge when human intelligence drives the strategy and machine efficiency scales the execution.

The Gap in Autonomous Pentesting

The cybersecurity industry is currently grappling with a significant gap between an AI's ability to run a tool and a human's ability to understand the nuanced logic of a target system. This has led to a shift in the narrative: AI is not replacing hackers, but is instead amplifying the capabilities of expert practitioners. By automating the tedious portions of a breach, AI allows a skilled operator to scale sophisticated attacks that would have previously taken weeks of manual effort, effectively turning the AI into a force multiplier for the human operator.

The Danger of Automation Bias

However, the integration of humans into this loop creates its own set of vulnerabilities. Kettle identifies "automation bias" as a critical risk, where human overseers begin to over-trust AI-generated output. This leads to a scenario where humans "rubber-stamp" actions without performing a critical review, effectively turning a supposed safeguard into a blind spot. If a human operator blindly approves a malicious action suggested by an AI, the "human-in-the-loop" security model becomes illusory, as the human ceases to be a critical filter and becomes a passive conduit.

The Threat of Fragmentation

Among the most concerning technical developments is the "fragmentation attack." In multi-agent AI environments, this technique allows a malicious payload to be split across different agents. Because no single agent sees the full payload, current AI-based defenses struggle to detect the attack. This specific method represents a realistic and potent threat in the evolving landscape of agentic AI, exploiting the lack of holistic visibility across distributed AI systems.

What's Next

As AI agents become more integrated into security workflows, the industry must move beyond the novelty of autonomous scripts and address the psychological vulnerability of automation bias. The focus is now shifting toward how to maintain rigorous human oversight without falling into the trap of blind trust. The future of cybersecurity will not be defined by the total autonomy of the machine, but by the quality of the human-AI partnership and the ability of experts to remain skeptical of the tools they employ.