Anthropic adds structural web awareness to Claude to fix coordinate errors
A new browser tool allows Claude to target web elements by reference, reducing the reliability gaps found in purely visual computer use.
Anthropic has launched a new browser tool for Claude designed to improve how the AI interacts with the web. The update moves the model beyond simple visual interpretation toward a more precise, structural understanding of web pages.
According to The New Stack, the tool provides Claude with a structured view of a web page's underlying architecture in addition to visually rendered content. This allows the AI to target specific web elements by reference. Previously, Claude relied on "computer use" capabilities that required guessing coordinates from screenshots—a process prone to mapping errors. By integrating structural data, the model can now identify and interact with elements more accurately.
The shift to browser-native interaction
This development follows the introduction of Anthropic's broader "Computer Use" capability, which enables Claude 3.5 Sonnet to navigate desktop environments by taking screenshots and moving a cursor. While that approach is versatile, its reliance on visual-only data often leads to precision failures in dense user interfaces.
The new browser tool represents a transition toward "browser-native" interaction. By combining visual perception with a DOM-like awareness of page structure, the AI no longer has to rely solely on the visual layout to determine where to click or what to select.
Implications for AI automation
Moving from purely visual interaction to a hybrid of visual and structural data significantly reduces the likelihood of the AI "hallucinating" coordinates. For the industry, this increases the overall reliability of AI agents performing autonomous tasks.
This precision is critical for professional workflows where a single misclick in a complex UI can disrupt a process. By increasing the hit rate for specific buttons and fields, Anthropic is making AI-driven web automation a more viable tool for high-stakes enterprise environments.
What remains to be seen
While the structural approach solves many of the coordinate issues associated with screen-native tools, the industry will be watching to see how this scales across diverse, non-standard web architectures. It remains to be seen how this toolset will be integrated into wider developer workflows and whether it will eventually replace the visual-only method for all web-based tasks.