AI Agents Build Virtual Union Square for $33, Exposing Spatial Reasoning Gaps

An experiment to reconstruct San Francisco's Union Square in a virtual environment cost just $33 in API calls, demonstrating the potential for AI to slash content creation costs. However, the project also highlighted significant limitations in how AI agents handle spatial reasoning and real-world geography.

Using Claude Fable 5.1, the project aimed to build a 3D browser reconstruction of the famous square using Three.js. The process consumed roughly 8 million tokens and utilized Playwright to capture screenshots, which were then used for visual verification against actual photographs of the location. While the financial barrier to entry for such a project has vanished, the technical execution revealed that agents still struggle with spatial accuracy and specific architectural details.

The Challenge of Digital Twins

This project serves as a critical case study in the current limitations of AI agents when tasked with complex, multi-step 3D world-building. The transition from raw spatial data to functional virtual assets requires more than just generative capability; it requires a fundamental understanding of how physical spaces are structured. In this instance, the agents encountered visual problems that conventional software tests would typically miss, often needing to make autonomous "calls" when source data was incomplete or ambiguous.

Why Spatial Hallucinations Matter

The results underscore a specific version of the "hallucination" problem: spatial reasoning. While AI can now generate a rough approximation of a city block for a fraction of the cost of traditional 3D modeling, the lack of precision makes these tools unreliable for high-stakes applications. For the development of digital twins—exact virtual replicas of physical assets used in urban planning or engineering—this gap suggests that human oversight remains indispensable. The ability to lower costs is a breakthrough, but it does not yet replace the need for human architectural validation.

The Path Forward

As AI agents move toward more autonomous environment creation, the industry must determine how to better integrate real-world geographic constraints into agent workflows. For now, the Union Square experiment proves that while the cost of creation has plummeted, the cost of accuracy remains high. Future developments will likely focus on whether agents can be taught to self-correct spatial errors through better visual feedback loops or more integrated geographic datasets. This shift from mere generation to precise reconstruction will be the true benchmark for the next generation of spatial AI.