AI Observatory Finds Users Engage in More Sensitive Behavior Than Labs Report
Independent research reveals a gap between corporate narratives and real-world usage, finding nearly half of conversations would be filtered by industry standards.
A new independent research project is challenging the official narratives provided by major AI labs regarding how people actually use generative AI. The AI Observatory, a collaboration involving researchers from Stanford and MIT, has analyzed real-world conversations to provide a counter-narrative to the curated reports typically published by companies like OpenAI and Anthropic.
According to data reported by MIT Technology Review, the AI Observatory aggregated 24,521 conversations across 85,633 turns from 5,000 users who interacted with 52 different models between 2023 and 2025. The findings suggest that users engage in significantly more sensitive behaviors—including harassment, adult content, and health-related queries—than AI companies generally disclose. In a striking comparison, researchers found that if they applied Anthropic's own work-focused filtering methods to the Observatory's dataset, 48% of the conversations would have been filtered out entirely.
The Gap in Corporate Reporting
Major AI labs regularly publish usage reports to demonstrate the utility and safety of their systems. However, academic researchers argue these reports are often designed to present the companies in the best possible light and lack independent corroboration. Shayne Longpre, a PhD graduate from the MIT Media Lab, noted that "no single company report tells the whole story."
This lack of transparency is a central concern for the academic community. Anka Reuel, a Computer Science PhD candidate at the Stanford Trustworthy AI Research (STAIR) Lab, emphasized that for many corporate claims, "there is no independent source to corroborate it." The AI Observatory was established specifically to create a public platform for assessing the actual risks and benefits of generative AI using raw, real-world data rather than proprietary, internal metrics.
Divergent Model Preferences
The study also identified distinct patterns in how users choose specific models for different tasks. The data shows that users favor Anthropic for coding tasks, while Gemini is the preferred choice for social interaction and roleplay. ChatGPT remains the primary tool for homework assistance.
Information retrieval patterns also varied by platform. While both Grok and Gemini are used frequently for seeking information, Grok is particularly popular for news and politics. However, the researchers also found that Grok's usage in these areas shows a concentration of misinformation.
Implications for Regulation
These findings arrive at a critical moment as policymakers and global stakeholders make consequential decisions regarding AI regulation and safety. Because many of these decisions are based on proprietary data provided by the AI labs themselves, the AI Observatory's results suggest that "company narratives" may be masking significant behavioral trends and systemic risks.
Moving forward, the project aims to continue monitoring how generative AI is deployed in the wild. The primary goal remains providing a transparent, independent baseline to ensure that safety regulations are based on how the public actually interacts with these tools, rather than how the developers wish those interactions to be perceived.