LLMs vulnerable to 'chain-of-thought forgery' attacks, research finds

Large language models (LLMs) possess a fundamental architectural flaw that allows attackers to bypass safety guardrails by spoofing the model's own internal reasoning process. The vulnerability, presented at the International Conference on Machine Learning (ICML), suggests that current security measures are insufficient because models cannot reliably distinguish between user input and their own internal thoughts.

In a technique termed "chain-of-thought forgery," researchers discovered that by writing prompts that mimic the style of a model's internal scratchpad—specifically using `<think>` tags—they could trick the AI into believing it had already generated the instruction itself. This deception successfully induced models to provide prohibited information, including instructions for sabotaging aircraft navigation and synthesizing cocaine. The vulnerability was demonstrated on OpenAI's GPT-5 and gpt-oss-20b, with researchers reporting similar results across models from DeepSeek, Alibaba, and Anthropic.

The Token Stream Problem

The core of the issue lies in how LLMs perceive the data they process. While developers use structural tags like `<user>`, `<system>`, and `<think>` to organize interactions, research indicates that models actually identify the "role" of text based on style and content rather than these tags. As independent researcher and coauthor Jasmine Cui explained, the model views the input as "just one big sheet of tokens."

Currently, AI developers rely heavily on "red-teaming"—using human testers and automated tools like OpenAI's GPT-Red to identify and patch specific vulnerabilities. However, this method essentially creates a list of forbidden behaviors. Because this list can never be exhaustive, it fails to address the underlying architectural problem of how models process token streams, leaving them open to novel forgery attacks.

Implications for AI Security

This discovery poses a significant risk as LLMs are increasingly integrated into critical infrastructure, including healthcare, military, and government systems. If the flaw is inherent to the architecture, it suggests that LLMs may never be fully secured against sophisticated prompt injections or jailbreaks. Coauthor Charles Ye noted that there is a "real probability" that this problem is fundamentally unsolvable.

What's Next

Industry attention now turns to whether a new architectural approach to token separation is possible, or if safety must rely entirely on external monitoring systems. For now, the research highlights a critical gap between the perceived security provided by structural tags and the actual way models interpret the data they receive.