A fundamental flaw leaves LLMs strikingly vulnerable to attack

It is impossible to make large language models fully secure against hacks because of a fundamental flaw in how they work, a team of researchers argue in a paper presented at the International Conference on Machine Learning (ICML) this month.

By taking advantage of this flaw — how LLMs identify who or what is giving them instructions — the researchers were able to make popular LLMs spit out information they had been trained not to provide, such as how to synthesize cocaine and how to sabotage a commercial aircraft’s navigation system.

“There’s a real probability that this is going to be a problem that’s fundamentally unsolvable,” says Charles Ye, an independent researcher and coauthor of the ICML paper.

The researchers found that writing instructions in a style that mimicked the text LLMs generate in their chain of thought — a scratch pad models use for notes — would often trick the LLM into behaving as if it had come up with that instruction itself.

They call this attack chain-of-thought forgery. The discovery won OpenAI’s red-teaming hackathon in August 2025. OpenAI researchers claim GPT-Red found a very similar attack around the same time, which they call a fake chain of thought.

Role confusion mechanism

LLMs see a continuous stream of text partitioned into roles (user, assistant, system, tool, chain-of-thought). Experiments showed models identify role not by tags but by writing style and lexical patterns. Swapping tags made almost no difference; if text looked like chain-of-thought, the model treated it as such.

Paper: “Prompt Injection as Role Confusion” (arXiv). CoT Forgery raised attack success from near zero to ~60% on frontier models. Results reported across OpenAI, Anthropic, Alibaba, and DeepSeek models.

Expert reaction

Florian Tramèr (ETH Zürich) likes the paper’s insight; notes defenses have improved but may not suffice for highly sensitive cases. Authors acknowledge many models tested were from last year but argue better training does not fully solve the structural issue.

Ye: organizations shouldn’t trust LLMs and should expect that anything done by agents could be unsafe — “There’s been no study of the fundamental science here. We’re all doing it ad hoc.”