Prompt Injection as Role Confusion
ICML 2026 paper. Authors: Charles Ye, Jasmine Cui, Dylan Hadfield-Menell.
Abstract: LLMs partition text into roles (user, tool, assistant, etc.) but perceive source from writing style, not labeled tags — “role confusion.” CoT Forgery injects fabricated chain-of-thought; models mistake it for their own reasoning. ~60% attack success against frontier models with near-zero baselines. Role confusion predicts attack success before generation. Mechanism generalizes to standard agent prompt injections.
Project site: https://role-confusion.github.io/
GitHub: https://github.com/role-confusion/prompt-injection-as-role-confusion