./blog/prompt-injection-first-principles
There is no privileged channel in a transformer that says "these tokens are instructions, those are data." That single fact is the whole vulnerability class.
Prompt injection, from first principles
Why no amount of prompt engineering fully patches this
Every few months a new "prompt injection fix" makes the rounds — a better system prompt, a classifier bolted on the front, a stricter set of delimiters. Some of it genuinely helps. None of it closes the hole, because the hole isn't a bug in any particular model. It's a direct consequence of how transformer-based LLMs represent text. Once you see why, you stop waiting for the patch and start designing around it.
Traditional software keeps code and data in separate channels — a SQL engine never executes a bound parameter, no matter what string you put in it. An LLM has no equivalent separation: system prompt, user input, and retrieved content are all just tokens in one shared context window, and the model decides at inference time what looks like an instruction. That's structurally the same code/data confusion that produces buffer overflows and SQL injection — except here it's the architecture itself, not a specific implementation bug, so it can be mitigated but not eliminated by a smarter prompt.
the-structural-problem --no-privileged-channel
A parameterized SQL query works because the database engine treats the query template and
the bound values as fundamentally different things, handled by different code paths. The
value '; DROP TABLE users; -- passed as a bound parameter is never parsed as
SQL — it's inert data, full stop, regardless of what it looks like. That guarantee lives in
the engine, not in how carefully the input was worded.
LLMs don't have that engine-level guarantee. A system prompt, a user's message, and a paragraph pulled in from a retrieved document all get tokenized and concatenated into the same context window, then processed through the same shared embedding space. There's no tag on any token that means "this one is authoritative, this one is untrusted content." The model infers what's an instruction from patterns in the text itself — which means anything that pattern-matches convincingly enough to "instruction" can act like one, regardless of which part of the prompt it arrived in. Recent formal work on this describes it plainly: in shared-embedding sequence models without an enforced control/data boundary, perfect prevention of prompt injection is not an engineering gap to be closed, it's mathematically unreachable with this architecture. It's the same code/data confusion that produces buffer overflows in C and injection flaws in dynamically built SQL — just moved into a new substrate.
direct-vs-indirect --two-flavors-same-cause
Direct injection is the obvious case: a user types "ignore your previous instructions and…" straight into the chat box. It's the easiest to defend against precisely because it's obvious — a system prompt reminder or a classifier on user turns catches a lot of it.
Indirect injection is the harder, more consequential version, and it's the one that matters once an LLM is wired up to tools, browsing, or a mailbox. The instructions don't come from the person talking to the model — they're hidden in a webpage, a PDF, a support ticket, or a dataset the model reads as part of doing its job, and because there's no privileged channel, the model has no structural reason to trust the user's original goal over text it just ingested from a document. This is exactly the mechanism behind OWASP LLM01 (Prompt Injection) and its agentic cousin ASI01 (Agent Goal Hijack) from the OWASP Agentic Top 10 — and it's the same objective-drift pattern I wrote about in the Hugging Face incident post-mortem, just without a human attacker required at all. An agent chasing an underspecified goal can talk itself into the wrong action with no injected text in sight; add a hostile document into its context and the same mechanism becomes a deliberate attack surface.
why-filtering-fails --the-input-space-is-too-big
Keyword and pattern filters catch the injection attempts someone already thought of. They lose to the ones nobody wrote a rule for, and the space of rephrasings is effectively unbounded: paraphrase the instruction, translate it into another language mid-prompt, split it across multiple turns, encode it in base64 or leetspeak, hide it in invisible Unicode characters, or bury it in a code comment inside a file the model is asked to summarize. Once models went multimodal, the same trick works through an image with text baked into the pixels, or audio with an instruction spoken under the audible content — the injection doesn't need to be readable to a filter scanning raw text if the model itself can still perceive it. Every one of these is the same underlying attack wearing a different outfit, and a filter built for last month's outfit doesn't generalize to next month's.
what-actually-helps --defense-in-depth
None of this means the situation is hopeless — it means the fix has to be architectural, not linguistic. The defenses that hold up in practice reduce blast radius and add friction rather than promising prevention:
Privilege-separated agents
Patterns like CaMeL split the job: a privileged planner works from the trusted user goal, while a quarantined executor processes untrusted content with no tool access of its own, and a policy layer checks every tool call against tracked data provenance before it runs.
Structured tool calls, typed arguments
Constrain what a model can ask a tool to do with a narrow, typed schema instead of free-form strings. An injected instruction can still try to trigger a call, but it can't smuggle arbitrary payloads through a field that only accepts an enum.
Spotlighting and sandwiching
Clearly delimiting untrusted content and repeating the real instruction after it measurably reduces success rates in practice — it's a mitigation, not a boundary, and should never be the only layer.
Output and action validation
Validate what the model produces — and what it's about to do — against a schema or policy before it reaches a database, a shell, or an API, the same way you'd never trust raw user input downstream without checking it first.
Assume it eventually succeeds
Design the system so a successful injection is contained, logged, and reversible — least-privilege credentials, human approval on consequential actions, span-level observability that turns a bypass into an alert instead of a silent success.
my-take --closing-thoughts
The framing that changed how I think about this: prompt injection isn't a content-moderation problem, it's an access-control problem wearing a content-moderation costume. Nobody expects a firewall to read every packet and decide if it "feels malicious" — we build segmentation, least privilege, and validation at trust boundaries instead, and accept that perimeter defenses will eventually be bypassed. LLM-integrated systems need the exact same posture, because the model itself cannot be the trust boundary. It's the thing operating inside the boundary, reading whatever crosses it.
sources --further-reading
Related on this site: Working through the OWASP LLM Top 10 · Reading the Hugging Face incident post-mortem