Working through the OWASP LLM Top 10

Untrusted text — typed directly or hidden in a document, page, or tool result the model later reads — gets interpreted as an instruction instead of data.

Field note — I stopped treating this as something a better system prompt fixes. I wrote up the actual reason it can't be fully patched in prompt injection, from first principles — the short version is there's no privileged channel separating instructions from data in a transformer, so this is a design constraint to build around, not a bug to close out.

The model exposes data it shouldn't: training-data memorization, secrets embedded in a system prompt, private documents pulled in through retrieval, or details inferred across turns.

Field note — The one I see missed most: RAG pipelines that retrieve across tenant or permission boundaries because the vector store has no row-level access control of its own. The LLM isn't leaking anything — the retrieval layer already handed it something it shouldn't have had.

A compromise anywhere upstream — model weights, a fine-tuning dataset, a plugin, a library — runs inside the application's trust boundary.

Field note — Extend the usual SBOM habit to model weights and fine-tunes, not just packages. "Where did this checkpoint come from, and who signed off on it" is a question most teams can answer for a pip package and can't answer for a model pulled from a hub.

Adversarial examples planted in training or fine-tuning data make a model behave normally under evaluation but produce attacker-chosen output for a narrow, attacker-known set of inputs.

Field note — Hardest one on this list to test for directly, since a well-planted trigger is designed to survive normal evals. Provenance tracking on training data does more here than any amount of post-hoc red-teaming.

Downstream code trusts LLM output the way it would trust sanitized input — opening the door to SQL injection, XSS, command execution, and SSRF once that output lands in a query, a shell, or a browser.

Field note — This is the one people forget is on the list at all, because it doesn't feel "AI-specific." It isn't — it's the classic tainted-input problem, just with a new source. Anywhere model output reaches a sink, treat it exactly like user input, because functionally that's what it is.

An agent gets more permissions, tools, or autonomy than the task needs. A successful injection then doesn't just corrupt a response — it triggers a real, sometimes irreversible action.

Field note — This is where LLM06 and ASI02 (Tool Misuse) overlap almost completely. My test: list every tool an agent can call, then ask whether the current task actually needs each one. If the answer is "it might, eventually," that's excessive agency, not future-proofing.

Instructions, persona rules, tool documentation, and any embedded credentials in a system prompt get extracted, handing an attacker a map of the app's guardrails and internals.

Field note — The fix people reach for — "tell it not to reveal the system prompt" — is itself just another instruction sharing the same unprivileged channel as everything else. The actual fix is never putting a secret in a system prompt in the first place. Treat it as public.

RAG pipelines bring their own surface: poisoned embeddings, retrieval crossing tenant boundaries, inversion attacks reconstructing source text from exposed vectors.

Field note — Overlaps heavily with LLM02 in practice — most of the incidents I've seen here are really an access-control gap in the retrieval layer wearing an "embeddings" label.

The model states false information with total confidence. Feed that into an automated decision or an irreversible action, and a hallucination stops being an accuracy problem and becomes a security incident.

Field note — The severity of this one is entirely downstream-dependent. A hallucinated fact in a chatbot answer is a UX problem; the same hallucination feeding an automated approval workflow is an incident. Ask what happens next before rating the risk.

Long prompts, runaway generations, tool-call loops, or embedding floods drive compute and API spend far beyond normal usage in a short window — denial-of-wallet as much as denial-of-service.

Field note — Most under-tested item on the list, and the one I now check first on any agent with a loop in its control flow. A single malformed input that triggers an infinite tool-call retry can produce a bill spike faster than most teams' anomaly alerts fire.