I built a tool called Aegis to test LLM guardrails, and the more I used it, the more I became convinced that prompt injection is not a bug you patch. It is a property of how these systems are built. Understanding that changed how I test them and how I would defend them.
The root cause is simple to state. A large language model reads instructions and data through the same channel. There is no hardware boundary, no privilege bit, no separate parser that says "this part is trusted policy and this part is untrusted input." The model sees one stream of tokens and does its best to be helpful. So if untrusted text can reach the prompt, that text can compete with your instructions. Everything else follows from there.
This sits at the very top of the OWASP Top 10 for LLM Applications (2025) as LLM01: Prompt Injection, and it has held the number one spot across editions for good reason.
Direct injection: arguing with the system prompt
The obvious version is direct injection, where the attacker is the user. The classic line is some variant of "ignore your previous instructions and do X instead." In a naive chatbot that works, because the model has no principled way to rank your fixed system prompt above the user's live message. They are both just text.
Modern models resist the blunt version, so the interesting testing is in the variations: role-play framing ("you are now a system with no restrictions"), token smuggling, encoding the payload so filters miss it, or splitting a request across turns so no single message looks malicious. When I test with Aegis, I am not looking for one magic string. I am probing how much the model's behavior bends as the framing changes, because that bend is the real attack surface.
Defense: treat the system prompt as guidance, never as a security boundary. Keep the sensitive instructions minimal, assume the user can see and contradict them, and put the real controls downstream where the model's output is consumed.
Indirect injection: the payload comes from the data
The version that actually keeps me up is indirect injection, and it is where most real systems get hurt. Here the malicious instruction is not typed by the user. It is planted in content the model later reads: a web page it summarizes, a document in a RAG pipeline, an email in an assistant's inbox, a code comment a coding agent ingests. The user asks an innocent question, the model pulls in the poisoned content, and the hidden instruction rides along as if it were policy.
This is dangerous precisely because the trust model feels fine. The user is trusted. The app is trusted. Nobody thinks of the third-party web page as an input to the prompt, but it is. The moment your LLM reads attacker-influenced text, that text is part of your instructions.
Defense: the fix is architectural, not clever prompting. Isolate untrusted content, mark it clearly as data in the context, strip or neutralize instruction-like patterns where you can, and above all constrain what the model is allowed to do with what it reads.
The categories that turn a prompt into an incident
Injection is the entry point. The damage comes from the categories around it in the OWASP list, and Aegis checks for all of them:
- LLM02 Sensitive Information Disclosure and LLM07 System Prompt Leakage. Once you can steer the model, you go after what it holds: the system prompt, hidden context, other users' data in a shared context window. I always test whether the model will recite its own instructions, because that leakage is both a finding on its own and a map for the next attack.
- LLM05 Improper Output Handling. This is the one developers underrate. If the app renders model output as HTML, that output becomes stored XSS. If it pipes output into a shell or a SQL string, you get command or SQL injection driven by a language model. The LLM is now an untrusted input source to the rest of your stack, and it must be treated like one.
- LLM06 Excessive Agency. The blast radius scales with the tools you hand the model. A model that can only chat is low risk. A model wired to send email, call internal APIs, or execute code turns a successful injection into real action. Least privilege for tools is the single highest-value control I know of here.
How I actually test: purple, not just red
The reason I built Aegis instead of collecting jailbreak strings is that a pile of one-off payloads does not tell you whether a system is safe. It tells you whether it blocked those specific strings today. So the harness is organized around the OWASP categories, generates families of test cases per category, and scores the outputs against what a safe response should look like. The goal is coverage and repeatability, so a fix can be proven and a regression can be caught, which is exactly how I think about detection work on the defensive side.
That framing matters for defenders too. Every offensive case in the harness has a matching control:
- Input side: validate and constrain user input, isolate and label untrusted retrieved content, and do not let external data silently become instructions.
- Model side: minimize the sensitive content in the system prompt, and never rely on the prompt alone to enforce a security rule.
- Output side: treat model output as untrusted. Encode it before rendering, never pass it unescaped into HTML, shells, or queries, and validate it against an expected schema.
- Action side: apply least privilege to tools, and require human confirmation for anything sensitive or irreversible. This is the LLM06 control, and it is the one that turns a critical finding into a contained one.
None of these is a silver bullet, and that is the point. Prompt injection does not have a single fix because it is not a single bug. You defend it in depth, the same way you defend anything where trusted and untrusted data share a path.
What testing LLMs taught me about the rest of security
The most useful takeaway was not a payload. It was a reframing. An LLM is a confused-deputy engine: powerful, eager to help, and unable to tell your instructions from an attacker's on its own. That is the same class of problem as a SQL query that concatenates user input, or a template that trusts its data. We have been fighting "code and data sharing a channel" for decades. LLMs are the newest and loudest instance of it.
So I stopped looking for the one prompt that breaks everything and started asking the boring, durable questions. What untrusted text can reach this model? What is it allowed to do with what it reads? And where does its output go next? Answer those three and you have defended most of the attack surface. Miss any one and no amount of clever system prompting will save you.
Resources
- OWASP Top 10 for LLM Applications (2025): https://owasp.org/www-project-top-10-for-large-language-model-applications/
- OWASP GenAI Security Project: https://genai.owasp.org
- Aegis (my LLM guardrail testing tool): https://github.com/gpamarthy/aegis