LLM Security: Prompt Injection and the New Attack Surface

Applications built on large language models have introduced a vulnerability class with no clean precedent. Prompt injection resembles older injection flaws in shape, but differs in a crucial respect: there is no equivalent of a prepared statement. The model consumes instructions and data through the same channel, and no reliable mechanism exists to guarantee separation.

Direct and indirect injection

Direct prompt injection is a user deliberately crafting input that overrides the application's intended instructions — persuading a support assistant to ignore its constraints, reveal its system prompt, or behave outside its remit.

Indirect prompt injection is more consequential and less understood. Here the malicious instruction is placed in content the model will later process: a web page it browses, a document it summarises, an email it reads, a code comment, or a database record. The user never sees it. The model reads it as part of its context and may act on it.

This matters enormously for agentic systems. A model that can browse, call APIs or execute tools becomes an actor that untrusted content can influence — and it holds the application's privileges while doing so.

Why filtering does not solve it

The instinctive defence is to filter malicious instructions from input. This is substantially weaker than it appears.

Natural language has effectively unlimited paraphrase. Instructions can be encoded, translated, embedded in images processed by multimodal models, split across documents, or expressed indirectly. A filter that blocks known phrasings addresses the examples it was trained on, not the space of possible inputs.

The deeper issue is architectural. In SQL injection, parameterisation separates code from data at the protocol level, which is why it works completely. No equivalent exists for a language model, because the model's input is inherently unstructured text and its "instructions" are not syntactically distinct from its data.

Design as though injection will succeed

Since the input channel cannot be made safe, security must come from limiting consequences.

1. Treat model output as untrusted input. Never pass it directly into a shell, query, filesystem call or rendered HTML without the validation you would apply to a user submission.
2. Apply least privilege to tools. If the model can call an API, that API should be scoped to precisely what the feature needs. An assistant that can read one mailbox is very different from one holding broad mail permissions.
3. Keep the user's authorisation, not the application's. Tool calls should execute with the requesting user's permissions, so the model cannot become a route to data that user could not otherwise reach.
4. Require confirmation for consequential actions. Sending, deleting, purchasing or changing configuration should involve a human decision, not model discretion.
5. Isolate untrusted content. Where the model must process external documents or web pages, restrict what it can do while doing so.
6. Constrain output format and validate it against a schema before acting on it.
7. Log prompts, tool calls and outputs so abuse is investigable after the fact.

The rest of the risk surface

• Sensitive information disclosure — models revealing content from their context, including data belonging to other users where context is poorly isolated.
• Insecure output handling — model output rendered as HTML producing XSS, or passed to an interpreter producing code execution.
• Excessive agency — granting a model more capability than the task requires, so a successful manipulation has a large blast radius.
• Supply chain risk — models, adapters and datasets obtained from third parties carry the same provenance questions as any dependency.
• Training data poisoning — relevant where you fine-tune on data that others can influence.
• Unbounded consumption — expensive inference invoked without rate limiting, which is both a cost and availability problem.

Testing these systems

Conventional application testing does not cover this. Assessing an LLM-backed system means attempting direct injection, planting indirect instructions in content the system will ingest, probing whether tool calls can be induced outside intended scope, and checking whether one user's context can leak into another's.

It is also inherently probabilistic. A model may resist an attempt once and comply on a rephrasing, so a single successful defence is not evidence of robustness. Testing should therefore focus on the containment architecture — what happens when manipulation succeeds — rather than only on whether a particular prompt was refused.

Related from TechBiz Security

• Web application penetration testing
• API security testing
• Secure code review

Sources & further reading

• OWASP Top 10 for Large Language Model Applications
• NIST AI Risk Management Framework
• NIST — Adversarial Machine Learning taxonomy
• MITRE ATLAS — Adversarial Threat Landscape for AI Systems

Frequently asked questions