When Retrieved Text Tries to Give Instructions

Retrieved content should be treated as evidence to inspect, not authority that can redefine an AI system's task

Direct answer

Retrieved content should be treated as evidence to inspect, not authority that can redefine an AI system’s task. A web page, PDF, email, support ticket, or database record may contain useful facts and also contain text that tells the model to ignore rules, reveal data, call a tool, or prefer a source. That is indirect prompt injection. Filtering suspicious phrases is insufficient because ordinary business prose can also contain instructions. The safer design separates trusted instructions from untrusted content, converts handoffs into constrained data, limits tool privileges, requires approval for consequential actions, and logs which source influenced each decision.

Retrieval blurs the line between data and commands

Traditional software usually distinguishes code from data through syntax and execution boundaries. Language models receive both as text. A system message may say “summarize the retrieved pages,” while a retrieved page contains a sentence framed as an instruction. Both are natural language, so the application must preserve their different trust levels outside the model’s interpretation.

This risk becomes more serious when retrieval is connected to tools. A research assistant that can only draft a summary may produce a misleading paragraph. An agent that can send mail, edit records, purchase goods, or expose private documents can turn the same confusion into an external action.

OpenAI’s agent safety guidance describes prompt injection as untrusted text or data attempting to override instructions, potentially causing data leakage or misaligned actions. OWASP’s Prompt Injection risk page distinguishes direct injection from indirect injection delivered through external sources such as websites or files. The control problem therefore begins at every boundary where outside content enters the agent.

A simple trust model

Every input should have an origin, a trust class, and a permitted use.

Trust is contextual. A company’s own help article may be authoritative for product documentation while remaining untrusted as a source of runtime instructions. A customer email may legitimately ask the company to change an order, but the request still needs identity, permission, and workflow checks outside the language model.

Direct and indirect prompt injection

Direct prompt injection comes from the person interacting with the model. Indirect injection is embedded in material the system later retrieves or processes. The second type is easy to overlook because the application sought the source for a legitimate reason.

The 2023 paper “More than you’ve asked for” formalized this remote attack pattern: an adversary can place instructions in data likely to be retrieved by an application, exploiting the blurred boundary between data and instruction. The paper tested systems available at the time. Its demonstrations should be read as evidence of the mechanism, not as a current failure rate for every model or product.

Indirect injection can appear in visible prose, hidden page elements, document metadata, comments, quoted email threads, code repositories, or data fields. A safety program should not publish reusable attack strings in ordinary training material. Defenders need the categories, entry points, expected policy, and observable outcomes rather than copy-ready payloads.

Why phrase blocking and “ignore instructions” are weak controls

A blocklist can catch known wording, but intent can be expressed in countless forms and languages. The word “ignore” is also common in harmless documentation. Aggressive filtering can destroy useful content while novel phrasing passes through.

Telling the model to ignore instructions in documents helps communicate policy but relies on the same model that is interpreting the hostile content. Sanitizing HTML removes scripts and layout hazards; it does not reliably identify a semantic instruction expressed as ordinary text. Retrieval-augmented generation does not solve the problem either. OWASP notes that RAG and fine-tuning do not fully mitigate prompt injection.

Use content detection as one layer. Do not make it the authorization boundary. The decisive controls should live in architecture: typed interfaces, scoped credentials, deterministic permission checks, destination allowlists, approval gates, and validation of tool arguments.

Keep untrusted data out of privileged instruction channels

The most damaging design error is concatenating outside content into a privileged developer or system message. That placement can make the content look like an application rule. OpenAI’s agent guidance recommends avoiding untrusted variables in developer messages and using structured outputs to constrain data passed between workflow steps.

A safer research flow can work as follows:

1. A retrieval component fetches sources and records their URLs, timestamps, and content types.
2. A parser extracts candidate passages as data objects.
3. A classification step labels claims, quoted instructions, forms, code, and uncertain content.
4. A synthesis step receives a fixed task plus the candidate passages in a clearly delimited data field.
5. The output is constrained to a schema such as claims, evidence locators, conflicts, and uncertainty.
6. A separate policy component decides whether any proposed tool action is allowed.

Structured output does not make a model immune. It reduces the room for one component to smuggle a new free-form instruction into the next and lets ordinary software validate field types and allowed values.

Treat source content as claims with provenance

GEO research already needs claim-level provenance. The same practice improves safety. Instead of passing a large document forward as an undifferentiated block, create evidence records:

• source URL or document ID;
• publisher or owner;
• retrieval time and version;
• exact passage or normalized fact;
• page, section, or field locator;
• content trust class;
• claim type and topic;
• confidence or review status;
• detected instruction-like language;
• allowed downstream use.

An instruction-like sentence can still be evidence. A manual may correctly say, “Delete the temporary file after export.” The evidence record should preserve that statement as documentation, while the agent’s action policy decides whether deleting any file is within scope. Meaning and authority are separate fields.

This also prevents citation laundering. If a retrieved page tells the model to cite a particular domain, that preference is not evidence that the domain supports the answer. Source selection should follow relevance, authority, and claim coverage rules defined by the application.

Least privilege limits the consequence of failure

Assume that some malicious or confusing text will reach the model. The next question is what the model can do with it.

Give each workflow only the tools required for its current task. A content summarizer does not need access to billing systems. A product-research agent may need read access to inventory but not permission to change prices. Credentials should be scoped by resource, action, tenant, and duration where the platform allows it.

OpenAI’s safety guidance recommends tool approvals and human confirmation for consequential actions. Approval is most useful when it shows a concrete action: destination, affected record, important arguments, source of the request, and expected result. A generic “allow tools?” prompt does not help a reviewer detect that a retrieved document changed the destination account.

Design the tool layer to reject:

• destinations outside an allowlist;
• identifiers not returned by a trusted lookup;
• data movement across tenants or accounts;
• arguments containing unexplained secrets;
• actions that exceed user or service permissions;
• large or unusual batches without review;
• calls whose purpose is not linked to the active user request.

These are normal application controls. They remain effective even when model behavior is uncertain.

A defensive example

Consider a fictional support assistant asked to summarize customer tickets and draft remediation steps. One uploaded ticket contains text that attempts to redirect the assistant toward an unrelated internal file and a different outbound destination.

A weak workflow places the full ticket in a privileged prompt and gives the model broad file and messaging tools. A stronger workflow treats the ticket as untrusted evidence. It extracts the reported symptoms, product version, timestamps, and quoted messages into typed fields. The drafting component can propose steps but cannot read unrelated files or send anything. If a human later chooses to notify the customer, the messaging workflow receives the approved recipient from the authenticated ticket record, not from the ticket body.

The point is not whether the injection wording looks suspicious. The workflow makes that wording unable to grant itself access or choose its own destination.

Review retrieval and action as separate evaluations

A useful security test has at least two scores.

Retrieval integrity asks whether the system selected relevant evidence, preserved provenance, resisted source-directed ranking, and represented conflicts. Action integrity asks whether the system stayed within authorized tools, arguments, destinations, and approval requirements.

The AgentDojo benchmark evaluates tool-using agents on realistic tasks that include untrusted data. Its published benchmark contains 97 tasks and 629 security test cases. Those figures describe the benchmark, not the prevalence of attacks in production. Its broader lesson is useful: utility and security need to be tested together, because a system that refuses every document is safe but unusable, while a highly capable agent with weak boundaries can be dangerous.

Create a test matrix that covers:

Do not test only one famous phrase. Vary document type, language, location, encoding, and the legitimate task surrounding the hostile content. Include accidental instructions and badly formatted data, because real failures do not always come from an attacker.

Logging for reconstruction, not surveillance

When a safety event occurs, the team needs to reconstruct how content moved through the workflow. Log source IDs, retrieval times, model and workflow versions, structured intermediate records, tool proposals, policy decisions, approvals, and final actions. Apply access controls and retention limits because logs may contain sensitive material.

Keep the original source separate from normalized records. If a reviewer sees only the final summary, they cannot tell whether a statement came from the user, a trusted policy, a retrieved page, or the model’s inference. Provenance should survive every handoff.

Useful alerts focus on boundary violations: attempts to access a new resource, changes in destination, unexpected secret-like output, repeated policy denials, or tool arguments unsupported by trusted state. A high count of the word “instruction” is not a meaningful security metric.

An implementation checklist

1. Inventory every place untrusted text enters the system.
2. Assign each input an origin, trust class, and permitted use.
3. Keep retrieved content out of privileged instruction fields.
4. Parse large sources into provenance-preserving evidence records.
5. Use structured schemas between model or workflow stages.
6. Validate tool arguments with deterministic application rules.
7. Scope credentials and tools to the current task.
8. Require concrete approval for consequential external actions.
9. Test benign utility and adversarial resistance together.
10. Log enough lineage to reconstruct decisions and corrections.
11. Re-run tests when models, tools, prompts, or connectors change.
12. Maintain an incident process for containment, review, and affected-data assessment.

Common mistakes

Trusting a source because its domain is reputable. A trusted site can host user content, compromised pages, quoted text, or instructions that are valid for readers but not for agents.

Stripping HTML and calling the content safe. Semantic instructions remain after tags and scripts are removed.

Letting the model decide its own permissions. Authorization must come from authenticated application state and policy checks.

Giving a research agent write tools for convenience. Split research, proposal, approval, and execution into scoped stages.

Measuring only attack success. Also measure completion quality on legitimate tasks, false positives, review burden, and recovery after suspicious content.

Publishing detailed live payloads in general documentation. Defensive teams can test controlled cases without distributing copy-ready attack material.

Frequently asked questions

Is every instruction in a retrieved document malicious?

1. Instructions can be legitimate document content. The key question is whether they have authority over the current workflow. Usually they should be summarized or quoted as evidence, not executed.

Can a classifier solve prompt injection?

Classifiers and guardrails can reduce risk, but they will have misses and false positives. Use them with privilege limits, structured handoffs, deterministic checks, and approvals.

Does RAG make an agent safer because answers are grounded?

Grounding can improve factual support, but retrieval also creates an indirect-input channel. Retrieved text needs provenance and trust boundaries just like any other external input.

When is human approval necessary?

Use it for actions with meaningful external effects, sensitive-data movement, new destinations, money, deletion, publication, or permission changes. Show the reviewer the exact proposed action and inputs.

How should publishers protect their own content from being treated as instructions?

Write factual pages with clear headings, provenance, and explicit audience context. Avoid embedding crawler-directed or model-directed commands in editorial copy. Publishers cannot control every downstream agent, so application operators still carry the security responsibility.

What does a successful test look like?

The system completes the legitimate task, uses relevant evidence, preserves source lineage, refuses unauthorized changes, and produces no external action beyond the user’s authenticated intent.

Source and method note

This defensive overview combines the Xindar knowledge base’s evidence-boundary and provenance methods with current public security guidance. The principal sources are OpenAI’s agent safety guidance, OWASP’s Prompt Injection risk page, Greshake and colleagues’ indirect prompt-injection paper, and the AgentDojo benchmark paper. These sources use different systems and evaluation settings; benchmark results should not be read as a universal commercial-system failure rate. The example is fictional, and the article intentionally omits reusable attack strings.