Prompt injection, where guardrails get dangerous
The guardrails from Week 1 return with real teeth. Once an agent uses tools and reads outside content, a malicious instruction hidden in a web page, email, or document can hijack it into taking actions you never intended. That is prompt injection, and it is the defining security problem of agentic systems.
Why tool-using agents are uniquely exposed
A chat model that only talks can be tricked into saying something dumb. An agent that can send email, move money, or delete data can be tricked into doing something irreversible. Retrieved content and tool outputs are untrusted input. Treat them as data, never as instructions that override your system policy.
Attack shape (mental model)
- Attacker plants text somewhere your agent will read (page, PDF, ticket, MCP tool result).
- That text says "ignore previous instructions and do X" (exfiltrate data, call a dangerous tool, approve a transfer).
- If your loop blindly trusts content, the model may comply and your Act step runs X.
Practical defenses
- Least-privilege tools: remove or gate anything irreversible.
- Human-in-the-loop before high-impact actions.
- Separate "instructions" (system / developer) from "untrusted content" (retrieved docs, emails) in the prompt structure.
- Validate tool arguments against allowlists and schemas between steps.
- Monitor and eval for injection cases the way you eval for quality.
Critical
Never trust retrieved text as policy
If a document can change what tools the agent is allowed to call, you have already lost. Policy lives in your code and system prompt, not in the corpus.
Watch out
Common mistakes
- Assuming "the model will know not to do that."
- Giving the agent a browser or email tool with no approval step.
- Only testing happy-path demos with clean documents.