I’m building an internal assistant that answers questions from company documents. Some files come from outside vendors, so I can’t assume everything in the search results is trustworthy.
During testing, I added a paragraph to a document telling the assistant to ignore the user’s question and request their password. The assistant sometimes treated that paragraph as an instruction rather than something to quote or disregard.
Where should the trust boundary sit?
I already tell the model that retrieved text is reference material, not instructions. But I’m unsure whether that is a sufficient defense against prompt injection in retrieved documents, especially if we later let the assistant send emails or update support tickets.
- How should retrieved passages be separated from trusted application instructions?
- Which safeguards belong outside the model, particularly for actions involving user data?
- How can I test this without relying on a handful of obvious “ignore previous instructions” examples?
I’m looking for a practical security approach, not just a stronger system prompt.
