An internal assistant that answers questions about regulations, contracts or technical documentation can save hours every week. It can also become the company's biggest security hole if it answers with a document the user had no permission to read.
The risk is not in the model. It is in the architecture that decides which information enters the context.
The problem is not the prompt
In a basic RAG setup, the flow is always the same: the user asks, the system searches for similar chunks in a vector database, puts them in the prompt and the model writes an answer. If the search does not distinguish between one client's contract and another's, the model will answer with the information you gave it.
Telling the model “do not reveal confidential data” is not access control. It is an instruction it can ignore, that can be manipulated and that does not know which documents the user should have seen. Security has to live in the system, not in the model's goodwill.
Three layers of control
1. Permissions on the document
Every indexed document needs access metadata: company, department, project, confidentiality level and the list of authorised users or groups. That metadata is stored next to the chunk and checked at retrieval time.
If the system does not know who can see a document, it should not index it.
2. Filter before the vector search
The filter must be applied before retrieval, not after. The vector database query should include the user's access conditions:
- same tenant or company;
- same groups or roles;
- same project, if applicable;
- allowed confidentiality level.
Retrieving ten chunks and then discarding seven is bad practice: the three sensitive ones were already in memory and can leak into logs, the answer or a later iteration.
3. Answers with verifiable citations
Every answer should include the source and the exact chunk that supports it. That lets the user verify the information and the team audit why the assistant said what it said. If an answer cannot be cited, it is a sign that the model is filling gaps with general knowledge.
Citations are not just a product feature. They are a security and quality tool.
What to log so you can investigate
An incident without traces is an incident you cannot close. The minimum log should include:
- user identifier and groups;
- original query;
- filters applied;
- retrieved documents and their access level;
- generated answer;
- model and prompt version;
- latency and cost.
With that data you can answer concrete questions: which documents did this user see, why did this chunk appear, did the filter fail or was the document mislabelled?
Common mistakes
Indexing everything and trusting the prompt. This is the most expensive mistake. The vector database becomes a copy of sensitive documents with no control.
Permissions only in the interface. Hiding a button does not stop the API from returning the document. Anyone with an open session can call the endpoint.
Incomplete metadata. A document without a department or tenant ends up appearing where it should not. Metadata validation should block indexing, not generate a warning.
Uncontrolled retries. If the assistant asks the model again when it does not like the answer, you multiply the chances of leaking context. Retries need a limit and a clear policy.
No quality evaluation. A RAG can be safe and still answer badly. You need a set of test questions with expected answers and a periodic accuracy review.
How to start without a six-month project
- Choose a narrow use case and a small group of users.
- Define which documents go in and with which metadata.
- Implement the permission filter before retrieval.
- Add citations and an audit log from day one.
- Measure accuracy, latency and cost before expanding.
A useful internal assistant is not the one that answers everything. It is the one that answers with the right information for the right person, and leaves a trace of why it did so.
At Coderwebs we build this kind of assistant as part of RAG AI assistants, with permissions, citations and evaluation. If the assistant also has to take actions, it is worth reviewing the design of AI agents and their limits.
Conclusion
RAG security is decided at retrieval. If the wrong document enters the context, it is already too late. Per-document permissions, pre-retrieval filters and audit logs are not extras: they are the foundation for using the assistant in a real company.