The company updates a customer policy, yet its internal AI assistant keeps quoting the old one. It is tempting to blame the model. But if the retrieval system supplied an obsolete document, switching models may not help at all.
That is the central challenge with RAG: giving AI access to documents is not the same as giving it reliable knowledge. Someone still has to own which sources were selected, who was allowed to see them and whether the answer follows from the evidence.
The model does not need to memorise your company
Retrieval-Augmented Generation searches external information before generating a response. For a company assistant, the sources might include policies, contracts, product documentation and a knowledge base.
Documents are typically prepared in advance: split into chunks, associated with their source, version and permissions, and placed in a search index. When a question arrives, relevant passages are retrieved and supplied to the language model alongside the user's request.
Search might use keywords, semantic similarity or both. Vector search is useful when people phrase the same question differently. But a strong semantic match can still be the wrong evidence. An older policy may look extremely relevant to a question about today's rules.
Keeping information outside the model makes updates easier than retraining for every policy change. It does not make freshness automatic. The index must be refreshed, obsolete chunks retired and the new version made retrievable.
A wrong answer can have several different causes
Consider a question about the refund terms of a particular contract. The assistant can fail in at least three ways:
- It retrieves the general terms instead of the customer's specific agreement.
- It retrieves the right agreement but omits a key exception or effective date when assembling context.
- It receives the necessary context and still interprets it incorrectly.
The customer sees the same symptom: a bad answer. The engineering work is different in every case. First inspect ranking and retrieval, then document preparation and the context supplied to the model, and finally how the model uses that evidence.
This is why I would not judge a RAG system by overall chatbot satisfaction alone. Build a test set from real questions with verified answers. Measure whether retrieval finds the necessary sources and whether the generated response actually agrees with them. Microsoft's RAG evaluation guidance separates these concerns for good reason.
Permissions belong in retrieval, not in a polite instruction
Imagine restricted HR documents indexed beside public company policies. An employee asks a question, retrieval picks a confidential passage and the model paraphrases it. The leak has already happened before anyone evaluates the tone of the answer.
Access controls have to be enforced on the retrieved material before it reaches the model. Permissions should follow document chunks, and changes or deletions in the source must propagate to the index. Telling the LLM not to disclose confidential content is not an access-control system.
Retrieved documents also contain untrusted text. A sentence inside a document telling the assistant to ignore its rules must remain source material, not become a higher-priority instruction. The OWASP RAG Security Cheat Sheet covers these risks across ingestion, indexing and retrieval.
Who owns the truth in the knowledge base?
Even excellent search cannot fix an organisation that has no agreed version of a policy. If three departments publish contradictory guidance, the assistant cannot decide which one has authority merely by comparing embeddings.
Before launch, assign ownership of source material, define versioning rules and decide how quickly updates and removals must reach retrieval. Deleted or newly restricted documents should not survive indefinitely in a derived index.
Users should be able to open the source they are shown, with a meaningful date and reference. If the evidence is inadequate, saying that the answer was not found is better than guessing. A citation does not itself prove correctness: an answer may cite the right document and still misunderstand it.
When retrieval adds more work than value
Not every AI feature needs RAG. A small, stable set of instructions may fit in a verified prompt context. A live product price or stock count is better obtained from a structured API than from yesterday's indexed spreadsheet.
RAG makes sense where answers depend on a substantial collection of changing, largely unstructured knowledge. I would start with real user questions, approved information sources and clear acceptance criteria before choosing a vector database or a framework.
RAG does not turn a disorganised archive into reliable knowledge. It makes document quality, search quality and access governance part of the quality of the AI product.