A retrieval-augmented generation prototype can look convincing after a few documents and a handful of questions. The difficult engineering begins when users ask questions that are incomplete, permissions differ, and the documents disagree.
RAG connects a model to retrieved context. Whether that context is appropriate is a separate question.
Treat the source as part of the product
A document needs more than extracted text. Record where it came from, who can access it, when it was updated, and whether someone considers it authoritative.
In Narravo, company knowledge carries sources and review states. Candidate knowledge is distinguished from approved knowledge. That design addresses a product requirement: a suggestion should not become a company fact simply because a model extracted it.
Other products may need different structures. A policy library might require an effective date and policy owner; a customer support system might distinguish public guidance from internal notes.
Enforce access before generation
Do not retrieve restricted passages, place them in a prompt, and then ask the model not to reveal them. The model should receive only the information the requesting user is permitted to access.
Evaluate that boundary directly. Use users with different roles and questions whose relevant answers are in restricted documents. An attractive answer is a failure if it exposes unauthorized information.
Tenant isolation and document permissions belong in the data and retrieval layers, not just in interface visibility.
Test retrieval independently
If the right passage never reaches the model, adjusting the prompt cannot reliably repair the answer.
Create representative questions and identify the sources needed to answer them. Examine whether retrieval finds those sources, whether it returns excessive unrelated text, and whether relevant material is displaced by superficially similar content.
Vector search is one tool. Structured filtering, conventional text search, and graph relationships can also be useful. Choose the combination based on the content and the questions.
Make citations inspectable
A citation should help a user verify the statement. Link an answer to the actual source passage and show enough context for its meaning to be checked.
If the retrieved sources support only part of a question, the answer should preserve that limitation. If sources conflict, expose the disagreement instead of silently merging them into a confident assertion.
The portfolio's guided RAG demonstration uses fictional documents so visitors can inspect that behavior without submitting private information.
Plan for no answer
Define the response when evidence is absent, stale, or inaccessible. That may be a request for clarification, an explanation of missing evidence, or escalation to an owner.
A product that always generates an answer can appear successful while concealing retrieval failures. A clear inability to establish a fact is often the more useful result.
Separate the evaluations
Review at least four dimensions:
- Retrieval: did the system find the necessary material?
- Permissions: was every retrieved source available to this user?
- Grounding: are the answer's claims supported by that material?
- Usefulness: can the user complete the intended task?
Keep examples versioned so ingestion, retrieval, and model changes can be compared. Add questions discovered in real use rather than relying only on the original demonstration set.
For a project-specific starting point, see RAG and enterprise knowledge consulting.