Design a RAG assistant over company documents
A chat assistant that answers employees' questions from the company's own documents, with sources.
Why interviewers ask it
It's the most common AI-engineering design question: it tests retrieval, prompting, permissions, evaluation and cost control together.
Functional requirements
- Ingest documents (PDFs, wiki pages, tickets) and keep them up to date.
- Answer questions with citations, and say 'I don't know' when the answer isn't there.
- Respect document permissions.
Non-functional requirements
- Answers in a few seconds, streamed.
- Accurate and grounded, with measurable quality.
- Cost per question under control; secure.
From version 1 to the final design
- Version 1: embed all the chunks, search them in memory with cosine similarity, and put the top 5 into a prompt.
- Version 2: a vector store, an ingestion pipeline and citations.
- Version 3: hybrid search, re-ranking, permission filters and question rewriting.
- Version 4: evaluation sets, tracing, caching, model routing and guardrails.
The design, step by step
- Ingestion: parse → clean → split into chunks with some overlap → embed → store in a vector store with permission metadata, triggered whenever a document changes.
- Query: rewrite the question using the conversation → hybrid retrieval (vectors plus keywords), filtered by the user's permissions → re-rank → build a prompt with the top chunks → the LLM streams an answer with citations.
- Guardrails: check retrieved text for prompt injection, and check the output.
- Observability: trace every step; run evaluations on a golden set; collect user feedback.
Deep dives
- Permissions are enforced in the retrieval filter; never rely on the LLM to hide data.
- Chunking and re-ranking usually bring the biggest quality gains.
- Evaluation: retrieval recall@k plus answer faithfulness on a golden set, run as regression tests for every change.
- Cost and latency: cache frequent answers; send easy questions to a smaller model.
Common mistakes
- No evaluation, so changes are judged by feel.
- Handling permissions by asking the LLM not to reveal things.
- Never updating the index when documents change.
Not true: Permissions: telling the LLM not to reveal restricted documents protects them.
In fact: A model can be talked into revealing what it has been shown. Filter by permission at retrieval, so that restricted text never reaches the model.
In the interview
Two pipelines. Ingestion parses, chunks and embeds documents into a vector store with permission metadata, updated whenever a document changes. At query time we rewrite the question, run hybrid retrieval filtered by the user's permissions, re-rank, and give the top chunks to the LLM, which streams a cited answer. Every request is traced, and quality is tracked against a golden evaluation set.
Part of Unit 9: AI and ML system design in the free System Design course.