Build a Minimal RAG System

What RAG does, in one sentence

A RAG (Retrieval-Augmented Generation) system lets an LLM answer questions about your documents — files it never saw in training — by fetching the relevant text at question time and handing it to the model as context. It's how "chat with your PDFs," support bots, and internal search assistants are built. Here's the whole pipeline, minimal but complete.

The four steps

Every RAG system, however fancy, is these four moves:

1. Chunk your documents into passages of a few hundred tokens.
2. Embed and store each chunk as a vector (in a vector database).
3. Retrieve the chunks most similar to the user's question.
4. Answer using those chunks as context.

Steps 1–2 happen once, upfront (indexing). Steps 3–4 run on every question.

Retrieve, then answer

The heart of the system is the query-time half — find the relevant chunks, stuff them into the prompt, and ask:

# 3 + 4: retrieve then answer
chunks = retrieve(query, k=3)          # nearest stored chunks
context = "\n\n".join(chunks)
prompt = f"Use ONLY this context to answer.\n\nContext:\n{context}\n\nQ: {query}"
answer = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": prompt}],
).choices[0].message.content

retrieve() embeds the question and returns the top-k nearest chunks from your store; you join them into a context block and ask the model to answer from it.

The golden rule

Always instruct the model to answer only from the provided context and to say "I don't know" when the answer isn't there. That one sentence in your prompt prevents the majority of hallucinations in RAG apps — without it, the model happily fills gaps from memory.

Where RAG goes wrong (and the fix)

• Bad retrieval — if the right chunk isn't fetched, no model can answer. Improve chunking or your embedding model first.
• Too much context — dumping 20 chunks buries the answer in noise; three to five focused chunks usually beats more.
• No grounding instruction — without "use only this context," you lose RAG's whole reliability benefit.

The takeaway

RAG = chunk, embed, retrieve, answer — with a strict "only from the context" instruction holding it together. Master this minimal loop and every production RAG system is just a more polished version of it.

Try it now: Index a handful of paragraphs from one document, then ask a question whose answer is in exactly one of them. Watching the right chunk get retrieved and grounded into the answer is the moment RAG clicks.