Build a Fully Local RAG with Ollama

Everything on your machine

Combine a local LLM (Ollama), local embeddings, and a local vector store (Chroma) for RAG with zero external calls — ideal for sensitive documents.

import ollama
emb = ollama.embeddings(model="nomic-embed-text", prompt=chunk)["embedding"]
answer = ollama.chat(model="llama3.2", messages=[
    {"role": "user", "content": f"Context:\n{ctx}\n\nQ: {q}"}])

Tradeoffs

Local models are smaller, so tune expectations and chunking. But for privacy, offline use, and zero per-token cost, a local RAG stack is hard to beat — and surprisingly capable for focused domains.