Build a Local RAG Pipeline: Ollama + ChromaDB Step-by-Step

Published on April 23, 2026 • 19 min read

RAG systems fail in the same four boring places, and none of them are the LLM: how the documents get chunked, which embedding model does the lookup, whether retrieval quality is measured at all, and whether you know when to stop fixing the prompt and start fixing the data. Get those wrong and you get a system that answers wrong questions fluently. Get them right and an 8B model on a laptop becomes genuinely useful.

This guide is a build order for getting them right. Ollama for the LLM, ChromaDB for the vector store, sentence-transformers or nomic-embed-text for embeddings — every component runs on your machine. No OpenAI key. No vendor lock-in. No data leaving your laptop.

The result: a working private RAG system you can deploy as an internal API in about 90 minutes.

---

Quick Start: A Working RAG in 5 Commands

# 1. Install Ollama and pull models
brew install ollama
ollama pull llama3.1:8b           # the LLM
ollama pull nomic-embed-text       # embeddings model

# 2. Install Python deps
pip install chromadb llama-index llama-index-llms-ollama \
  llama-index-embeddings-ollama langchain-text-splitters

# 3. Drop your documents into ./docs/
mkdir docs && cp ~/Downloads/*.pdf docs/

# 4. Run the indexer (script provided below)
python rag_index.py

# 5. Ask a question
python rag_query.py "What is our company's PTO policy?"

That works. It is not production-quality yet — but it answers questions about your own documents, fully offline, in five commands. The rest of this guide explains what to fix before you trust it for anything important.

---

What RAG Actually Is (and What It Is Not)

RAG — Retrieval Augmented Generation — is three boring components in a trench coat:

1. A retriever that finds relevant document chunks for a question.
2. A vector store that holds embeddings of your chunks.
3. An LLM that reads the retrieved chunks and writes an answer.

That's it. The magic is in the joins, not the components.

What RAG is not: a magic way to get the LLM to "know" your documents. The LLM still hallucinates. It just hallucinates less because it sees relevant text in its context window. The retrieval quality is the ceiling on the whole system.

User question
   │
   ▼
[ Retriever ] ──► [ Embeddings ] ──► [ ChromaDB ]
                                          │
                              top-k chunks│
                                          ▼
                                  [ Ollama LLM ]
                                          │
                                          ▼
                                       Answer

If the retriever returns garbage, the LLM produces a confident, well-written answer to the wrong question. This is the failure mode nearly every broken RAG system shares.

---

Architecture: Why These Components

You can swap the vector store for Qdrant, Weaviate, or pgvector. ChromaDB wins for local development because it persists to a single directory and requires no infrastructure. For team-scale deployments Qdrant scales further, but most internal RAG projects never outgrow ChromaDB.

---

Step 1: Choose the Right Embedding Model

This is the most underrated decision in any RAG system. Your embedding model determines whether the right chunk gets retrieved.

My recommendations:

• For English-only documents: nomic-embed-text (fast, 768-dim, very high recall on standard benchmarks)
• For multilingual or technical documents: bge-m3
• For prototypes and tiny corpora: all-MiniLM-L6-v2 (fast, but lower recall)

Pull the embedding model into Ollama:

ollama pull nomic-embed-text

Verify dimensions:

import requests
r = requests.post("http://localhost:11434/api/embeddings",
    json={"model": "nomic-embed-text", "prompt": "test string"})
print(len(r.json()["embedding"]))   # 768

Match this dimension when you create your ChromaDB collection. Mismatched dims is the source of half the "RAG returns nothing" bug reports.

---

Step 2: Chunking — The Most Important Decision

If you only get one thing right in your RAG system, get chunking right.

Bad chunking — splitting on arbitrary character counts — destroys semantic boundaries and gives the retriever incoherent fragments. Good chunking respects the structure of your documents.

Chunking strategy by document type:

A working chunker for mixed text:

# rag_chunker.py
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n\n", "\n", ". ", " ", ""],
    length_function=len,
    is_separator_regex=False,
)

def chunk(text: str, metadata: dict) -> list[dict]:
    chunks = splitter.split_text(text)
    return [
        {"text": c, "metadata": {**metadata, "chunk_idx": i, "len": len(c)}}
        for i, c in enumerate(chunks)
    ]

The 200-character overlap matters more than people realize. Without overlap, a question whose answer spans a chunk boundary is unanswerable.

For PDFs specifically, prefer open-parse or Docling over PyMuPDF when you need table preservation. PDF chunking is its own rabbit hole — bad PDF parsing is the second-most common cause of bad RAG.

---

Step 3: Build the Index

Here is the full indexer. It loads documents, chunks them, embeds with Ollama, and persists to ChromaDB.

# rag_index.py
import chromadb
import os
from pathlib import Path
from chromadb.utils.embedding_functions import OllamaEmbeddingFunction
from langchain_text_splitters import RecursiveCharacterTextSplitter
import pypdf

DB_DIR = "./chroma_db"
DOCS_DIR = "./docs"
COLLECTION_NAME = "kb"
OLLAMA_URL = "http://localhost:11434"
EMBED_MODEL = "nomic-embed-text"

client = chromadb.PersistentClient(path=DB_DIR)

embed_fn = OllamaEmbeddingFunction(
    url=f"{OLLAMA_URL}/api/embeddings",
    model_name=EMBED_MODEL,
)

collection = client.get_or_create_collection(
    name=COLLECTION_NAME,
    embedding_function=embed_fn,
    metadata={"hnsw:space": "cosine"},
)

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000, chunk_overlap=200,
    separators=["\n\n", "\n", ". ", " ", ""],
)

def read_pdf(path: Path) -> str:
    reader = pypdf.PdfReader(path)
    return "\n\n".join(p.extract_text() for p in reader.pages if p.extract_text())

def read_text(path: Path) -> str:
    return path.read_text(encoding="utf-8", errors="ignore")

READERS = {".pdf": read_pdf, ".md": read_text, ".txt": read_text}

def index_all():
    docs_added = 0
    for path in Path(DOCS_DIR).rglob("*"):
        if path.suffix.lower() not in READERS:
            continue
        text = READERS[path.suffix.lower()](path)
        chunks = splitter.split_text(text)
        ids = [f"{path.stem}-{i}" for i in range(len(chunks))]
        metas = [{"source": str(path), "chunk": i} for i in range(len(chunks))]
        collection.upsert(documents=chunks, ids=ids, metadatas=metas)
        docs_added += len(chunks)
        print(f"Indexed {path.name}: {len(chunks)} chunks")
    print(f"\nDone. {docs_added} chunks total.")

if __name__ == "__main__":
    index_all()

Run it:

python rag_index.py

Index size on disk for ChromaDB: roughly 10 KB per chunk including the embedding vector. A 100,000-chunk index is about 1 GB.

---

Step 4: Build the Query Service

# rag_query.py
import chromadb
import sys
import requests
from chromadb.utils.embedding_functions import OllamaEmbeddingFunction

DB_DIR = "./chroma_db"
COLLECTION_NAME = "kb"
EMBED_MODEL = "nomic-embed-text"
LLM_MODEL = "llama3.1:8b"
TOP_K = 5

client = chromadb.PersistentClient(path=DB_DIR)
embed_fn = OllamaEmbeddingFunction(
    url="http://localhost:11434/api/embeddings",
    model_name=EMBED_MODEL,
)
collection = client.get_collection(COLLECTION_NAME, embedding_function=embed_fn)

PROMPT_TEMPLATE = """You are a careful assistant. Answer ONLY using the context below.
If the answer is not in the context, say "I do not know based on the provided documents."

CONTEXT:
{context}

QUESTION: {question}

ANSWER:"""

def ask(question: str) -> dict:
    result = collection.query(query_texts=[question], n_results=TOP_K)
    chunks = result["documents"][0]
    sources = [m["source"] for m in result["metadatas"][0]]

    context = "\n\n---\n\n".join(
        f"[Source: {s}]\n{c}" for s, c in zip(sources, chunks)
    )

    response = requests.post(
        "http://localhost:11434/api/generate",
        json={
            "model": LLM_MODEL,
            "prompt": PROMPT_TEMPLATE.format(context=context, question=question),
            "stream": False,
            "options": {"temperature": 0.1, "num_ctx": 4096},
        },
        timeout=120,
    )
    answer = response.json()["response"].strip()
    return {"answer": answer, "sources": list(set(sources))}

if __name__ == "__main__":
    q = " ".join(sys.argv[1:]) or "What is in these documents?"
    out = ask(q)
    print(f"\nANSWER:\n{out['answer']}\n\nSOURCES:")
    for s in out["sources"]:
        print(f"  - {s}")

Run it:

python rag_query.py "What does the security policy say about laptop encryption?"

Output includes the answer plus the source files used. Source attribution is mandatory in any RAG system you let anyone trust.

---

Step 5: Wrap It in a FastAPI Service

For internal team use, expose the query layer as an HTTP API:

# rag_api.py
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from rag_query import ask

app = FastAPI(title="Local RAG")

class Query(BaseModel):
    question: str
    top_k: int | None = 5

@app.post("/api/ask")
def api_ask(q: Query):
    if not q.question.strip():
        raise HTTPException(400, "empty question")
    return ask(q.question)

@app.get("/api/health")
def health():
    return {"status": "ok"}

pip install fastapi uvicorn
uvicorn rag_api:app --host 0.0.0.0 --port 8000

Now curl -X POST http://localhost:8000/api/ask -H 'Content-Type: application/json' -d '{"question":"..."}' is your private knowledge endpoint. Any client can sit in front of it — one popular option is to build a Telegram bot with local AI so the team can query the index from their phones without exposing the box to the internet.

---

Step 6: Evaluate Retrieval Quality

If you cannot measure your RAG system, you cannot improve it. Build a tiny evaluation harness with a hand-curated test set.

# rag_eval.py
import json
from rag_query import ask, collection

# Hand-curated test cases
EVAL = [
    {
        "question": "What is our PTO policy for new hires?",
        "must_contain": ["10 days", "first year"],
        "must_cite": ["hr_handbook.pdf"],
    },
    {
        "question": "Who is the security incident lead?",
        "must_contain": ["security@", "incident"],
        "must_cite": ["security_policy.pdf"],
    },
]

def evaluate():
    pass_count = 0
    for case in EVAL:
        out = ask(case["question"])
        ans = out["answer"].lower()
        srcs = " ".join(out["sources"]).lower()

        contain_ok = all(s.lower() in ans for s in case["must_contain"])
        cite_ok = all(s.lower() in srcs for s in case["must_cite"])

        ok = contain_ok and cite_ok
        pass_count += int(ok)
        status = "PASS" if ok else "FAIL"
        print(f"[{status}] {case['question']}")
        if not ok:
            print(f"  Answer: {out['answer'][:200]}")
            print(f"  Sources: {out['sources']}")
    print(f"\n{pass_count}/{len(EVAL)} passed")

if __name__ == "__main__":
    evaluate()

This is not fancy. It is also the difference between a RAG you tune by guessing and one you tune by measuring. Aim for 80%+ pass rate before deploying to a team.

For deeper RAG evaluation, look at RAGAS for automated metrics like faithfulness and context precision. RAGAS works with local LLMs as the judge model — useful for fully self-hosted eval.

---

Step 7: Tune Retrieval — The Knobs That Matter

After your first eval run, you will see failures. Tune in this order — cheapest and highest-leverage first:

1. Top-k. Default is 5. For dense, narrowly scoped corpora, 3 may be enough. For diverse corpora, try 8-10.

2. Chunk size. If the LLM keeps saying "context insufficient," your chunks are too small. If retrieval brings back irrelevant content, they are too big.

3. Hybrid search. Pure vector search misses exact-match keywords (names, IDs, codes). Combine with BM25:

# Add BM25 alongside vector retrieval
pip install rank-bm25
from rank_bm25 import BM25Okapi
# tokenize chunks, build BM25 index, return weighted union of vector + BM25 top-k

4. Re-ranking. After retrieving top-20 with vectors, re-rank with a cross-encoder (e.g., mixedbread-ai/mxbai-rerank-large-v1) and keep top-5. This is usually the largest single retrieval win available after chunking, and the reason is structural: a bi-encoder compares two vectors that were computed without ever seeing each other, while a cross-encoder reads the query and the candidate chunk together and scores the pair. It costs one extra model pass over 20 candidates. Measure it on your own eval set — that is what the harness in the previous step is for.

5. Metadata filtering. ChromaDB supports where filters. Filter by document type, date, or department before vector search. Smaller search space = better results.

collection.query(
    query_texts=[q],
    n_results=5,
    where={"department": "engineering"},
)

6. Prompt engineering. Move "ground answers in context" instructions higher. Add: "If the question requires a number or date, quote it directly from the source."

---

Speed Ceilings: How Fast Can This Be?

Query latency in a local RAG is almost entirely the LLM's decode phase — retrieval is a millisecond-scale index lookup by comparison — and decode speed is bounded by memory bandwidth, because producing each token requires streaming the whole model through the compute units once. That gives you a ceiling you can compute before buying anything:

Ceiling in tokens/sec = memory bandwidth / weight bytes

At Q4_K_M, weights run about 0.6 GB per billion parameters. Applying that to the two models most people run here, using published card bandwidths:

These are arithmetic upper bounds, not measured throughput. Real output lands well below the ceiling once prompt prefill, sampling, and the KV cache for your retrieved context are accounted for — and a RAG prompt carries five chunks of context, so prefill is not free. Look up your own chip or card's bandwidth (Apple Silicon spans roughly 100 GB/s on base chips to 800 GB/s on Ultra parts; the RTX 4090 is 1008 GB/s) and divide.

The rest of the pipeline is cheap next to generation. Embedding a query is one forward pass through a 137 MB model. ChromaDB's HNSW index is a sub-linear lookup rather than a scan, so search cost grows slowly with corpus size. Indexing is a one-time batch cost, dominated by PDF parsing and embedding throughput rather than by the vector store.

Cloud RAG on a frontier model will usually win on raw latency. What it costs you is a per-query bill that never stops, and the fact that every chunk you retrieve is also a chunk you upload. Local is free after the hardware.

For our cost analysis, see Ollama vs ChatGPT API cost breakdown.

---

The 12 Mistakes That Break Local RAG

1. Wrong embedding dimension. Pull the model, hardcode the dim, or read it dynamically. Mismatch = silent wrong answers.
2. Tiny chunk overlap. 0% overlap loses cross-boundary answers. 200 chars is the sweet spot.
3. No source attribution. Users will not trust answers without citations. Bake source IDs into every response.
4. Mixing chunk sizes across formats. PDFs and code need different splitters. One-size-fits-all hurts retrieval.
5. Skipping evaluation. Without an eval set you are tuning by vibes.
6. Trusting the LLM to refuse. Add a "I do not know" instruction in the prompt. Local LLMs often confabulate without it.
7. Forgetting hybrid search. Vector-only misses literal IDs, names, and codes.
8. Indexing duplicates. Use upsert with stable IDs; never let duplicate chunks accumulate.
9. No incremental re-index. Build a "watch documents folder" daemon or your index goes stale.
10. Ignoring token budgets. Llama 3.1 8B has 128K context, but practical accuracy drops above 16K. Truncate context.
11. Letting users dump huge questions. Long, multi-part questions confuse retrieval. Split or pre-rewrite.
12. No backups. Back up the chroma_db directory. Rebuilding from raw documents takes hours on a real corpus.

---

Going to Production: Hardening Checklist

If you are about to deploy this internally, walk through these:

• Stable doc IDs and incremental re-indexing
• Source attribution in every response
• Eval set of 20-50 questions with expected answers
• Re-ranker enabled (top-20 vector → top-5 cross-encoder)
• Hybrid search (vector + BM25)
• Metadata filtering wired to your document taxonomy
• Backups of the ChromaDB persistent directory
• FastAPI behind authentication (keys, OIDC, or mTLS)
• Rate limiting (60 req/min per user)
• Audit logging of every question and answer
• Health check endpoint and Prometheus metrics
• Run the full pipeline against Ollama production deployment checklist

For the auth, monitoring, and audit pieces, our securing Ollama guide covers the patterns we use.

---

What I Would Build Next

The pipeline above is solid for single-tenant team RAG. Three additions worth considering:

1. Streaming responses. Switch stream: True in the Ollama call and yield chunks via FastAPI's StreamingResponse. Massive UX improvement. If you want a browser front-end on top of it, Ollama + the Vercel AI SDK gives you token-by-token streaming in a web app with very little glue code.

2. Conversation memory. Store chat history per user and feed last 2-3 turns plus retrieved context. The model gives much better follow-up answers.

3. Tool calling. Let the model trigger live searches, calendar lookups, or SQL queries when retrieval is insufficient. See our Ollama function calling guide.

For the broader RAG ecosystem context, the canonical reference is the ChromaDB official documentation. For embedding model evaluation, the MTEB leaderboard is the most rigorous public benchmark.

---

Closing Thoughts

It is tempting to spend the first week choosing an LLM. That is the wrong week. A RAG answer is mostly a restatement of the text you handed the model, so if the right chunk is in the context almost any competent model can produce the answer, and if it is not, no amount of model size rescues you. The whole game is the retrieval pipeline — embeddings, chunking, hybrid search, re-ranking. The LLM is the last part of the work, not the first.

If you take one thing away: build the eval harness before you tune anything. Without measurement you will spend weeks "improving" your RAG with no idea whether it is getting better or worse.

The code in this guide is a complete starting point. Drop your documents in ./docs/, run the indexer, and you have a private RAG that runs without ever calling out to the internet.