Ollama Semantic Search: Private Document Search

Published April 23, 2026 · Updated August 23, 2026

Semantic search on Ollama takes three moving parts: an embedding model (ollama pull nomic-embed-text), a vector store to hold the resulting vectors (ChromaDB is the easiest), and about 60 lines of Python to chunk your documents and query them. Everything runs on your own machine, so no document and no query ever reaches a third party. The naive version works surprisingly well; the difference between that and a good search engine is chunk sizing, hybrid retrieval and a reranker — in that order.

A keyword index needs you to remember the exact wording. Semantic search does not: ask for "the legal review of the Acme acquisition" and it finds the right document even when those words appear nowhere in it, because both query and document are represented by their meaning rather than their spelling.

Hosted semantic search does this too. It also bills per document indexed and per query, ships every document to a third-party provider, and is subject to whatever legal process reaches that provider. For legal, healthcare, finance, R&D — anywhere the document contents are the asset — local embeddings remove the bill and the exposure in one move.

Quick Start: ollama pull nomic-embed-text plus the indexer below gives you working semantic search over a folder of PDFs. Everything after that is about making it good.

---

What does semantic search actually do?

A keyword engine indexes the words in each document and matches the words in the query. It is exact when you remember the phrasing and useless when you do not. "Quarterly numbers" misses a document titled "Q3 financials."

Semantic search indexes the meaning of each chunk as a vector — a list of 768 or 1024 floating-point numbers. At query time the query becomes a vector too, and you return the chunks whose vectors sit closest by cosine similarity. "Quarterly numbers" and "Q3 financials" land near each other because the embedding model learned they describe similar things.

Three parts:

1. Indexing pipeline — read documents, split into chunks, embed each chunk with Ollama, store the vector alongside the source text and metadata.
2. Query pipeline — embed the query, find the top-K nearest vectors, return the source chunks.
3. Optional reranker / hybrid layer — re-score the top-K with a slower, more accurate model, or blend in keyword search.

Everything else is implementation detail. The choices that matter are which embedding model, which chunk size, which vector store, and whether you need the reranker.

For the broader use case — chat over these results, team access control, document ingestion pipelines — the private AI knowledge base guide picks up where this one stops.

---

Which Ollama embedding model should I use?

Ollama hosts several embedding models. Three are worth your time; the rest are mostly historical.

The storage column is arithmetic, not a benchmark: dimensions × 4 bytes per float × number of chunks. At 768 dimensions each vector is 3,072 bytes ≈ 3 KB, so a million chunks costs about 3.1 GB. Halving dimensions halves your index — which is the entire argument for all-minilm and the entire argument against reaching for 1024 dimensions by reflex.

nomic-embed-text is the right default. Small, quick, and for most search workloads hard to distinguish from the heavier models once retrieval is tuned.

bge-m3 is the pick for multilingual collections or long documents. It handles 8K input tokens natively and produces dense, sparse and multi-vector representations from one model, which lets you build hybrid retrieval without a separate BM25 index.

mxbai-embed-large is the accuracy-first choice for English-only collections where indexing time does not matter.

For quality rankings, use the maintained scoreboard rather than any figure in a blog post: the MTEB leaderboard tracks embedding models across retrieval tasks and it changes month to month. Shortlist two from it, then run both against your own evaluation set — domain rankings routinely disagree with the global average.

ollama pull nomic-embed-text
ollama pull mxbai-embed-large
ollama pull bge-m3

For local versus hosted embedding quality, the local embeddings vs OpenAI embeddings analysis covers the head-to-head.

---

How big should my chunks be?

Chunking is where homegrown search engines quietly give away recall they never needed to lose.

Rule 1: match chunk size to the model's real limit

Most Ollama embedding models cap around 512 effective tokens regardless of the longer limits in their documentation. Anything past that is silently truncated — no error, just a worse vector. Use 200-400 token chunks for nomic, mxbai and snowflake; 800-1200 for bge-m3.

Rule 2: add overlap

A 300-token chunk with 30 tokens of overlap costs about ten percent more storage and stops queries that straddle a chunk boundary from falling through the gap.

Rule 3: respect document structure

Splitting mid-sentence destroys the semantic signal. Use a recursive splitter that prefers paragraph breaks, then sentence breaks, then word breaks.

from typing import List

def chunk_text(text: str, target_tokens: int = 300, overlap: int = 30) -> List[str]:
    # Approximate tokens as words * 1.3
    target_words = int(target_tokens / 1.3)
    overlap_words = int(overlap / 1.3)

    paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
    chunks, buffer = [], []
    word_count = 0

    for para in paragraphs:
        words = para.split()
        if word_count + len(words) <= target_words:
            buffer.append(para)
            word_count += len(words)
        else:
            if buffer:
                chunks.append("\n\n".join(buffer))
            # Start next buffer with overlap from previous
            tail = " ".join(" ".join(buffer).split()[-overlap_words:])
            buffer = [tail, para]
            word_count = len(tail.split()) + len(words)

    if buffer:
        chunks.append("\n\n".join(buffer))
    return chunks

Rule 4: prepend the document title

Adding the title to the start of every chunk improves retrieval for queries that name the document. f"Document: {title}\n\n{chunk_text}" is the simplest version and costs nothing.

Rule 5: test two or three sizes on your own data

Build a small evaluation set — 50 queries with known-correct chunks — and measure recall@5 at 200, 300 and 500 tokens. The best size for your corpus regularly differs from the common default, and this is the only way to find out. The harness is in the sizing section.

---

ChromaDB, FAISS or pgvector?

For most Ollama projects: ChromaDB to prototype, Qdrant or pgvector for production, FAISS when you need raw speed over millions of vectors and can treat the index as a build artifact.

import chromadb
client = chromadb.PersistentClient(path="./chroma_data")
collection = client.create_collection("docs")

Three method calls and you have a working store — which is why the prototype almost always starts here.

---

How do I build the indexer?

A working end-to-end indexer for a folder of PDFs.

import os
import hashlib
from pathlib import Path
import chromadb
import ollama
from pypdf import PdfReader

EMBED_MODEL = "nomic-embed-text"
CHUNK_TOKENS = 300
OVERLAP_TOKENS = 30

client = chromadb.PersistentClient(path="./chroma_data")
collection = client.get_or_create_collection(
    "docs",
    metadata={"hnsw:space": "cosine"},
)

def extract_text(pdf_path: Path) -> str:
    reader = PdfReader(str(pdf_path))
    return "\n\n".join((p.extract_text() or "") for p in reader.pages)

def chunk_text(text, target=300, overlap=30):
    target_words = int(target / 1.3)
    overlap_words = int(overlap / 1.3)
    paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
    chunks, buffer, count = [], [], 0
    for para in paragraphs:
        words = para.split()
        if count + len(words) <= target_words:
            buffer.append(para); count += len(words)
        else:
            if buffer: chunks.append("\n\n".join(buffer))
            tail = " ".join(" ".join(buffer).split()[-overlap_words:]) if buffer else ""
            buffer = [tail, para] if tail else [para]
            count = len(tail.split()) + len(words)
    if buffer: chunks.append("\n\n".join(buffer))
    return chunks

def index_folder(folder: str):
    folder = Path(folder)
    pdfs = list(folder.rglob("*.pdf"))
    print(f"Indexing {len(pdfs)} PDFs from {folder}")

    batch_texts, batch_ids, batch_meta = [], [], []
    for pdf in pdfs:
        try:
            text = extract_text(pdf)
        except Exception as e:
            print(f"skip {pdf}: {e}"); continue

        title = pdf.stem
        for i, chunk in enumerate(chunk_text(text)):
            doc_id = hashlib.sha1(f"{pdf}::{i}".encode()).hexdigest()[:16]
            payload = f"Document: {title}\n\n{chunk}"
            batch_texts.append(payload)
            batch_ids.append(doc_id)
            batch_meta.append({"path": str(pdf), "title": title, "chunk": i})

            if len(batch_texts) >= 64:
                embed_and_store(batch_texts, batch_ids, batch_meta)
                batch_texts, batch_ids, batch_meta = [], [], []

    if batch_texts:
        embed_and_store(batch_texts, batch_ids, batch_meta)

def embed_and_store(texts, ids, metas):
    result = ollama.embed(model=EMBED_MODEL, input=texts)
    collection.upsert(
        ids=ids,
        embeddings=result.embeddings,
        documents=texts,
        metadatas=metas,
    )
    print(f"  +{len(texts)} chunks indexed")

if __name__ == "__main__":
    index_folder("./documents")

Three details matter. Batching minimises per-request Ollama overhead — 64 is a reasonable starting point, and raising it helps until you run out of memory. The metadata stores the source path and chunk index so results can show context. The cosine HNSW space matches normalised embeddings, which is what nomic-embed-text produces.

The official ChromaDB docs at docs.trychroma.com cover collection sharding, persistence and migrations for larger deployments.

---

How do I serve the search API?

A FastAPI service that accepts a query and returns ranked results.

from fastapi import FastAPI, Query
import chromadb
import ollama

app = FastAPI()
client = chromadb.PersistentClient(path="./chroma_data")
collection = client.get_collection("docs")

@app.get("/search")
def search(q: str = Query(..., min_length=2), k: int = 5):
    query_emb = ollama.embed(model="nomic-embed-text", input=[q]).embeddings[0]
    results = collection.query(
        query_embeddings=[query_emb],
        n_results=k,
        include=["documents", "metadatas", "distances"],
    )
    hits = []
    for doc, meta, dist in zip(
        results["documents"][0],
        results["metadatas"][0],
        results["distances"][0],
    ):
        hits.append({
            "title": meta["title"],
            "path": meta["path"],
            "chunk": meta["chunk"],
            "score": 1 - dist,         # convert cosine distance to similarity
            "preview": doc[:280],
        })
    return {"query": q, "results": hits}

Run with uvicorn search_api:app --port 8090 and curl http://localhost:8090/search?q=quarterly+revenue.

That is already a usable search engine. Everything below makes it better.

---

How do I return relevant results, not just close vectors?

Pure vector search misses exact matches — product codes, ticket numbers, surnames, acronyms — and ranks less reliably than the combination of dense retrieval, sparse retrieval and a reranker.

Step 1: add BM25 keyword search

from rank_bm25 import BM25Okapi
import nltk; nltk.download("punkt", quiet=True)
from nltk.tokenize import word_tokenize

# Build the BM25 index alongside the vector index
all_docs = collection.get(include=["documents"])
tokenised = [word_tokenize(d.lower()) for d in all_docs["documents"]]
bm25 = BM25Okapi(tokenised)
ids = all_docs["ids"]

Step 2: reciprocal rank fusion

def rrf(rankings: list[list[str]], k: int = 60):
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores.items(), key=lambda x: -x[1])

def hybrid_search(q: str, top_k: int = 50):
    # Dense
    q_emb = ollama.embed(model="nomic-embed-text", input=[q]).embeddings[0]
    dense = collection.query(query_embeddings=[q_emb], n_results=top_k)
    dense_ids = dense["ids"][0]

    # Sparse
    bm25_scores = bm25.get_scores(word_tokenize(q.lower()))
    bm25_top = sorted(range(len(bm25_scores)), key=lambda i: -bm25_scores[i])[:top_k]
    sparse_ids = [ids[i] for i in bm25_top]

    return rrf([dense_ids, sparse_ids])[:10]

Reciprocal rank fusion needs no score calibration between the two systems, which is exactly why it is the standard choice — BM25 scores and cosine similarities are not on comparable scales, but their ranks are.

Step 3: cross-encoder reranker (optional)

from sentence_transformers import CrossEncoder
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")

def rerank(q: str, candidates: list[tuple[str, float]]):
    docs = collection.get(ids=[c[0] for c in candidates])["documents"]
    pairs = [[q, d] for d in docs]
    scores = reranker.predict(pairs)
    out = sorted(zip(candidates, scores), key=lambda x: -x[1])
    return [c[0] for c, s in out]

A bi-encoder embeds the query and the document separately and can never compare them directly; a cross-encoder reads both together and scores the pair, which is why it ranks better and why it cannot be precomputed. That is the whole tradeoff: you run it on 50 candidates instead of the whole corpus, and you pay one forward pass per candidate at query time. It is the single largest quality jump available in this pipeline, and the single largest latency cost. Measure both on your own data before deciding — the harness for that is next.

---

How much disk, RAM and time will this need?

Three numbers you can work out before downloading anything.

1. How many chunks will my corpus produce?

chunks = total words / (chunk tokens / 1.3)

At 300-token chunks that is roughly 230 words per chunk. A collection of 40,000 PDFs averaging 5,000 words is 200 million words ÷ 230 ≈ 870,000 chunks.

2. How much will the index cost?

vector bytes = chunks x dimensions x 4 bytes

870,000 chunks at 768 dimensions = 870,000 × 768 × 4 ≈ 2.7 GB of vectors. The HNSW graph adds neighbour links on top — small next to the vectors themselves — and the source text you store alongside is roughly 1.2 KB per 300-token chunk, so about 1 GB more. A corpus most people would call large fits in about 4 GB. Vector storage is rarely the constraint people brace for; it becomes one somewhere past 10 million chunks, which is where product quantization or an on-disk store starts to earn its complexity.

3. Where does query time go?

Query latency is three stages: embedding the query (one forward pass through a small model), the approximate-nearest-neighbour lookup (sublinear in collection size — this is the cheap part, and it stays cheap as you grow), and the optional reranker (one cross-encoder pass per candidate, so 50 candidates costs 50 passes). If your search feels slow, it is almost always the reranker or the disk reads that fetch display text — not the vector math.

Measure it on your own corpus

Generic numbers are worth very little here because latency depends on your hardware and quality depends on your documents. This harness gives you both, on your data:

import time, json, statistics

# eval_set.json: [{"query": "...", "correct_ids": ["abc123", ...]}, ...]
eval_set = json.load(open("eval_set.json"))

def recall_at_k(search_fn, k=5):
    hits, timings = 0, []
    for case in eval_set:
        t0 = time.perf_counter()
        result_ids = search_fn(case["query"])[:k]
        timings.append((time.perf_counter() - t0) * 1000)
        if any(rid in case["correct_ids"] for rid in result_ids):
            hits += 1
    return {
        "recall_at_k": hits / len(eval_set),
        "p50_ms": statistics.median(timings),
        "p95_ms": sorted(timings)[int(len(timings) * 0.95) - 1],
    }

print("dense only:", recall_at_k(dense_search))
print("hybrid:    ", recall_at_k(hybrid_search))
print("+ reranker:", recall_at_k(reranked_search))

Fifty hand-judged queries take an afternoon to assemble and pay for themselves the first time a "obvious improvement" turns out to make retrieval worse. Run the harness after every change to chunk size, embedding model or retrieval strategy — that is the entire difference between tuning and guessing.

For a full RAG pipeline with chat over the retrieved chunks, the Ollama + ChromaDB RAG pipeline is the natural follow-up.

---

What goes wrong?

Pitfall 1: forgetting to normalise vectors

Symptom: cosine similarity returns wildly inconsistent scores.

Cause: ChromaDB with hnsw:space=cosine expects unit vectors. Some embedding models normalise their output; some do not.

Fix: either use hnsw:space=l2 (works regardless), or normalise explicitly:

import numpy as np
arr = np.array(embeddings)
arr = arr / np.linalg.norm(arr, axis=1, keepdims=True)

Pitfall 2: PDF extraction producing garbage

Symptom: results are fragments of footers, page numbers and running headers.

Cause: pypdf and pdfplumber extract per-page text, headers and all.

Fix: strip the top and bottom ten percent of each page, or use a layout-aware extractor such as Unstructured.io or Marker.

Pitfall 3: indexing the same document twice

Symptom: the top five results are five copies of the same chunk.

Cause: re-running the indexer without deduplication.

Fix: upsert() instead of add() so re-indexing replaces rather than duplicates. Stable, content-derived IDs make this work.

Pitfall 4: embedding model mismatch

Symptom: relevance collapses after a "small" change.

Cause: indexed with nomic-embed-text, querying with mxbai-embed-large. Vectors from different models are not comparable — not slightly worse, meaningless.

Fix: store the embedding model name in collection metadata and refuse to query when it does not match.

Pitfall 5: no evaluation set

Symptom: you cannot tell whether a change improved or regressed quality.

Cause: no held-out queries with known-correct answers.

Fix: the harness above. Without it, every "improvement" is a guess with a confident tone.

For server-side problems underneath all of this, the Ollama troubleshooting guide covers the failure modes that are not about search at all.

---

Common questions

Which Ollama embedding model is best for semantic search? For English content in chunks under 512 tokens, nomic-embed-text v1.5 is the sensible default — 768 dimensions, small, quick. For multilingual or long-document work, bge-m3 supports 8192 tokens and three retrieval modes in one model. mxbai-embed-large trades indexing speed for 1024-dimensional vectors on English text. Check the MTEB leaderboard for current quality rankings, then confirm on your own corpus.

How big should my chunks be? 200-400 tokens with 20-50 tokens of overlap for nomic, mxbai and snowflake; 800-1200 for bge-m3's longer context. Short Q&A retrieval prefers smaller chunks, long-document summarisation prefers larger. Evaluate on 50-100 held-out queries before committing — this is the highest-leverage half hour in the whole build.

ChromaDB, FAISS or pgvector? ChromaDB for prototyping and collections under about 5M vectors. FAISS for raw speed at 10M+ vectors where an index file you can ship as an artifact is useful. pgvector when you already run Postgres and want vectors next to relational data — HNSW and IVFFlat indexes make it competitive well into the tens of millions. Qdrant and Weaviate are solid too, especially when you need rich metadata filtering.

How do I get relevant results rather than merely close vectors? Three additions, in order of payoff per effort. Hybrid search: fuse BM25 keyword ranks with vector ranks using reciprocal rank fusion, so exact-match queries stop failing. Reranking: take the top 50 vector hits and re-score them with a cross-encoder such as BAAI/bge-reranker-v2-m3, keep the top 5. Metadata filtering: prefilter by document type, date or author to shrink the candidate space before search runs at all.

Can Ollama embeddings run on CPU only? Yes. Embedding models are far smaller than chat models — nomic-embed-text is 137M parameters, mxbai-embed-large 335M — so CPU inference is practical. Indexing is the throughput-hungry phase and it is a one-off; queries embed a single short string. If you continuously index a large, changing document store, a GPU shortens the ingest window considerably; for a mostly-static corpus, CPU is fine.

How do I keep the index in sync when documents change? Store a content hash with each chunk and re-embed only chunks whose source hash changed. The more robust variant also stores a document version ID and runs a daily reconcile that diffs source against index. Avoid full re-indexing: its cost scales with the whole corpus, while incremental cost scales with what actually changed.

How private is "private semantic search"? If Ollama runs on your hardware, your network and your storage, no document content leaves your control and no query touches a third-party API. That is a categorical difference from hosted vector search, where every document and every query is processed by an external provider. For legal, healthcare and finance work it is usually the only architecture that clears compliance review without additional contractual controls.

What query latency should I expect? It depends entirely on your hardware and pipeline, so measure it with the harness above rather than trusting a number from someone else's machine. Structurally: query embedding is one pass through a small model, ANN lookup is sublinear in corpus size and stays cheap as you grow, and a cross-encoder reranker over 50 candidates is 50 forward passes and will dominate everything else. Dense-only search is comfortably interactive on ordinary hardware; adding a reranker is the change you will actually feel.

---

Final notes

A working private semantic search engine is a weekend. A good one — hybrid, reranked, with an evaluation set and an indexer that survives dirty PDFs — is two or three weeks. Both are less work than negotiating a data processing agreement with a vendor, and both leave you owning the result.

Pull nomic-embed-text. Index a folder. Query it from the FastAPI service. Then add BM25, then the reranker, then the evaluation set — in that order, measuring after each step so you know which changes actually helped.

The hosted semantic search market exists because building this used to require ML expertise and dedicated infrastructure. Ollama, a vector store and 200 lines of Python removed that moat. The interesting question is no longer how to build it — it is what to point it at.