Local vs OpenAI Embeddings: A 10,000-Document RAG Benchmark

Published April 23, 2026 - 21 min read

Most "local AI" guides skip embeddings, because the marketing line "OpenAI is best, just use them" has gone uncontested for years. The line is wrong. As of April 2026, three open embedding models match or beat OpenAI's text-embedding-3-large on retrieval accuracy in the actual RAG benchmarks I have run, while costing $0 per million tokens, leaking zero documents to a third party, and running comfortably on a $300 GPU.

This is the benchmark I ran to settle the question. 10,000 documents across legal, technical, and clinical domains. Five local embedding models against OpenAI's two flagship offerings. Recall@5, mean reciprocal rank (MRR), latency, cost-per-million-tokens, and the operational pain of each option. Reproducible code, real numbers, no chart inflation.

The headline result up front: BGE-Large-EN-v1.5 and Nomic-Embed-Text-v1.5 each match OpenAI text-embedding-3-large within a 1.5-point margin on standard RAG metrics, at roughly 1/40th the cost and zero data exposure. That is not "good enough." That is parity.

Quick Start: Try Local Embeddings in 60 Seconds

1. ollama pull bge-large (336MB)
2. Use the OpenAI-compatible endpoint at http://localhost:11434/v1/embeddings
3. Swap your openai.embeddings.create call's model from text-embedding-3-large to bge-large and point base_url at Ollama
4. You are now embedding locally. The rest of your RAG pipeline does not need to change.

---

Table of Contents

1. Why Embeddings Matter More Than the LLM
2. The Five Local Embedding Models
3. Benchmark Setup
4. Retrieval Accuracy Results
5. Latency & Throughput
6. Cost Analysis at Scale
7. When OpenAI Still Wins
8. Migration Path: Drop-In Replacement
9. Pitfalls
10. FAQ

---

Why Embeddings Matter More Than the LLM {#why-embeddings}

Most teams obsess over which LLM to use and treat embeddings as an afterthought. This is backwards. In RAG pipelines, the bottleneck on quality is almost always retrieval, not generation. If the right context never reaches the LLM, no model - GPT-4o, Claude Opus, Llama 3.1 70B - can save the answer.

The math is brutal: a RAG system retrieves the wrong chunk 12% of the time, your end-to-end correctness is capped at 88%, regardless of how good your generation model is. Improve retrieval to 96%, and the same generation model gives you 96% end-to-end. Embeddings determine retrieval. Embeddings are the foundation.

Why this hasn't been obvious until recently: until late 2023, OpenAI's text-embedding-ada-002 genuinely was the best general-purpose embedding model. The open-source community has caught up - quietly, methodically - with BGE in late 2023, GTE-Large in early 2024, and Nomic + Stella + mxbai in 2024-2025. Most teams never re-benchmarked.

The MTEB (Massive Text Embedding Benchmark) at huggingface.co/spaces/mteb/leaderboard is the standard reference. As of April 2026, six open models sit in the top 20 - several above OpenAI's text-embedding-3-large.

---

The Five Local Embedding Models {#five-models}

Versus OpenAI:

The OpenAI models are closed-weights, so you cannot run them locally. Cost is for the API only. Throughput is bounded by OpenAI's rate limits (currently 1M tokens/min on tier 5).

---

Benchmark Setup {#benchmark-setup}

This is the test rig:

• Corpus 1: Legal - 3,500 contract clauses from public SEC filings (10-K, 10-Q)
• Corpus 2: Technical - 4,200 chunks from open-source software documentation (Postgres, Kubernetes, FastAPI)
• Corpus 3: Clinical - 2,300 deidentified case study chunks from PubMed open-access articles
• Total: 10,000 chunks, 1,200 evaluation queries with ground-truth correct chunks
• Hardware: RTX 4090 24GB on Ubuntu 22.04, Python 3.11
• Vector store: Qdrant 1.10 with cosine similarity, HNSW index (M=16, ef=128)

For each model and corpus, I:

1. Embedded all chunks
2. Embedded all queries
3. Retrieved top-K (K=5, 10) per query
4. Computed Recall@5, Recall@10, and MRR
5. Recorded embedding latency on a 1,000-document batch

Code is up on the LocalAIMaster GitHub - see the comparable Ollama ChromaDB RAG pipeline walkthrough for the wiring template I used.

---

Retrieval Accuracy Results {#accuracy-results}

Legal Corpus (3,500 clauses, 400 queries)

Technical Corpus (4,200 docs, 500 queries)

Clinical Corpus (2,300 docs, 300 queries)

What These Numbers Mean

• Stella-EN-1.5B-v5 is the highest-quality open embedding model in this benchmark, beating OpenAI text-embedding-3-large on every metric in every corpus.
• GTE-Large-EN-v1.5 comes second and edges out OpenAI text-embedding-3-large on Recall@5 in the technical and clinical corpora.
• BGE-Large-EN-v1.5 is within 1-2 points of OpenAI on every metric. For most teams this is the right default - small (1.3GB), fast, free, and battle-tested.
• Nomic-Embed-Text-v1.5 is the smallest (280MB) and supports Matryoshka truncation - you can store 256-dim vectors at small accuracy cost. Useful when storage is the bottleneck.
• OpenAI text-embedding-3-small is comfortably beaten by every open large-tier model except mxbai. The "small" tier exists primarily as a cheaper option, not a quality tier.

---

Latency & Throughput {#latency}

Embedding 1,000 average-length documents (roughly 250 tokens each). RTX 4090 for local models, OpenAI API from US-East.

The headline:

• Nomic and mxbai are faster than OpenAI's API, even before you add network latency or rate limit pauses.
• Stella is slower but still acceptable for most batch ingestion workloads.
• For online query embedding, all local models are sub-50ms per single query on a 4090, vs ~150-300ms for the OpenAI API round trip.

---

Cost Analysis at Scale {#cost}

Scenario: 1 million documents, 1 million queries/month (e.g., a B2B RAG product with ~33K daily active queries)

Document tokens: ~250M (one-time, but with re-ingestion this happens periodically) Query tokens: ~50M/month

OpenAI text-embedding-3-large:

• Documents: 250M × $0.13/1M = $32.50 (one-time)
• Queries: 50M × $0.13/1M = $6.50/month
• 12-month total: $32.50 + $78 = $110.50

OpenAI text-embedding-3-small:

• Documents: 250M × $0.02/1M = $5.00
• Queries: 50M × $0.02/1M = $1.00/month
• 12-month total: $17.00

Local (any of BGE, GTE, Nomic, Stella, mxbai) on dedicated hardware:

• Hardware: RTX 4060 Ti 16GB ($420 used in April 2026)
• Power: ~140W under load × $0.12/kWh × 24 × 365 = $147/year
• 12-month total: $147 (electricity), or $567 if amortizing the GPU year 1

For most teams the math at small scale favors OpenAI's "small" model, if you ignore privacy. At the scale where embeddings start mattering for budget (10M+ queries/month, 100M+ documents), local wins decisively.

But cost is rarely the actual decision driver. The decision drivers are:

1. Privacy / regulated data: every document and every query becomes prompt history at OpenAI. For HIPAA, GDPR-sensitive, attorney-client, or trade-secret data, local is the only defensible answer.
2. Vendor lock-in: OpenAI deprecated text-embedding-ada-002 in 2024, forcing every customer to re-embed against text-embedding-3. A local model running from a fixed checkpoint is yours forever.
3. Latency floor: OpenAI's API is fast but not free of network round trips. Local embedding at <50ms per query unlocks UX patterns (live search-as-you-type) that OpenAI's API cannot match.

---

When OpenAI Still Wins {#openai-wins}

I am not going to pretend this is one-sided. OpenAI's embedding API is genuinely superior in three specific cases:

1. Multilingual at scale. text-embedding-3-large handles 100+ languages well out of the box. BGE-M3 (multilingual variant of BGE) is the closest open competitor, but for low-resource languages OpenAI still has an edge from its training data scale.
2. Zero infrastructure. If you want one HTTP call and no GPU, OpenAI's API is the answer. The total integration time is 10 minutes; local embedding is 1-2 hours of setup.
3. Long-document semantic search with the 8191-token window. GTE-Large-EN-v1.5 supports 8192 tokens and is competitive, but OpenAI's training data on long documents is broader.

For everything else - English RAG, code search, technical documentation, support knowledge bases, contract retrieval - local has matched or beaten OpenAI for at least 12 months. Most teams just have not re-benchmarked.

---

Migration Path: Drop-In Replacement {#migration}

The pragmatic migration is one line of code if your stack uses the OpenAI Python SDK.

Before (OpenAI)

from openai import OpenAI

client = OpenAI()  # uses OPENAI_API_KEY

resp = client.embeddings.create(
    model="text-embedding-3-large",
    input="What is the capital of France?",
)
vec = resp.data[0].embedding  # 3072-dim

After (Ollama, OpenAI-compatible)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

resp = client.embeddings.create(
    model="bge-large",
    input="What is the capital of France?",
)
vec = resp.data[0].embedding  # 1024-dim

Two changes: base_url and model. Your retrieval and reranking code does not need to change at all.

Re-Embedding Existing Indexes

If you already have a vector store full of OpenAI embeddings, you cannot mix and match - the dimensions and embedding spaces are incompatible. You must re-embed everything against the new model.

import qdrant_client
from openai import OpenAI

old_client = qdrant_client.QdrantClient("localhost", port=6333)
new = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

# Stream documents from old collection, re-embed, upsert to new collection
points, _ = old_client.scroll(collection_name="docs_old", limit=1000, with_payload=True)
for batch_start in range(0, len(points), 64):
    batch = points[batch_start:batch_start + 64]
    texts = [p.payload["text"] for p in batch]
    resp = new.embeddings.create(model="bge-large", input=texts)
    new_vectors = [d.embedding for d in resp.data]
    old_client.upsert(
        collection_name="docs_new",
        points=[{
            "id": p.id,
            "vector": v,
            "payload": p.payload,
        } for p, v in zip(batch, new_vectors)]
    )

For a 1M-document collection on a 4090, expect this to run in roughly 4-6 hours. Not painful. You only do it once.

For broader RAG architecture, our local AI embeddings guide and Ollama ChromaDB RAG pipeline cover the supporting plumbing.

---

Common Pitfalls {#pitfalls}

1. Comparing apples to oranges. Make sure you compare against text-embedding-3-large, not -ada-002 or -3-small. Newcomers benchmark against ada-002 (deprecated) and conclude open is way ahead. Use the current OpenAI flagship.

2. Not normalizing vectors. Most local models output normalized vectors by default. OpenAI also returns normalized vectors. But some embedding pipelines re-normalize and others don't - mismatched normalization across query and document side is a common silent bug.

3. Wrong distance metric. BGE, GTE, and most open models are trained for cosine similarity. Some libraries default to dot product. Cosine and dot are equivalent on normalized vectors but differ if you skip normalization.

4. Mixing embedding models in one index. You cannot run BGE on documents and Nomic on queries - they live in different vector spaces. One model per collection.

5. Skipping the reranker. A small cross-encoder reranker (BGE-reranker-large) on top of any of these models adds 3-5 points of MRR at ~50ms per query. Do not skip it for production.

6. Forgetting query prefixes. BGE family was trained with query prefixes ("Represent this sentence for searching relevant passages: "). Document side is plain. Mixing this up costs ~3-5 points of recall.

7. Benchmarking on a too-small corpus. 100 documents is not a benchmark - all models look identical. Use 5,000+ for meaningful differences.

---

Frequently Asked Questions {#faq}

Are local embeddings really as good as OpenAI's?

For English RAG over 10,000+ documents, yes. As of April 2026, BGE-Large, GTE-Large, Stella, and Nomic all match or exceed OpenAI text-embedding-3-large within margin of error on standard MTEB metrics and the corpus-specific benchmark in this article. The "OpenAI is best" reflex is two years out of date for English.

Which local embedding model should I use?

Start with BGE-Large-EN-v1.5 - it is the practical default. Battle-tested, MIT-licensed, 1.3GB, fast. If you need 8K-token input, switch to GTE-Large-EN-v1.5. If you need maximum quality and have spare GPU time, Stella-EN-1.5B-v5. If storage is your bottleneck, Nomic-Embed-Text-v1.5 with Matryoshka truncation to 256-dim.

How fast are local embeddings vs OpenAI?

On an RTX 4090, BGE-Large embeds 33,800 tokens per second locally, vs OpenAI's text-embedding-3-large at roughly 22,300 tokens/sec including network latency. Nomic and mxbai are faster still. For single-query latency, local is sub-50ms vs 150-300ms for the OpenAI API.

Can I run embeddings on a CPU?

Yes, slowly. BGE-Large on a Ryzen 5 5600X embeds at roughly 600 tokens/sec - usable for batch ingestion of small corpora, painful for query-time. Nomic-Embed-Text-v1.5 is the most CPU-friendly option at ~1,400 tokens/sec on the same CPU. For real-time RAG, plan on a GPU or Apple Silicon.

When does OpenAI still make sense for embeddings?

Three cases: (1) you need 100+ language support out of the box and BGE-M3 is not enough; (2) you have zero infrastructure budget and time-to-integration is everything; (3) you specifically need the 8191-token context window and GTE-Large is not sufficient for your domain. For most English RAG use cases, local matches or beats.

What about hybrid search (BM25 + vectors)?

Hybrid search adds 4-8 points of recall on top of any embedding model, including OpenAI's. The improvement is roughly model-agnostic. If you are running pure vector search, switch to hybrid before agonizing over which embedding model to pick. Qdrant 1.10+, Weaviate, and Elasticsearch all support hybrid out of the box.

Will my data be private with local embeddings?

Yes. Ollama, sentence-transformers, and HuggingFace Transformers all run fully offline once the model is downloaded. You can verify by blocking outbound network from these processes - they will continue to embed normally. The only exception is if you also expose your Ollama server on the network without auth.

How often should I re-benchmark?

Every six months. The open embedding ecosystem has shipped a meaningful improvement roughly every quarter for the last two years. The model that was best in October may not be the best in April. MTEB leaderboard is the canonical reference.

---

The Bottom Line

The "OpenAI is the embeddings standard" line is outdated by at least a year. Three open models (BGE, GTE, Stella) match or beat text-embedding-3-large on retrieval accuracy. Two (Nomic, mxbai) are faster than OpenAI's API for batch ingestion. All five run on a $300 used GPU and never see your documents.

If you are running RAG in production today against the OpenAI API, here is the migration plan: pick BGE-Large-EN-v1.5 as a default, change two lines of code, re-embed your corpus over a weekend, and benchmark against your production query log. Most teams find Recall@5 unchanged or slightly improved, latency improved, and cost cut to electricity. Some teams discover their production retrieval was being silently bottlenecked by the OpenAI API rate limit.

Privacy and cost are real but neither is the strongest argument. The strongest argument is that you stop being a tenant on someone else's deprecation schedule. text-embedding-ada-002 is gone. text-embedding-3 will be deprecated eventually. The model checkpoints you download today will still embed your documents identically in 2030.

That permanence is the local advantage that gets undersold. Run the benchmark on your own data. The result will surprise you.