Local AI Embeddings: Models, APIs, and Integration
Want to go deeper than this article?
Free account unlocks the first chapter of all 25 courses — RAG, agents, MCP, voice AI, MLOps, real GitHub repos.
Go from reading about AI to building with AI 25 structured courses. Hands-on projects. Runs on your machine. Start free.
Published on April 23, 2026 • 18 min read
When a retrieval pipeline returns junk, the embedding model is one of the first things to check. Get the embedding wrong and no amount of clever prompting at the LLM stage can rescue the answer, because the right passages never reach the model. Get it right and a small local model can answer well from the passages it is given.
This guide skips the theory and walks straight into four local embedding models worth knowing, the API patterns for serving them, and the integration recipes that move recall. Quality scores below come from each model's publisher and the public MTEB leaderboard, with the source named; we have not re-run them.
Quick Start: Run Local Embeddings in 4 Minutes
# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
# Pull the workhorse embedding model (137M params, 274 MB)
ollama pull nomic-embed-text
# Generate an embedding via the REST API
curl http://localhost:11434/api/embed -d '{
"model": "nomic-embed-text",
"input": "Local embeddings keep RAG private."
}'
You now have a 768-dimensional vector representing that sentence. At 274MB the model is light enough for a CPU; on any recent GPU, embedding short texts is fast enough that the vector search, not the embedding, is rarely your bottleneck.
Reading articles is good. Building is better.
Free account = the first chapter of all 25 courses, with a per-chapter AI tutor. No card.
Table of Contents
- Why Local Embeddings Matter
- Four Models Worth Knowing
- MTEB Benchmark Comparison
- Embedding Dimension and Storage Math
- API Patterns: Ollama, Infinity, TEI
- Integration with Vector Stores
- End-to-End RAG Pipeline
- Common Pitfalls and Fixes
- FAQs
Why Local Embeddings Matter
OpenAI reported an MTEB average of 64.6 for text-embedding-3-large when it launched (January 2024). Mixedbread reports 64.68 for the open mxbai-embed-large-v1, and Jina reports 65.52 on English MTEB for jina-embeddings-v3. On the publishers' own numbers, the accuracy gap that justified cloud embeddings has closed.
What does still exist:
- Privacy: every chunk you embed in the cloud is content you sent to a third party.
- Cost: OpenAI lists text-embedding-3-large at $0.13 per million tokens. At roughly 1,000 tokens per chunk, 100M chunks is 100B tokens, or about $13,000. Locally, the cost is the electricity for however many GPU hours your measured embedding rate implies.
- Latency: every cloud embedding call includes a network round trip; a local model on the same machine as your retriever does not, so per-query latency is set by your own hardware.
- Determinism: local models are pinned. Cloud models silently change behind the same name.
If you ship anything that processes customer documents, legal contracts, medical records, or proprietary code, local embeddings are no longer the "compromise" path. They are the default.
For a side-by-side of the trade-offs, see our local vs OpenAI embeddings deep dive.
Four Models Worth Knowing
Four open models cover most local RAG needs. Each fills a different slot.
1. nomic-embed-text (the daily driver)
- Parameters: 137M
- Dimensions: 768
- Context window: 8192 tokens
- Disk size: 274 MB
- License: Apache 2.0
This is the sensible default when nothing fancy is required. At 274MB it runs comfortably on a CPU, and it holds its own on English retrieval for its size (Nomic reports an MTEB average of 62.28 for v1.5). Pull it with ollama pull nomic-embed-text.
2. bge-m3 (multilingual king)
- Parameters: 567M
- Dimensions: 1024
- Context window: 8192 tokens
- Disk size: 1.1 GB
- License: MIT
Built by BAAI. Supports dense, sparse, and multi-vector retrieval out of one model. If your corpus mixes English, Chinese, Spanish, German, Arabic, or Hindi, it is the strongest open choice in this list; BAAI reports its results on multilingual retrieval benchmarks such as MIRACL and MKQA rather than a single MTEB average.
ollama pull bge-m3
3. mxbai-embed-large-v1 (English specialist)
- Parameters: 335M
- Dimensions: 1024
- Context window: 512 tokens
- Disk size: 670 MB
- License: Apache 2.0
When the corpus is purely English and short documents (chat logs, support tickets, product reviews), it is a strong pick. Mixedbread reports an MTEB average of 64.68. It supports Matryoshka truncation and binary quantization, so you can cut dimensions to 512 to shrink the index; check the recall cost on your own data.
ollama pull mxbai-embed-large
4. jina-embeddings-v3 (long-context champion)
- Parameters: 570M
- Dimensions: 1024 (Matryoshka, can truncate to 256)
- Context window: 8192 tokens
- License: CC-BY-NC 4.0 (research) or commercial via Jina AI
When you need to embed entire research papers or legal contracts as single units, this model holds context coherence that smaller embedders lose. Run via Hugging Face's Text Embeddings Inference server.
Run this on your own machine and stop paying every month
Pay once and keep it. No renewal, no per-token bill, and nothing you feed it ever leaves your hardware.
MTEB Benchmark Comparison
The English averages below are the figures each publisher reported for the original (v1) English MTEB benchmark. We have not re-run them, and the current MTEB leaderboard uses newer task sets, so check it for retrieval-specific scores before you choose.
| Model | English Avg | Multilingual Avg | Retrieval (NDCG@10) | Speed (emb/sec, RTX 4090) |
|---|---|---|---|---|
| nomic-embed-text (v1.5) | 62.28 (Nomic) | n/a | see MTEB leaderboard | not measured |
| mxbai-embed-large-v1 | 64.68 (Mixedbread) | n/a | see MTEB leaderboard | not measured |
| bge-m3 | not published as an MTEB average | MIRACL / MKQA results in BAAI's paper | see MTEB leaderboard | not measured |
| jina-embeddings-v3 | 65.52 (Jina) | see Jina's paper | see MTEB leaderboard | not measured |
| OpenAI text-embedding-3-small (cloud reference) | 62.3 (OpenAI) | n/a | see MTEB leaderboard | varies |
| OpenAI text-embedding-3-large (cloud reference) | 64.6 (OpenAI) | n/a | see MTEB leaderboard | varies |
The takeaway: on the publishers' own numbers, the open models in this list sit level with OpenAI's embedding models.
For the underlying methodology, the MTEB paper on arXiv is the canonical reference.
Embedding Dimension and Storage Math
Dimension drives everything downstream — index size, query latency, and how many vectors you can fit in RAM.
A vector of 768 float32 numbers is 3,072 bytes. A vector of 1024 float32 numbers is 4,096 bytes.
| Vectors | 768-dim float32 | 1024-dim float32 | 768-dim int8 |
|---|---|---|---|
| 100 K | 293 MB | 391 MB | 73 MB |
| 1 M | 2.9 GB | 3.9 GB | 732 MB |
| 10 M | 29 GB | 39 GB | 7.3 GB |
| 100 M | 293 GB | 391 GB | 73 GB |
Quantization: int8 quantization (one byte per dimension instead of four) cuts storage 4x, and binary quantization (one bit per dimension) cuts it 32x. Mixedbread trained mxbai-embed-large-v1 to tolerate both; measure the recall cost on your own corpus, since it varies with the data.
Matryoshka truncation: mxbai-embed-large-v1 and jina-embeddings-v3 are trained so the first N dimensions still encode meaning. Truncating mxbai from 1024 to 512 dimensions halves the index size; the recall cost is usually small, but measure it on your data before you commit.
API Patterns: Ollama, Infinity, TEI
Pick the server that matches your deployment.
Pattern A: Ollama (simplest)
The fastest path to a working endpoint. Ollama runs every model in this guide except jina-v3.
import requests
def embed(texts: list[str], model: str = "nomic-embed-text") -> list[list[float]]:
resp = requests.post(
"http://localhost:11434/api/embed",
json={"model": model, "input": texts},
timeout=60,
)
resp.raise_for_status()
return resp.json()["embeddings"]
vectors = embed(["First sentence.", "Second sentence."])
print(len(vectors), len(vectors[0])) # 2 768
For setup help, our Ollama Python API guide covers the full surface area.
Pattern B: Infinity (highest throughput)
Infinity batches aggressively and often hits 2-3x Ollama throughput on the same hardware. OpenAI-compatible API.
docker run --gpus all -p 7997:7997 \
michaelf34/infinity:latest \
v2 --model-id mixedbread-ai/mxbai-embed-large-v1 \
--port 7997
from openai import OpenAI
client = OpenAI(base_url="http://localhost:7997/v1", api_key="EMPTY")
resp = client.embeddings.create(
model="mixedbread-ai/mxbai-embed-large-v1",
input=["First sentence.", "Second sentence."]
)
Pattern C: Hugging Face TEI (production)
Text Embeddings Inference is Hugging Face's Rust-based server, a good fit for production jobs that need low latency.
docker run --gpus all -p 8080:80 \
-v $PWD/data:/data \
ghcr.io/huggingface/text-embeddings-inference:1.6 \
--model-id BAAI/bge-m3 \
--max-batch-tokens 16384
curl http://localhost:8080/embed \
-X POST \
-d '{"inputs":["What is local AI?"]}' \
-H 'Content-Type: application/json'
Latency depends on the GPU, batch size and chunk length, so time a few requests of your own (short queries and full-length chunks) before sizing a deployment.
Integration with Vector Stores
The embedding model is one half of retrieval. The vector index is the other.
Chroma (easiest)
import chromadb
from chromadb.utils.embedding_functions import OllamaEmbeddingFunction
client = chromadb.PersistentClient(path="./chroma")
embedder = OllamaEmbeddingFunction(
url="http://localhost:11434/api/embeddings",
model_name="nomic-embed-text",
)
collection = client.get_or_create_collection("docs", embedding_function=embedder)
collection.add(
documents=["Ollama runs models locally.", "Embeddings turn text into vectors."],
ids=["doc1", "doc2"],
)
results = collection.query(query_texts=["What is Ollama?"], n_results=2)
print(results["documents"])
Good up to 1-2M vectors. Past that, switch.
Qdrant (production)
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct
client = QdrantClient(host="localhost", port=6333)
client.recreate_collection(
collection_name="docs",
vectors_config=VectorParams(size=768, distance=Distance.COSINE),
)
vectors = embed(["First sentence.", "Second sentence."])
points = [PointStruct(id=i, vector=v, payload={"text": t})
for i, (t, v) in enumerate(zip(["First sentence.", "Second sentence."], vectors))]
client.upsert(collection_name="docs", points=points)
hits = client.search(
collection_name="docs",
query_vector=embed(["What is the first sentence?"])[0],
limit=3,
)
Qdrant's HNSW index is built for large collections; check query latency at your own collection size and filter settings.
For the bigger picture on RAG architectures, our Ollama + ChromaDB RAG pipeline post walks through the full retrieve-rerank-generate flow.
End-to-End RAG Pipeline
Here is a complete recipe. Embedding model: bge-m3. LLM: llama3.1:8b. Vector store: Qdrant. Reranker: bge-reranker-base.
import requests
from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance
OLLAMA = "http://localhost:11434"
QDRANT = QdrantClient(host="localhost", port=6333)
def embed_batch(texts: list[str], model: str = "bge-m3") -> list[list[float]]:
r = requests.post(f"{OLLAMA}/api/embed", json={"model": model, "input": texts})
return r.json()["embeddings"]
def rerank(query: str, candidates: list[str]) -> list[float]:
pairs = [{"query": query, "passage": c} for c in candidates]
r = requests.post(
"http://localhost:8081/rerank",
json={"query": query, "texts": candidates},
)
return r.json()["scores"]
def chat(prompt: str, model: str = "llama3.1:8b") -> str:
r = requests.post(f"{OLLAMA}/api/generate", json={
"model": model, "prompt": prompt, "stream": False
})
return r.json()["response"]
def rag(question: str, k: int = 20, top_n: int = 5) -> str:
# 1. Embed query
q_vec = embed_batch([question])[0]
# 2. Retrieve top-k candidates from Qdrant
hits = QDRANT.search(collection_name="docs", query_vector=q_vec, limit=k)
candidates = [h.payload["text"] for h in hits]
# 3. Rerank to top-n
scores = rerank(question, candidates)
ranked = [c for _, c in sorted(zip(scores, candidates), reverse=True)][:top_n]
# 4. Generate answer with retrieved context
context = "\n\n".join(ranked)
prompt = f"""Answer using the provided context only. If the context lacks the answer, say so.
Context:
{context}
Question: {question}
Answer:"""
return chat(prompt)
print(rag("How does Ollama handle quantization?"))
Time each stage on your own hardware (embed, search, rerank, generate); generation is almost always the largest share. The reranking step is the highest-leverage upgrade after the embedding model itself, because it re-scores the top candidates with a model that reads query and passage together. Measure recall@5 with and without it on a small labelled set of your own questions.
Common Pitfalls and Fixes
Pitfall 1: Mixing distance metrics
Cosine similarity and dot product are not interchangeable unless your vectors are L2-normalized. nomic-embed-text and bge-m3 output normalized vectors. mxbai does not by default. Symptom: nonsensical search results that look like random retrieval.
Fix: always normalize on insert and query.
import numpy as np
def normalize(v):
return (np.array(v) / np.linalg.norm(v)).tolist()
Pitfall 2: Wrong query prefix
bge-m3 was trained with the query prefix "Represent this sentence for searching relevant passages: ". Skip the prefix and recall drops 6-10 points.
def embed_query(q):
return embed_batch([f"Represent this sentence for searching relevant passages: {q}"])[0]
def embed_doc(d):
return embed_batch([d])[0]
mxbai uses the prefix "Represent this sentence for searching relevant passages: " for queries and no prefix for documents. nomic-embed-text uses "search_query: " and "search_document: ".
Pitfall 3: Chunking by character count
Chunking on character boundaries breaks sentences mid-word. The embedding still works but quality drops. Use semantic or sentence-aware chunking via LangChain's RecursiveCharacterTextSplitter or LlamaIndex's SemanticSplitterNodeParser.
Pitfall 4: Embedding noise
If you embed footers, page numbers, or boilerplate, every document looks similar. A practical rule: strip text that repeats across a large share of chunks (headers, footers, legal boilerplate) before embedding.
Pitfall 5: Using a 512-token model for long contexts
mxbai-embed-large-v1 has a 512-token window. Pass it 1500 tokens and the back half is silently truncated. Symptom: precise queries about content near the end of a chunk return no hits. Fix: switch to nomic-embed-text or bge-m3 (8192 tokens) when chunks exceed 400 tokens.
Hardware Sizing for Embedding Workloads
| Workload | Hardware | Throughput | Notes |
|---|---|---|---|
| Indexing 100K docs once | M2 Pro CPU | not measured | Fine for a one-off job; run it overnight if needed |
| Live RAG, 5 QPS | RTX 3060 12GB | not measured | Query embedding is a tiny load for any GPU |
| Indexing 10M docs | RTX 4090 | not measured | Batch size and chunk length decide throughput |
| Indexing 1B docs | 8x A100 | not measured | Shard the corpus across GPUs |
Time = number of chunks / embeddings per second, so measure your own rate on a 10,000-chunk sample before planning a large job.
Embedding is embarrassingly parallel. Bigger batches help linearly until VRAM saturates.
When to Pick Which Model
Use this decision tree:
- English only, short docs (< 400 tokens), max recall? mxbai-embed-large-v1 with 1024 dims.
- Multilingual or mixed? bge-m3.
- Long documents (> 1500 tokens) as single units? jina-embeddings-v3.
- Tight RAM, CPU-only inference, "good enough"? nomic-embed-text.
- Don't know yet? Start with nomic-embed-text. Switch only when you have a benchmark proving the upgrade matters.
Conclusion
Embedding quality compounds. A small jump from 53 NDCG@10 to 58 NDCG@10 means the LLM gets the right context far more often, which means hallucinations drop, which means users trust the system. Local models have closed every meaningful gap with cloud APIs while keeping data on your hardware and reducing per-query cost to effectively zero.
The blueprint is simple: pick the model that matches your language and document length, serve it through Ollama for prototypes or Infinity/TEI for production, normalize your vectors, use the model's expected query prefix, and rerank before sending context to the LLM. That gets you 90% of the way to a retrieval system that beats whatever you have today.
Building a RAG pipeline next? Our Ollama + ChromaDB RAG pipeline and private AI knowledge base guides assemble the full stack around the embedding choices in this post.
Go from reading about AI to building with AI
25 structured courses. Hands-on projects. Runs on your machine. Start free.
Liked this? 25 full AI courses are waiting.
From fundamentals to RAG, agents, MCP servers, voice AI, and production deployment with real GitHub repos. First chapter free, every course.
Build Real AI on Your Machine
RAG, agents, NLP, vision, and MLOps - chapters across 25 courses that take you from reading about AI to building AI.
Want the structured version?
Hands-on courses on local AI, from $8.99 a month. The first chapter of each is free.
Keep going
Comments (0)
No comments yet. Be the first to share your thoughts!
