★ Reading this for free? Get 25 structured AI courses + per-chapter AI tutor — the first chapter of every course free, no card.Start free in 30 secondsOr own every course: $149 once
Developer Guide

Local AI Embeddings: Models, APIs, and Integration

April 23, 2026
18 min read
Barsha Mohapatra

Want to go deeper than this article?

Free account unlocks the first chapter of all 25 courses — RAG, agents, MCP, voice AI, MLOps, real GitHub repos.

📚AI Learning Path

Go from reading about AI to building with AI 25 structured courses. Hands-on projects. Runs on your machine. Start free.

Start free
Or own it for life — Lifetime $149, pay once

Published on April 23, 2026 • 18 min read

When a retrieval pipeline returns junk, the embedding model is one of the first things to check. Get the embedding wrong and no amount of clever prompting at the LLM stage can rescue the answer, because the right passages never reach the model. Get it right and a small local model can answer well from the passages it is given.

This guide skips the theory and walks straight into four local embedding models worth knowing, the API patterns for serving them, and the integration recipes that move recall. Quality scores below come from each model's publisher and the public MTEB leaderboard, with the source named; we have not re-run them.


Quick Start: Run Local Embeddings in 4 Minutes

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull the workhorse embedding model (137M params, 274 MB)
ollama pull nomic-embed-text

# Generate an embedding via the REST API
curl http://localhost:11434/api/embed -d '{
  "model": "nomic-embed-text",
  "input": "Local embeddings keep RAG private."
}'

You now have a 768-dimensional vector representing that sentence. At 274MB the model is light enough for a CPU; on any recent GPU, embedding short texts is fast enough that the vector search, not the embedding, is rarely your bottleneck.


Reading articles is good. Building is better.

Free account = the first chapter of all 25 courses, with a per-chapter AI tutor. No card.

Table of Contents

  1. Why Local Embeddings Matter
  2. Four Models Worth Knowing
  3. MTEB Benchmark Comparison
  4. Embedding Dimension and Storage Math
  5. API Patterns: Ollama, Infinity, TEI
  6. Integration with Vector Stores
  7. End-to-End RAG Pipeline
  8. Common Pitfalls and Fixes
  9. FAQs

Why Local Embeddings Matter

OpenAI reported an MTEB average of 64.6 for text-embedding-3-large when it launched (January 2024). Mixedbread reports 64.68 for the open mxbai-embed-large-v1, and Jina reports 65.52 on English MTEB for jina-embeddings-v3. On the publishers' own numbers, the accuracy gap that justified cloud embeddings has closed.

What does still exist:

  • Privacy: every chunk you embed in the cloud is content you sent to a third party.
  • Cost: OpenAI lists text-embedding-3-large at $0.13 per million tokens. At roughly 1,000 tokens per chunk, 100M chunks is 100B tokens, or about $13,000. Locally, the cost is the electricity for however many GPU hours your measured embedding rate implies.
  • Latency: every cloud embedding call includes a network round trip; a local model on the same machine as your retriever does not, so per-query latency is set by your own hardware.
  • Determinism: local models are pinned. Cloud models silently change behind the same name.

If you ship anything that processes customer documents, legal contracts, medical records, or proprietary code, local embeddings are no longer the "compromise" path. They are the default.

For a side-by-side of the trade-offs, see our local vs OpenAI embeddings deep dive.


Four Models Worth Knowing

Four open models cover most local RAG needs. Each fills a different slot.

1. nomic-embed-text (the daily driver)

  • Parameters: 137M
  • Dimensions: 768
  • Context window: 8192 tokens
  • Disk size: 274 MB
  • License: Apache 2.0

This is the sensible default when nothing fancy is required. At 274MB it runs comfortably on a CPU, and it holds its own on English retrieval for its size (Nomic reports an MTEB average of 62.28 for v1.5). Pull it with ollama pull nomic-embed-text.

2. bge-m3 (multilingual king)

  • Parameters: 567M
  • Dimensions: 1024
  • Context window: 8192 tokens
  • Disk size: 1.1 GB
  • License: MIT

Built by BAAI. Supports dense, sparse, and multi-vector retrieval out of one model. If your corpus mixes English, Chinese, Spanish, German, Arabic, or Hindi, it is the strongest open choice in this list; BAAI reports its results on multilingual retrieval benchmarks such as MIRACL and MKQA rather than a single MTEB average.

ollama pull bge-m3

3. mxbai-embed-large-v1 (English specialist)

  • Parameters: 335M
  • Dimensions: 1024
  • Context window: 512 tokens
  • Disk size: 670 MB
  • License: Apache 2.0

When the corpus is purely English and short documents (chat logs, support tickets, product reviews), it is a strong pick. Mixedbread reports an MTEB average of 64.68. It supports Matryoshka truncation and binary quantization, so you can cut dimensions to 512 to shrink the index; check the recall cost on your own data.

ollama pull mxbai-embed-large

4. jina-embeddings-v3 (long-context champion)

  • Parameters: 570M
  • Dimensions: 1024 (Matryoshka, can truncate to 256)
  • Context window: 8192 tokens
  • License: CC-BY-NC 4.0 (research) or commercial via Jina AI

When you need to embed entire research papers or legal contracts as single units, this model holds context coherence that smaller embedders lose. Run via Hugging Face's Text Embeddings Inference server.


Own it instead of renting it

Run this on your own machine and stop paying every month

Pay once and keep it. No renewal, no per-token bill, and nothing you feed it ever leaves your hardware.

MTEB Benchmark Comparison

The English averages below are the figures each publisher reported for the original (v1) English MTEB benchmark. We have not re-run them, and the current MTEB leaderboard uses newer task sets, so check it for retrieval-specific scores before you choose.

ModelEnglish AvgMultilingual AvgRetrieval (NDCG@10)Speed (emb/sec, RTX 4090)
nomic-embed-text (v1.5)62.28 (Nomic)n/asee MTEB leaderboardnot measured
mxbai-embed-large-v164.68 (Mixedbread)n/asee MTEB leaderboardnot measured
bge-m3not published as an MTEB averageMIRACL / MKQA results in BAAI's papersee MTEB leaderboardnot measured
jina-embeddings-v365.52 (Jina)see Jina's papersee MTEB leaderboardnot measured
OpenAI text-embedding-3-small (cloud reference)62.3 (OpenAI)n/asee MTEB leaderboardvaries
OpenAI text-embedding-3-large (cloud reference)64.6 (OpenAI)n/asee MTEB leaderboardvaries

The takeaway: on the publishers' own numbers, the open models in this list sit level with OpenAI's embedding models.

For the underlying methodology, the MTEB paper on arXiv is the canonical reference.


Embedding Dimension and Storage Math

Dimension drives everything downstream — index size, query latency, and how many vectors you can fit in RAM.

A vector of 768 float32 numbers is 3,072 bytes. A vector of 1024 float32 numbers is 4,096 bytes.

Vectors768-dim float321024-dim float32768-dim int8
100 K293 MB391 MB73 MB
1 M2.9 GB3.9 GB732 MB
10 M29 GB39 GB7.3 GB
100 M293 GB391 GB73 GB

Quantization: int8 quantization (one byte per dimension instead of four) cuts storage 4x, and binary quantization (one bit per dimension) cuts it 32x. Mixedbread trained mxbai-embed-large-v1 to tolerate both; measure the recall cost on your own corpus, since it varies with the data.

Matryoshka truncation: mxbai-embed-large-v1 and jina-embeddings-v3 are trained so the first N dimensions still encode meaning. Truncating mxbai from 1024 to 512 dimensions halves the index size; the recall cost is usually small, but measure it on your data before you commit.


API Patterns: Ollama, Infinity, TEI

Pick the server that matches your deployment.

Pattern A: Ollama (simplest)

The fastest path to a working endpoint. Ollama runs every model in this guide except jina-v3.

import requests

def embed(texts: list[str], model: str = "nomic-embed-text") -> list[list[float]]:
    resp = requests.post(
        "http://localhost:11434/api/embed",
        json={"model": model, "input": texts},
        timeout=60,
    )
    resp.raise_for_status()
    return resp.json()["embeddings"]

vectors = embed(["First sentence.", "Second sentence."])
print(len(vectors), len(vectors[0]))  # 2 768

For setup help, our Ollama Python API guide covers the full surface area.

Pattern B: Infinity (highest throughput)

Infinity batches aggressively and often hits 2-3x Ollama throughput on the same hardware. OpenAI-compatible API.

docker run --gpus all -p 7997:7997 \
  michaelf34/infinity:latest \
  v2 --model-id mixedbread-ai/mxbai-embed-large-v1 \
  --port 7997
from openai import OpenAI

client = OpenAI(base_url="http://localhost:7997/v1", api_key="EMPTY")
resp = client.embeddings.create(
    model="mixedbread-ai/mxbai-embed-large-v1",
    input=["First sentence.", "Second sentence."]
)

Pattern C: Hugging Face TEI (production)

Text Embeddings Inference is Hugging Face's Rust-based server, a good fit for production jobs that need low latency.

docker run --gpus all -p 8080:80 \
  -v $PWD/data:/data \
  ghcr.io/huggingface/text-embeddings-inference:1.6 \
  --model-id BAAI/bge-m3 \
  --max-batch-tokens 16384
curl http://localhost:8080/embed \
  -X POST \
  -d '{"inputs":["What is local AI?"]}' \
  -H 'Content-Type: application/json'

Latency depends on the GPU, batch size and chunk length, so time a few requests of your own (short queries and full-length chunks) before sizing a deployment.


Integration with Vector Stores

The embedding model is one half of retrieval. The vector index is the other.

Chroma (easiest)

import chromadb
from chromadb.utils.embedding_functions import OllamaEmbeddingFunction

client = chromadb.PersistentClient(path="./chroma")
embedder = OllamaEmbeddingFunction(
    url="http://localhost:11434/api/embeddings",
    model_name="nomic-embed-text",
)
collection = client.get_or_create_collection("docs", embedding_function=embedder)

collection.add(
    documents=["Ollama runs models locally.", "Embeddings turn text into vectors."],
    ids=["doc1", "doc2"],
)

results = collection.query(query_texts=["What is Ollama?"], n_results=2)
print(results["documents"])

Good up to 1-2M vectors. Past that, switch.

Qdrant (production)

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct

client = QdrantClient(host="localhost", port=6333)
client.recreate_collection(
    collection_name="docs",
    vectors_config=VectorParams(size=768, distance=Distance.COSINE),
)

vectors = embed(["First sentence.", "Second sentence."])
points = [PointStruct(id=i, vector=v, payload={"text": t})
          for i, (t, v) in enumerate(zip(["First sentence.", "Second sentence."], vectors))]
client.upsert(collection_name="docs", points=points)

hits = client.search(
    collection_name="docs",
    query_vector=embed(["What is the first sentence?"])[0],
    limit=3,
)

Qdrant's HNSW index is built for large collections; check query latency at your own collection size and filter settings.

For the bigger picture on RAG architectures, our Ollama + ChromaDB RAG pipeline post walks through the full retrieve-rerank-generate flow.


End-to-End RAG Pipeline

Here is a complete recipe. Embedding model: bge-m3. LLM: llama3.1:8b. Vector store: Qdrant. Reranker: bge-reranker-base.

import requests
from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance

OLLAMA = "http://localhost:11434"
QDRANT = QdrantClient(host="localhost", port=6333)

def embed_batch(texts: list[str], model: str = "bge-m3") -> list[list[float]]:
    r = requests.post(f"{OLLAMA}/api/embed", json={"model": model, "input": texts})
    return r.json()["embeddings"]

def rerank(query: str, candidates: list[str]) -> list[float]:
    pairs = [{"query": query, "passage": c} for c in candidates]
    r = requests.post(
        "http://localhost:8081/rerank",
        json={"query": query, "texts": candidates},
    )
    return r.json()["scores"]

def chat(prompt: str, model: str = "llama3.1:8b") -> str:
    r = requests.post(f"{OLLAMA}/api/generate", json={
        "model": model, "prompt": prompt, "stream": False
    })
    return r.json()["response"]

def rag(question: str, k: int = 20, top_n: int = 5) -> str:
    # 1. Embed query
    q_vec = embed_batch([question])[0]
    # 2. Retrieve top-k candidates from Qdrant
    hits = QDRANT.search(collection_name="docs", query_vector=q_vec, limit=k)
    candidates = [h.payload["text"] for h in hits]
    # 3. Rerank to top-n
    scores = rerank(question, candidates)
    ranked = [c for _, c in sorted(zip(scores, candidates), reverse=True)][:top_n]
    # 4. Generate answer with retrieved context
    context = "\n\n".join(ranked)
    prompt = f"""Answer using the provided context only. If the context lacks the answer, say so.

Context:
{context}

Question: {question}
Answer:"""
    return chat(prompt)

print(rag("How does Ollama handle quantization?"))

Time each stage on your own hardware (embed, search, rerank, generate); generation is almost always the largest share. The reranking step is the highest-leverage upgrade after the embedding model itself, because it re-scores the top candidates with a model that reads query and passage together. Measure recall@5 with and without it on a small labelled set of your own questions.


Common Pitfalls and Fixes

Pitfall 1: Mixing distance metrics

Cosine similarity and dot product are not interchangeable unless your vectors are L2-normalized. nomic-embed-text and bge-m3 output normalized vectors. mxbai does not by default. Symptom: nonsensical search results that look like random retrieval.

Fix: always normalize on insert and query.

import numpy as np
def normalize(v):
    return (np.array(v) / np.linalg.norm(v)).tolist()

Pitfall 2: Wrong query prefix

bge-m3 was trained with the query prefix "Represent this sentence for searching relevant passages: ". Skip the prefix and recall drops 6-10 points.

def embed_query(q):
    return embed_batch([f"Represent this sentence for searching relevant passages: {q}"])[0]

def embed_doc(d):
    return embed_batch([d])[0]

mxbai uses the prefix "Represent this sentence for searching relevant passages: " for queries and no prefix for documents. nomic-embed-text uses "search_query: " and "search_document: ".

Pitfall 3: Chunking by character count

Chunking on character boundaries breaks sentences mid-word. The embedding still works but quality drops. Use semantic or sentence-aware chunking via LangChain's RecursiveCharacterTextSplitter or LlamaIndex's SemanticSplitterNodeParser.

Pitfall 4: Embedding noise

If you embed footers, page numbers, or boilerplate, every document looks similar. A practical rule: strip text that repeats across a large share of chunks (headers, footers, legal boilerplate) before embedding.

Pitfall 5: Using a 512-token model for long contexts

mxbai-embed-large-v1 has a 512-token window. Pass it 1500 tokens and the back half is silently truncated. Symptom: precise queries about content near the end of a chunk return no hits. Fix: switch to nomic-embed-text or bge-m3 (8192 tokens) when chunks exceed 400 tokens.


Hardware Sizing for Embedding Workloads

WorkloadHardwareThroughputNotes
Indexing 100K docs onceM2 Pro CPUnot measuredFine for a one-off job; run it overnight if needed
Live RAG, 5 QPSRTX 3060 12GBnot measuredQuery embedding is a tiny load for any GPU
Indexing 10M docsRTX 4090not measuredBatch size and chunk length decide throughput
Indexing 1B docs8x A100not measuredShard the corpus across GPUs

Time = number of chunks / embeddings per second, so measure your own rate on a 10,000-chunk sample before planning a large job.

Embedding is embarrassingly parallel. Bigger batches help linearly until VRAM saturates.


When to Pick Which Model

Use this decision tree:

  1. English only, short docs (< 400 tokens), max recall? mxbai-embed-large-v1 with 1024 dims.
  2. Multilingual or mixed? bge-m3.
  3. Long documents (> 1500 tokens) as single units? jina-embeddings-v3.
  4. Tight RAM, CPU-only inference, "good enough"? nomic-embed-text.
  5. Don't know yet? Start with nomic-embed-text. Switch only when you have a benchmark proving the upgrade matters.

Conclusion

Embedding quality compounds. A small jump from 53 NDCG@10 to 58 NDCG@10 means the LLM gets the right context far more often, which means hallucinations drop, which means users trust the system. Local models have closed every meaningful gap with cloud APIs while keeping data on your hardware and reducing per-query cost to effectively zero.

The blueprint is simple: pick the model that matches your language and document length, serve it through Ollama for prototypes or Infinity/TEI for production, normalize your vectors, use the model's expected query prefix, and rerank before sending context to the LLM. That gets you 90% of the way to a retrieval system that beats whatever you have today.


Building a RAG pipeline next? Our Ollama + ChromaDB RAG pipeline and private AI knowledge base guides assemble the full stack around the embedding choices in this post.

🎯
AI Learning Path

Go from reading about AI to building with AI

25 structured courses. Hands-on projects. Runs on your machine. Start free.

Or own it for life — Lifetime $149 $599, pay once

Liked this? 25 full AI courses are waiting.

From fundamentals to RAG, agents, MCP servers, voice AI, and production deployment with real GitHub repos. First chapter free, every course.

Reading now
Join the discussion
TagsEmbeddingsRAGOllamaVector Searchbge-m3mxbai

Barsha Mohapatra

Local AI Master writes hands-on courses and hardware guides for running AI on machines you own. Content is checked against current releases and corrected when readers tell us it is wrong.

Build Real AI on Your Machine

RAG, agents, NLP, vision, and MLOps - chapters across 25 courses that take you from reading about AI to building AI.

Want the structured version?

Hands-on courses on local AI, from $8.99 a month. The first chapter of each is free.

AI Learning Path

Comments (0)

No comments yet. Be the first to share your thoughts!

📅 Published: April 23, 2026🔄 Last Updated: November 2, 2026✓ Manually Reviewed
Barsha Mohapatra

Written by

Barsha Mohapatra

Founder and Editor, Local AI Master · Founder, Little AI Master

Founder of Little AI Master and the editor behind every course and guide on Local AI Master. Years of tutoring students taught her where AI education actually fails.

Build Better RAG, Privately

Get our weekly engineering deep-dives on local embeddings, vector stores, and retrieval optimization. No cloud API noise.

Build Real AI on Your Machine

RAG, agents, NLP, vision, and MLOps - chapters across 25 courses that take you from reading about AI to building AI.

Was this helpful?

Related Guides

Continue your local AI journey with these comprehensive guides

📚
Free · no account required

Grab the AI Starter Kit — career roadmap, cheat sheet, setup guide

No spam. Unsubscribe with one click.

🎯
AI Learning Path

Go from reading about AI to building with AI

25 structured courses. Hands-on projects. Runs on your machine. Start free.

Or own it for life — Lifetime $149 $599, pay once
Free Tools & Calculators