Private AI Knowledge Base: Self-Hosted Team Setup

Published April 11, 2026 · Updated August 23, 2026

A private AI knowledge base is four self-hosted pieces: Ollama for the language model, an embedding model to turn text into vectors, ChromaDB to store and search those vectors, and AnythingLLM as the web front end your team actually uses. Your documents are chunked, embedded and indexed on your own machine, so a question like "what is our remote work policy for contractors?" returns an answer grounded in your own files without a single byte leaving your network. Setup is an afternoon; keeping the document pipeline fresh is the part that decides whether anyone still trusts it in six months.

Company knowledge tends to be scattered across Confluence spaces, Slack history, Google Docs, a shared drive nobody remembers the password to, and the heads of three people who have been around since 2018. A new engineer asking "how do we deploy to staging?" burns twenty minutes finding out. A sales rep chasing the current pricing matrix pings three people and gets three answers.

Retrieval-augmented search fixes that permanently, and the reason to self-host it is not ideology. Internal runbooks, HR policy, compensation bands and financial reporting are exactly the categories a data-governance review will not let you paste into a third-party chat product. Running the whole stack locally removes the question rather than answering it.

---

How does the architecture fit together? {#architecture}

Four components, all self-hosted:

+---------------------------+
|  Team Members             |
|  (Browser -> AnythingLLM) |
+----------+----------------+
           |
           v
+----------+----------------+
|  AnythingLLM              |
|  (Web UI, workspaces,     |
|   user management)        |
+----------+----------------+
           |
     +-----+------+
     |            |
     v            v
+----+----+  +----+-------+
| Ollama  |  | ChromaDB   |
| (LLM)   |  | (Vectors)  |
+---------+  +----+-------+
                  |
                  v
         +--------+---------+
         | Embedding Model  |
         | (nomic-embed-    |
         |  text via Ollama)|
         +------------------+

How a query flows:

1. User asks "What is our policy on remote work for contractors?"
2. AnythingLLM sends the query to the embedding model, which converts it to a 768-dimensional vector
3. ChromaDB searches its vector index for the most similar document chunks
4. The top-k matching chunks are sent to Ollama along with the original question
5. Ollama generates an answer grounded in the retrieved documents
6. The user sees the answer with source references

The important thing to notice: the language model never "knows" your documents. It only ever sees the handful of chunks retrieval hands it. That is why almost every quality problem in this stack is a retrieval problem, not a model problem.

---

What do I install first? {#install-foundation}

Ollama + models

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull the LLM (choose based on your hardware)
# 24GB+ VRAM: best quality
ollama pull llama3.3:70b-instruct-q4_K_M

# 12-16GB VRAM: good balance
ollama pull qwen2.5:14b-instruct-q6_K

# 8GB VRAM: functional but less nuanced
ollama pull llama3.1:8b-instruct-q4_K_M

# Pull the embedding model (required for all setups)
ollama pull nomic-embed-text

Not sure which of those your machine can hold? Skip ahead to the sizing arithmetic — one multiplication tells you.

ChromaDB

# Run ChromaDB in Docker
docker run -d \
  --name chromadb \
  -p 8000:8000 \
  -v /data/chromadb:/chroma/chroma \
  -e ANONYMIZED_TELEMETRY=false \
  -e ALLOW_RESET=false \
  chromadb/chroma:latest

AnythingLLM (web interface)

# Run AnythingLLM with persistent storage
docker run -d \
  --name anythingllm \
  -p 3001:3001 \
  -v /data/anythingllm:/app/server/storage \
  -e LLM_PROVIDER=ollama \
  -e OLLAMA_BASE_PATH=http://host.docker.internal:11434 \
  -e EMBEDDING_ENGINE=ollama \
  -e EMBEDDING_MODEL_PREF=nomic-embed-text \
  -e VECTOR_DB=chroma \
  -e CHROMA_ENDPOINT=http://host.docker.internal:8000 \
  -e AUTH_TOKEN=your-secret-token-here \
  -e DISABLE_TELEMETRY=true \
  mintplexlabs/anythingllm

For a detailed walkthrough of the AnythingLLM interface and configuration, see the AnythingLLM setup guide.

---

How do I get my documents in? {#ingest-documents}

Supported sources and conversion

Scanned PDFs are the one category that silently fails: there is no text layer to extract, so they index as empty. Run them through OCR (Tesseract) before ingestion.

Slack archive conversion

Slack exports are JSON. Convert them to documents the AI can index:

#!/bin/bash
# convert-slack-export.sh
# Converts Slack JSON export to indexable text files

SLACK_EXPORT_DIR="$1"
OUTPUT_DIR="$2"

mkdir -p "${OUTPUT_DIR}"

for channel_dir in "${SLACK_EXPORT_DIR}"/*/; do
    channel=$(basename "${channel_dir}")
    echo "Processing channel: ${channel}"

    # Combine all messages for the channel
    outfile="${OUTPUT_DIR}/slack-${channel}.txt"
    echo "# Slack Channel: #${channel}" > "${outfile}"
    echo "" >> "${outfile}"

    for json_file in "${channel_dir}"/*.json; do
        python3 -c "
import json, sys
with open('${json_file}') as f:
    messages = json.load(f)
for msg in messages:
    if msg.get('type') == 'message' and 'subtype' not in msg:
        user = msg.get('user_profile', {}).get('real_name', msg.get('user', 'Unknown'))
        text = msg.get('text', '')
        if len(text) > 20:  # Skip short messages
            print(f'{user}: {text}')
            print()
" >> "${outfile}" 2>/dev/null
    done
done

echo "Converted $(ls "${OUTPUT_DIR}" | wc -l) channel files"

Confluence export

# Export from Confluence admin panel as HTML
# Then convert to clean text for better chunking

find /data/confluence-export -name "*.html" | while read html; do
    txtfile="${html%.html}.txt"
    pandoc "${html}" -t plain --wrap=none -o "${txtfile}"
done

Bulk upload via AnythingLLM

Once documents are converted, upload them through the AnythingLLM web interface. For large document sets, use the API:

# Upload documents programmatically
for doc in /data/documents/*.txt; do
    curl -X POST http://localhost:3001/api/v1/document/upload \
        -H "Authorization: Bearer your-secret-token-here" \
        -F "file=@${doc}"
done

# Trigger embedding for a workspace
curl -X POST http://localhost:3001/api/v1/workspace/company-kb/update-embeddings \
    -H "Authorization: Bearer your-secret-token-here" \
    -H "Content-Type: application/json" \
    -d '{"adds": ["all-uploaded-docs"]}'

---

What chunk size should I use? {#chunking-strategy}

Chunking is where knowledge bases fail silently. Wrong chunk size means retrieval hands the model irrelevant context, the model answers confidently from it, and the user blames the AI for a pipeline bug.

Start at 512 tokens with 50 tokens of overlap. That is the common default in AnythingLLM and LangChain, and it is a reasonable starting point for prose documents: big enough to hold a complete thought, small enough that a match is precise. Every size above and below trades the same two things against each other.

Do not take those on faith — that is what the evaluation set in retrieval tuning is for. Write 30-50 questions you already know the answers to, note which document should win, then change one variable at a time and count how often the right chunk lands in the top 5. Half an hour of that beats any generic recommendation, including this one.

Section-based chunking (advanced)

For well-structured documents (Markdown, HTML with headers), chunk by section instead of fixed size:

# section_chunker.py — preserves document structure
import re

def chunk_by_sections(text, max_tokens=1024, overlap_tokens=50):
    """Split text at headers while respecting max size."""
    # Split on Markdown headers
    sections = re.split(r'(?=^#{1,3} )', text, flags=re.MULTILINE)
    chunks = []
    current_chunk = ""

    for section in sections:
        word_count = len(section.split())
        if word_count > max_tokens:
            # Section too large — fall back to fixed-size splitting
            words = section.split()
            for i in range(0, len(words), max_tokens - overlap_tokens):
                chunk = " ".join(words[i:i + max_tokens])
                chunks.append(chunk)
        elif len(current_chunk.split()) + word_count > max_tokens:
            # Would exceed max — save current and start new
            chunks.append(current_chunk.strip())
            current_chunk = section
        else:
            current_chunk += "
" + section

    if current_chunk.strip():
        chunks.append(current_chunk.strip())

    return chunks

This preserves the logical structure of documents. A section about "Remote Work Policy" stays together instead of being split mid-paragraph.

---

Which embedding model should I pick? {#embedding-models}

The embedding model converts text into vectors. It is a completely separate model from the LLM that writes the answer, and swapping it invalidates your entire index — so pick once, deliberately.

Storage is arithmetic, not a benchmark: dimensions × 4 bytes per float × number of chunks. For nomic-embed-text that is 768 × 4 = 3,072 bytes ≈ 3 KB per chunk, so a million chunks costs about 3.1 GB of vectors. Add the raw text you keep alongside them and a large corporate corpus still lands in single-digit gigabytes. Vector storage is almost never the constraint people expect it to be.

For quality rankings, do not trust a table on a blog — including this one. The MTEB leaderboard is the maintained scoreboard for embedding models and it moves monthly. Filter it to the retrieval task, then check whether the model you want is actually available in Ollama.

If your corpus is mostly papers rather than corporate docs, see local AI for researchers: private lit review and paper drafting for the retrieval settings that suit academic PDFs. For a broader treatment of embedding choice and hybrid retrieval, Ollama semantic search goes a layer deeper.

Pull and test

# Pull the recommended embedding model
ollama pull nomic-embed-text

# Test embedding generation
curl -s http://localhost:11434/api/embeddings \
    -d '{"model": "nomic-embed-text", "prompt": "What is our vacation policy?"}' | \
    python3 -c "import sys,json; d=json.load(sys.stdin); print(f'Dimensions: {len(d["embedding"])}')"
# Output: Dimensions: 768

---

How do I tune retrieval so answers stop being wrong? {#retrieval-tuning}

Default retrieval settings in most RAG tools are conservative. Four parameters do most of the work.

Test retrieval on its own, before you blame the model

# Test retrieval without the LLM (see what chunks are returned)
curl -s http://localhost:8000/api/v1/collections/company-kb/query \
    -H "Content-Type: application/json" \
    -d '{
        "query_texts": ["What is our remote work policy for contractors?"],
        "n_results": 8
    }' | python3 -c "
import sys, json
data = json.load(sys.stdin)
for i, (doc, dist) in enumerate(zip(data['documents'][0], data['distances'][0])):
    similarity = 1 - dist  # ChromaDB returns distance, not similarity
    print(f'\nChunk {i+1} (similarity: {similarity:.3f}):')
    print(doc[:200] + '...')
"

If the answer is not in those chunks, no language model can save the response. Fix retrieval first. For a deeper dive into the whole pipeline, see the RAG local setup guide.

---

Can different teams see different documents? {#access-control}

Not everyone should query every document. Engineering does not need HR compensation data. Interns should not reach board minutes.

AnythingLLM supports workspaces — each with its own document collection and user permissions:

Workspaces:
├── engineering/        → Engineering team only
│   ├── runbooks/
│   ├── architecture-docs/
│   └── post-mortems/
├── sales/              → Sales + Leadership
│   ├── pricing/
│   ├── competitive-intel/
│   └── case-studies/
├── hr/                 → HR team only
│   ├── policies/
│   ├── compensation/
│   └── procedures/
└── company-wide/       → Everyone
    ├── handbook/
    ├── benefits/
    └── general-policies/

User role configuration

# Create workspace via API
curl -X POST http://localhost:3001/api/v1/workspace/new \
    -H "Authorization: Bearer your-secret-token-here" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "engineering",
        "openAiTemp": 0.2,
        "topN": 6,
        "similarityThreshold": 0.3
    }'

# Add user with workspace access
curl -X POST http://localhost:3001/api/v1/admin/users/new \
    -H "Authorization: Bearer your-secret-token-here" \
    -H "Content-Type: application/json" \
    -d '{
        "username": "jsmith",
        "password": "secure-password",
        "role": "default",
        "workspaces": ["engineering", "company-wide"]
    }'

In a custom ChromaDB setup, the equivalent is one collection per department plus role-based routing at query time.

---

How do I keep it from going stale? {#update-pipeline}

A knowledge base with stale data is worse than none, because people trust it and act on wrong answers.

Auto-ingest new documents

#!/bin/bash
# auto-ingest.sh — watches for new documents and re-embeds

WATCH_DIR="/data/documents"
ANYTHINGLLM_URL="http://localhost:3001"
API_KEY="your-secret-token-here"
WORKSPACE="company-wide"

# Track processed files
HASH_FILE="/data/anythingllm/.processed_hashes"
touch "${HASH_FILE}"

process_file() {
    local filepath="$1"
    local hash=$(sha256sum "${filepath}" | cut -d' ' -f1)

    # Skip if already processed with same hash
    if grep -q "${hash}" "${HASH_FILE}" 2>/dev/null; then
        return
    fi

    echo "[$(date)] Ingesting: ${filepath}"

    # Upload to AnythingLLM
    response=$(curl -s -X POST "${ANYTHINGLLM_URL}/api/v1/document/upload" \
        -H "Authorization: Bearer ${API_KEY}" \
        -F "file=@${filepath}")

    if echo "${response}" | grep -q "success"; then
        echo "${hash} ${filepath}" >> "${HASH_FILE}"
        echo "[$(date)] Success: ${filepath}"
    else
        echo "[$(date)] Failed: ${filepath} — ${response}"
    fi
}

# Process all files in the watch directory
find "${WATCH_DIR}" -type f \( -name "*.pdf" -o -name "*.docx" -o -name "*.txt" -o -name "*.md" \) | while read f; do
    process_file "$f"
done

# Run nightly via cron
echo "0 2 * * * /opt/knowledge-base/auto-ingest.sh >> /var/log/kb-ingest.log 2>&1" | sudo tee /etc/cron.d/kb-ingest

Hashing is the whole trick: only files whose contents changed get re-embedded, so a nightly run over a large corpus normally touches a handful of documents.

Confluence sync (automated)

#!/bin/bash
# sync-confluence.sh — pull latest from Confluence API

CONFLUENCE_URL="https://yourcompany.atlassian.net/wiki"
CONFLUENCE_TOKEN="your-api-token"
OUTPUT_DIR="/data/documents/confluence"

# List all pages modified in the last 24 hours
curl -s "${CONFLUENCE_URL}/rest/api/content?type=page&orderby=modified&limit=50&expand=body.storage" \
    -H "Authorization: Bearer ${CONFLUENCE_TOKEN}" \
    -H "Accept: application/json" | \
python3 -c "
import json, sys, os
from datetime import datetime, timedelta

data = json.load(sys.stdin)
cutoff = datetime.utcnow() - timedelta(hours=24)

for page in data.get('results', []):
    modified = datetime.strptime(page['version']['when'][:19], '%Y-%m-%dT%H:%M:%S')
    if modified > cutoff:
        title = page['title'].replace('/', '-')
        body = page['body']['storage']['value']
        filepath = f'${OUTPUT_DIR}/{title}.html'
        with open(filepath, 'w') as f:
            f.write(f'<h1>{page["title"]}</h1>
{body}')
        print(f'Updated: {title}')
"

---

How fast will it be on my hardware? {#performance}

You can work this out before you download anything, with two pieces of arithmetic. Both are ceilings — real output lands below them — but they tell you instantly whether a model is viable.

1. Will it fit? At Q4_K_M quantization, weights occupy roughly 0.6 GB per billion parameters, plus about 1-2 GB of headroom for the KV cache and context.

2. How fast can it possibly generate? Every token requires reading the whole model out of memory once, so:

tokens/second ceiling = memory bandwidth (GB/s) / model size in memory (GB)

Bandwidth is a published spec on every GPU and memory datasheet. Worked example: an RTX 4090 is specified at 1,008 GB/s; an 8B model at Q4_K_M is 4.8 GB; 1008 ÷ 4.8 ≈ 210 tokens/second, ceiling. The same card on a 70B model would be 1008 ÷ 42 ≈ 24 tokens/second — if the weights fit at all, which on 24 GB they do not.

Read those as upper bounds. Attention overhead, prompt processing and quantization inefficiency all pull the real number down, often by a third or more. What the table is genuinely good for is elimination: if the ceiling for a configuration is 2 tokens/second, no amount of tuning makes it a pleasant knowledge base.

Where the time actually goes. Vector search over a corporate corpus is the cheap part — HNSW lookup is sublinear in collection size, so retrieval cost barely moves as you go from 5,000 to 500,000 chunks. Generation dominates end-to-end response time, and generation is governed by the arithmetic above. That is why picking a 14B instead of a 70B is the single biggest lever on how responsive the thing feels.

---

Why do knowledge bases fail? {#failure-modes}

Five failure modes account for most broken deployments. Each has a distinctive symptom, which makes them easy to tell apart once you know the list.

1. Wrong chunk size

Symptom: confident answers built on irrelevant information. Cause: chunks too large, dragging unrelated content along with the match. Fix: drop from 1024 to 512 tokens and re-measure retrieval on your evaluation questions.

2. Poor embedding model

Symptom: retrieval returns documents about an entirely different topic. Cause: a general-purpose model that does not represent your domain vocabulary well. Fix: move from all-minilm to nomic-embed-text or larger. Consider fine-tuned embeddings if your domain is heavy on internal jargon.

3. Retrieval misses

Symptom: "I don't have information about that" when the document plainly exists. Cause: similarity threshold too high, or query phrasing far from the document's wording. Fix: lower similarity_threshold to 0.3, raise top_k to 8, and add a query-expansion step that rephrases the question before embedding it.

4. Stale data

Symptom: outdated answers — old pricing, retired processes. Cause: someone uploaded documents once and never again. Fix: the auto-ingest cron above, plus a visible last-ingested timestamp so staleness is obvious.

5. No access control

Symptom: an intern asks about executive compensation and gets a detailed answer. Cause: every document in one workspace, readable by everyone. Fix: per-department workspaces with role-based access.

---

Is self-hosting actually cheaper? {#cost-comparison}

Do the arithmetic rather than trusting a marketing table, because the answer depends entirely on headcount.

Your running cost is electricity, and it is a formula:

monthly cost = (average watts / 1000) x 720 hours x price per kWh

A knowledge base server idles most of the day and spikes on queries. At an average 150 W and $0.15/kWh: (150 ÷ 1000) × 720 × 0.15 = $16.20/month. At European rates near $0.30/kWh the same box is about $32/month. Plug in your own tariff — it is on your bill.

The cloud side is seats × price. Per-seat AI search and assistant products publish their own list prices, and they change; check the vendor's pricing page rather than a number copied into a blog post. Then break-even is one division:

break-even months = hardware cost / (cloud monthly - electricity monthly)

Worked at $1,500 of hardware and $16/month of electricity:

The shape matters more than any single cell: self-hosted cost is flat in headcount and per-seat cost is linear, so the case gets stronger with every person you add and is weakest for very small teams. Below roughly ten seats, self-hosting is a privacy and control decision, not a cost decision — and that is a perfectly good reason to do it.

If you already run a NAS, the hardware line drops close to zero — turning a QNAP or TrueNAS box into a private AI server covers hosting this same stack on hardware you already own. For the underlying container setup, the Ollama + Open WebUI Docker guide handles the foundation.

---

Common questions {#faq}

How is this different from a regular search engine? A search engine matches keywords; this matches meaning. Search "vacation policy for employees who started mid-year" and a keyword index needs those words to appear. An embedding index puts the query and the documents in the same vector space, so it still finds the policy when the document says "prorated PTO accrual for new hires."

What document types can I ingest? PDF, Word, plain text, Markdown, HTML, CSV and most common formats — AnythingLLM handles the conversion. Confluence exports as HTML, Slack as JSON (use the script above), Google Docs need exporting to Word or PDF first. Scanned PDFs need OCR before they index at all, because there is no text layer to read.

What embedding model should I use? nomic-embed-text is the sensible default for English business documents: 768 dimensions, 137M parameters, runs happily on CPU. mxbai-embed-large and bge-large-en-v1.5 produce 1024-dimensional vectors and cost more storage and indexing time. Check the MTEB leaderboard for current quality rankings, then test the top two on your own questions — the ranking that matters is the one on your corpus.

How do I handle document updates? Store a hash per file and re-embed only the ones that changed, which is exactly what the auto-ingest script does. AnythingLLM also supports re-syncing a workspace from the UI. Full re-indexing is the thing to avoid: it scales with corpus size while incremental updates scale with how much actually changed.

What chunk size should I start with? 512 tokens, 50-token overlap. Move to 1024 for technical documentation full of long code blocks, 256 for FAQ-style content. Irrelevant results usually mean chunks are too large; missing context usually means too small.

Can different teams have different access? Yes — AnythingLLM workspaces have separate document collections and per-user permissions, so engineering, HR and leadership can see different corpora from the same install. In a custom ChromaDB build, use one collection per department and route on user role. This is also what makes the setup defensible in a data-governance review.

How does answer quality compare to a hosted assistant on the same documents? Honestly: a frontier hosted model still has the edge on multi-hop questions that need synthesis across several documents. On straightforward factual lookups — "what is our conference reimbursement policy?" — the gap largely disappears, because the answer is sitting in the retrieved chunk and the model only has to relay it. The bigger quality variable in this stack is not which model you run, it is whether retrieval put the right chunk in front of it. Build the evaluation set, then judge for yourself on your own documents rather than trusting anyone's published win rate.

How much disk do I need? Two components: the raw text and the vectors. Vectors are exact arithmetic (dimensions × 4 bytes × chunks — about 3 KB per chunk at 768 dimensions), and the text is whatever your documents already occupy. A corpus of 100,000 documents split into roughly 800,000 chunks costs around 2.4 GB of vectors. Disk is rarely the binding constraint; VRAM is.

---

Conclusion

A private AI knowledge base changes how a team reaches its own institutional knowledge. Instead of searching six tools and pinging three colleagues, anyone asks a question in plain language and gets an answer grounded in the documents you actually maintain.

The stack is mature. Ollama serves inference reliably, ChromaDB handles hundreds of thousands of chunks without drama, and AnythingLLM gives non-technical users an interface that needs no training.

The hard part is not the software — it is the discipline of the document pipeline. Automate ingestion, make freshness visible, and re-run your evaluation questions every quarter. Start with one department's documentation, prove it works, then expand.

---

For the technical foundation, begin with the RAG local setup guide. Want a managed interface instead? The AnythingLLM setup guide gets you running quickly.