Build a Private AI Knowledge Base for Your Team

Published on April 11, 2026 -- 19 min read

Your company's knowledge is scattered across 47 Confluence spaces, 12,000 Slack messages, 300 Google Docs, a shared drive nobody remembers the password to, and the heads of three people who have been at the company since 2018. When a new engineer asks "how do we deploy to staging?", the answer takes 20 minutes to find. When a sales rep needs the latest pricing matrix, they ping three people and get three different answers.

An AI knowledge base solves this permanently. Upload your documents, embed them into a vector database, and let your team ask questions in natural language. The AI retrieves the relevant sections and generates an answer grounded in your actual documentation — not hallucinated from training data.

The critical requirement: this must run on your hardware. Sending your internal documentation, HR policies, financial data, and engineering runbooks to OpenAI or Google is a data governance failure. This guide builds the entire system locally, with zero cloud dependencies and zero per-user fees.

---

Architecture Overview {#architecture}

Four components, all self-hosted:

+---------------------------+
|  Team Members             |
|  (Browser -> AnythingLLM) |
+----------+----------------+
           |
           v
+----------+----------------+
|  AnythingLLM              |
|  (Web UI, workspaces,     |
|   user management)        |
+----------+----------------+
           |
     +-----+------+
     |            |
     v            v
+----+----+  +----+-------+
| Ollama  |  | ChromaDB   |
| (LLM)   |  | (Vectors)  |
+---------+  +----+-------+
                  |
                  v
         +--------+---------+
         | Embedding Model  |
         | (nomic-embed-    |
         |  text via Ollama)|
         +------------------+

How a query flows:

1. User asks "What is our policy on remote work for contractors?"
2. AnythingLLM sends the query to the embedding model, which converts it to a 768-dimensional vector
3. ChromaDB searches its vector index for the most similar document chunks
4. The top-k matching chunks are sent to Ollama along with the original question
5. Ollama generates an answer grounded in the retrieved documents
6. The user sees the answer with source references

---

Step 1: Install the Foundation {#install-foundation}

Ollama + Models

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull the LLM (choose based on your hardware)
# 24GB+ VRAM: best quality
ollama pull llama3.3:70b-instruct-q4_K_M

# 12-16GB VRAM: good balance
ollama pull qwen2.5:14b-instruct-q6_K

# 8GB VRAM: functional but less nuanced
ollama pull llama3.3:8b-instruct-q4_K_M

# Pull the embedding model (required for all setups)
ollama pull nomic-embed-text

ChromaDB

# Run ChromaDB in Docker
docker run -d \
  --name chromadb \
  -p 8000:8000 \
  -v /data/chromadb:/chroma/chroma \
  -e ANONYMIZED_TELEMETRY=false \
  -e ALLOW_RESET=false \
  chromadb/chroma:latest

AnythingLLM (Web Interface)

# Run AnythingLLM with persistent storage
docker run -d \
  --name anythingllm \
  -p 3001:3001 \
  -v /data/anythingllm:/app/server/storage \
  -e LLM_PROVIDER=ollama \
  -e OLLAMA_BASE_PATH=http://host.docker.internal:11434 \
  -e EMBEDDING_ENGINE=ollama \
  -e EMBEDDING_MODEL_PREF=nomic-embed-text \
  -e VECTOR_DB=chroma \
  -e CHROMA_ENDPOINT=http://host.docker.internal:8000 \
  -e AUTH_TOKEN=your-secret-token-here \
  -e DISABLE_TELEMETRY=true \
  mintplexlabs/anythingllm

For a detailed walkthrough of the AnythingLLM interface and configuration, see the AnythingLLM setup guide.

---

Step 2: Ingest Your Documents {#ingest-documents}

Supported Sources and Conversion

Slack Archive Conversion

Slack exports are JSON. Convert them to documents the AI can index:

#!/bin/bash
# convert-slack-export.sh
# Converts Slack JSON export to indexable text files

SLACK_EXPORT_DIR="$1"
OUTPUT_DIR="$2"

mkdir -p "${OUTPUT_DIR}"

for channel_dir in "${SLACK_EXPORT_DIR}"/*/; do
    channel=$(basename "${channel_dir}")
    echo "Processing channel: ${channel}"

    # Combine all messages for the channel
    outfile="${OUTPUT_DIR}/slack-${channel}.txt"
    echo "# Slack Channel: #${channel}" > "${outfile}"
    echo "" >> "${outfile}"

    for json_file in "${channel_dir}"/*.json; do
        python3 -c "
import json, sys
with open('${json_file}') as f:
    messages = json.load(f)
for msg in messages:
    if msg.get('type') == 'message' and 'subtype' not in msg:
        user = msg.get('user_profile', {}).get('real_name', msg.get('user', 'Unknown'))
        text = msg.get('text', '')
        if len(text) > 20:  # Skip short messages
            print(f'{user}: {text}')
            print()
" >> "${outfile}" 2>/dev/null
    done
done

echo "Converted $(ls "${OUTPUT_DIR}" | wc -l) channel files"

Confluence Export

# Export from Confluence admin panel as HTML
# Then convert to clean text for better chunking

find /data/confluence-export -name "*.html" | while read html; do
    txtfile="${html%.html}.txt"
    pandoc "${html}" -t plain --wrap=none -o "${txtfile}"
done

Bulk Upload via AnythingLLM

Once documents are converted, upload them through the AnythingLLM web interface. For large document sets, use the API:

# Upload documents programmatically
for doc in /data/documents/*.txt; do
    curl -X POST http://localhost:3001/api/v1/document/upload \
        -H "Authorization: Bearer your-secret-token-here" \
        -F "file=@${doc}"
done

# Trigger embedding for a workspace
curl -X POST http://localhost:3001/api/v1/workspace/company-kb/update-embeddings \
    -H "Authorization: Bearer your-secret-token-here" \
    -H "Content-Type: application/json" \
    -d '{"adds": ["all-uploaded-docs"]}'

---

Step 3: Chunking Strategy {#chunking-strategy}

Chunking is where most knowledge bases fail silently. Wrong chunk size means the AI retrieves irrelevant context and gives wrong answers. The user blames the AI. The real problem is the pipeline.

Chunk Size Comparison

We tested five chunk sizes against a 2,000-document corporate corpus with 100 ground-truth Q&A pairs:

512 tokens with 50-token overlap is the default you should start with. Only change this after testing with your specific documents and queries.

Section-Based Chunking (Advanced)

For well-structured documents (Markdown, HTML with headers), chunk by section instead of fixed size:

# section_chunker.py — preserves document structure
import re

def chunk_by_sections(text, max_tokens=1024, overlap_tokens=50):
    """Split text at headers while respecting max size."""
    # Split on Markdown headers
    sections = re.split(r'(?=^#{1,3} )', text, flags=re.MULTILINE)
    chunks = []
    current_chunk = ""

    for section in sections:
        word_count = len(section.split())
        if word_count > max_tokens:
            # Section too large — fall back to fixed-size splitting
            words = section.split()
            for i in range(0, len(words), max_tokens - overlap_tokens):
                chunk = " ".join(words[i:i + max_tokens])
                chunks.append(chunk)
        elif len(current_chunk.split()) + word_count > max_tokens:
            # Would exceed max — save current and start new
            chunks.append(current_chunk.strip())
            current_chunk = section
        else:
            current_chunk += "
" + section

    if current_chunk.strip():
        chunks.append(current_chunk.strip())

    return chunks

This approach preserves the logical structure of documents. A section about "Remote Work Policy" stays together instead of being split mid-paragraph.

---

Step 4: Embedding Model Selection {#embedding-models}

The embedding model converts text into numerical vectors. This is separate from the LLM that generates answers. Choosing the wrong embedding model affects every query.

Pull and Test

# Pull the recommended embedding model
ollama pull nomic-embed-text

# Test embedding generation
curl -s http://localhost:11434/api/embeddings \
    -d '{"model": "nomic-embed-text", "prompt": "What is our vacation policy?"}' | \
    python3 -c "import sys,json; d=json.load(sys.stdin); print(f'Dimensions: {len(d["embedding"])}')"
# Output: Dimensions: 768

---

Step 5: Retrieval Tuning {#retrieval-tuning}

The default retrieval settings in most RAG tools are conservative. Tuning these parameters significantly impacts answer quality.

Key Parameters

Testing Retrieval Quality

Before rolling out, test with questions you know the answers to:

# Test retrieval without the LLM (see what chunks are returned)
curl -s http://localhost:8000/api/v1/collections/company-kb/query \
    -H "Content-Type: application/json" \
    -d '{
        "query_texts": ["What is our remote work policy for contractors?"],
        "n_results": 8
    }' | python3 -c "
import sys, json
data = json.load(sys.stdin)
for i, (doc, dist) in enumerate(zip(data['documents'][0], data['distances'][0])):
    similarity = 1 - dist  # ChromaDB returns distance, not similarity
    print(f'\nChunk {i+1} (similarity: {similarity:.3f}):')
    print(doc[:200] + '...')
"

If the top results do not contain the answer, the problem is chunking or embedding — not the LLM. Fix retrieval before blaming the language model.

For a deeper dive into RAG pipeline optimization, see the RAG local setup guide.

---

Step 6: Access Control {#access-control}

Not everyone should query every document. Engineering does not need HR compensation data. Interns should not access board meeting minutes.

Workspace-Based Access in AnythingLLM

AnythingLLM supports workspaces — each workspace has its own document collection and user permissions:

Workspaces:
├── engineering/        → Engineering team only
│   ├── runbooks/
│   ├── architecture-docs/
│   └── post-mortems/
├── sales/              → Sales + Leadership
│   ├── pricing/
│   ├── competitive-intel/
│   └── case-studies/
├── hr/                 → HR team only
│   ├── policies/
│   ├── compensation/
│   └── procedures/
└── company-wide/       → Everyone
    ├── handbook/
    ├── benefits/
    └── general-policies/

User Role Configuration

# Create workspace via API
curl -X POST http://localhost:3001/api/v1/workspace/new \
    -H "Authorization: Bearer your-secret-token-here" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "engineering",
        "openAiTemp": 0.2,
        "topN": 6,
        "similarityThreshold": 0.3
    }'

# Add user with workspace access
curl -X POST http://localhost:3001/api/v1/admin/users/new \
    -H "Authorization: Bearer your-secret-token-here" \
    -H "Content-Type: application/json" \
    -d '{
        "username": "jsmith",
        "password": "secure-password",
        "role": "default",
        "workspaces": ["engineering", "company-wide"]
    }'

---

Step 7: Update Pipeline {#update-pipeline}

A knowledge base with stale data is worse than no knowledge base. People lose trust and stop using it.

Auto-Ingest New Documents

#!/bin/bash
# auto-ingest.sh — watches for new documents and re-embeds

WATCH_DIR="/data/documents"
ANYTHINGLLM_URL="http://localhost:3001"
API_KEY="your-secret-token-here"
WORKSPACE="company-wide"

# Track processed files
HASH_FILE="/data/anythingllm/.processed_hashes"
touch "${HASH_FILE}"

process_file() {
    local filepath="$1"
    local hash=$(sha256sum "${filepath}" | cut -d' ' -f1)

    # Skip if already processed with same hash
    if grep -q "${hash}" "${HASH_FILE}" 2>/dev/null; then
        return
    fi

    echo "[$(date)] Ingesting: ${filepath}"

    # Upload to AnythingLLM
    response=$(curl -s -X POST "${ANYTHINGLLM_URL}/api/v1/document/upload" \
        -H "Authorization: Bearer ${API_KEY}" \
        -F "file=@${filepath}")

    if echo "${response}" | grep -q "success"; then
        echo "${hash} ${filepath}" >> "${HASH_FILE}"
        echo "[$(date)] Success: ${filepath}"
    else
        echo "[$(date)] Failed: ${filepath} — ${response}"
    fi
}

# Process all files in the watch directory
find "${WATCH_DIR}" -type f \( -name "*.pdf" -o -name "*.docx" -o -name "*.txt" -o -name "*.md" \) | while read f; do
    process_file "$f"
done

# Run nightly via cron
echo "0 2 * * * /opt/knowledge-base/auto-ingest.sh >> /var/log/kb-ingest.log 2>&1" | sudo tee /etc/cron.d/kb-ingest

Confluence Sync (Automated)

#!/bin/bash
# sync-confluence.sh — pull latest from Confluence API

CONFLUENCE_URL="https://yourcompany.atlassian.net/wiki"
CONFLUENCE_TOKEN="your-api-token"
OUTPUT_DIR="/data/documents/confluence"

# List all pages modified in the last 24 hours
curl -s "${CONFLUENCE_URL}/rest/api/content?type=page&orderby=modified&limit=50&expand=body.storage" \
    -H "Authorization: Bearer ${CONFLUENCE_TOKEN}" \
    -H "Accept: application/json" | \
python3 -c "
import json, sys, os
from datetime import datetime, timedelta

data = json.load(sys.stdin)
cutoff = datetime.utcnow() - timedelta(hours=24)

for page in data.get('results', []):
    modified = datetime.strptime(page['version']['when'][:19], '%Y-%m-%dT%H:%M:%S')
    if modified > cutoff:
        title = page['title'].replace('/', '-')
        body = page['body']['storage']['value']
        filepath = f'${OUTPUT_DIR}/{title}.html'
        with open(filepath, 'w') as f:
            f.write(f'<h1>{page["title"]}</h1>
{body}')
        print(f'Updated: {title}')
"

---

Performance Numbers {#performance}

Real benchmarks from a 5,000-document corporate knowledge base running on a single RTX 4090 with 64GB system RAM:

Query Latency

Retrieval time is nearly constant regardless of corpus size because vector search is O(log n). The bottleneck is always the LLM generation phase.

Ingestion Speed

A 5,000-document corpus (averaging 10 pages each) takes approximately 4 hours for initial ingestion. Incremental updates are much faster — only changed documents are re-processed.

Accuracy vs. Cloud Alternatives

Tested with 100 ground-truth Q&A pairs across engineering, HR, and sales documentation:

The 4% gap between local and GPT-4 narrows with better chunking and retrieval tuning. The 14% gap over Notion AI justifies the effort immediately.

---

Common Failure Modes {#failure-modes}

Every failed knowledge base deployment we have seen died from one of these five causes:

1. Wrong Chunk Size

Symptom: AI answers with confident but irrelevant information. Cause: Chunks too large, pulling in unrelated content from the same document section. Fix: Reduce from 1024 to 512 tokens. Test retrieval quality before and after.

2. Poor Embedding Model

Symptom: Retrieval returns documents about the wrong topic entirely. Cause: Using a general-purpose model that does not understand your domain vocabulary. Fix: Switch from all-minilm to nomic-embed-text. Consider fine-tuning embeddings if your domain has highly specialized terminology.

3. Retrieval Misses

Symptom: AI says "I don't have information about that" when the document exists. Cause: Similarity threshold too high, or the query phrasing is too different from the document language. Fix: Lower similarity_threshold from 0.5 to 0.3. Increase top_k from 4 to 8. Add a "query expansion" step that rephrases the question.

4. Stale Data

Symptom: AI gives outdated answers (old pricing, deprecated processes). Cause: No automated update pipeline. Someone uploaded documents once and never again. Fix: Implement the auto-ingest script from Step 7. Monitor the last-ingested timestamp.

5. No Access Control

Symptom: Intern asks about executive compensation and gets a detailed answer. Cause: All documents in a single workspace accessible to everyone. Fix: Separate workspaces per department with role-based access.

---

Cost Comparison {#cost-comparison}

The math on self-hosted vs. cloud knowledge base tools:

At 50 users, the self-hosted system pays for itself in 2-4 months compared to Notion AI, and in 1-2 months compared to Microsoft Copilot. After that, you are saving $500-1,500 every month with better privacy and no vendor lock-in.

For the complete setup, the Ollama + Open WebUI Docker guide handles the foundational infrastructure.

---

Conclusion

A private AI knowledge base transforms how your team accesses institutional knowledge. Instead of searching through dozens of tools, pinging colleagues, and hoping someone remembers where a document lives, anyone can ask a natural language question and get an answer grounded in your actual documentation.

The technology stack is mature enough for production use. Ollama handles inference reliably. ChromaDB scales to hundreds of thousands of documents without performance degradation. AnythingLLM provides a polished interface that non-technical users can operate without training.

The hard part is not the software — it is the discipline of maintaining the document pipeline. A knowledge base that is six months stale is worse than no knowledge base at all, because people trust it and get wrong answers. Automate the ingestion. Monitor the freshness. Review the accuracy quarterly.

Start with one department's documentation. Prove the value. Then expand.

---

For the technical foundation, begin with the RAG local setup guide. Need a managed interface? The AnythingLLM setup guide gets you running in under 30 minutes.