KoboldCpp Setup Guide (v1.124): One-Binary Local LLM Server with UI

KoboldCpp is the all-in-one local AI appliance — a single executable that runs your LLM, ships a usable web UI, speaks the OpenAI API, generates images via Stable Diffusion, transcribes audio with Whisper, synthesizes speech with OuteTTS, exposes embeddings, and supports every modern sampler (including DRY and XTC). No install, no Python, no Docker required. Download one file and run it.

This guide covers everything: installation across Windows / Linux / macOS, GPU acceleration paths (CUDA, Vulkan, Metal, CPU), the Kobold Lite UI, the OpenAI-compatible API, multimodal pipelines, SillyTavern integration, sampler presets for chat / code / roleplay / creative writing, and tuning for common GPUs.

Download KoboldCpp

The only official download source is the GitHub releases page: github.com/LostRuins/koboldcpp/releases. The current release is v1.124.2 (October 31, 2026; the v1.124 release with hotfixes 1.124.1 and 1.124.2 folded in); the releases/latest link always points at the newest build. Pick your file:

That single file is the whole program — no installer, no Python, no dependencies. Releases ship frequently (often multiple per month), so re-downloading the same filename from /releases/latest is also how you update. If you like the one-file approach but want something even more minimal, llamafile is the other single-binary option — it bakes the model into the executable, while KoboldCpp keeps models separate and adds far more features.

Table of Contents

1. What KoboldCpp Is
2. Why It Beats Ollama for Some Workloads
3. Hardware Requirements
4. Installation: Windows, Linux, macOS
5. Your First Run
6. GPU Acceleration: CUDA, Vulkan, Metal, CPU
7. Picking GGUF Quantization
8. Kobold Lite Web UI
9. OpenAI-Compatible API
10. Sampler Reference (with DRY and XTC)
11. Context Shifting and Long Context
12. Speculative Decoding (Draft Models)
13. Vision Models (LLaVA, Llama 3.2 Vision, Qwen 2 VL)
14. Stable Diffusion Image Generation
15. Whisper Speech-to-Text
16. Text-to-Speech (OuteTTS)
17. Embeddings Server
18. SillyTavern Integration
19. Multi-User and Concurrency
20. Tuning Recipes by GPU
21. Troubleshooting
22. FAQ

---

What KoboldCpp Is

KoboldCpp (LostRuins/koboldcpp on GitHub) is a fork of llama.cpp packaged as a single executable. It includes:

• llama.cpp inference engine (same as Ollama under the hood)
• Kobold Lite — a web chat / story / roleplay UI
• Three APIs: KoboldAI legacy, OpenAI-compatible, and SSE streaming
• Stable Diffusion integration (txt2img, img2img, inpainting)
• Whisper speech-to-text
• OuteTTS text-to-speech
• CLIP / vision encoder support for multimodal models
• Embeddings endpoint compatible with OpenAI /v1/embeddings

All in one ~400 MB binary that needs no Python, no install, no virtualenv.

---

Why It Beats Ollama for Some Workloads

For a creative-writing single-user stack, KoboldCpp + SillyTavern is the de facto choice in 2026. If you're still deciding between the mainstream desktop options, our Jan vs LM Studio vs Ollama comparison covers that trio — KoboldCpp sits apart from all three as the portable, sampler-heavy power tool.

---

Hardware Requirements

KoboldCpp has the broadest hardware compatibility of any local LLM server — even integrated GPUs and 10-year-old CPUs run small models.

---

Installation: Windows, Linux, macOS

Windows

Download koboldcpp.exe from github.com/LostRuins/koboldcpp/releases (or the nocuda / oldpc variant — see the download table above). That's it — no install. Double-click to launch the GUI loader, or run from CLI.

Linux

# Pre-built binary (use koboldcpp-linux-x64-nocuda if you have no NVIDIA GPU)
wget https://github.com/LostRuins/koboldcpp/releases/latest/download/koboldcpp-linux-x64
chmod +x koboldcpp-linux-x64
./koboldcpp-linux-x64 --help

# Or build from source for ROCm / latest features
git clone https://github.com/LostRuins/koboldcpp
cd koboldcpp
make LLAMA_HIPBLAS=1   # AMD ROCm
# or:
make LLAMA_CUBLAS=1    # NVIDIA CUDA
# or:
make LLAMA_VULKAN=1    # Vulkan (any GPU)

macOS

# Pre-built ARM binary
wget https://github.com/LostRuins/koboldcpp/releases/latest/download/koboldcpp-mac-arm64
chmod +x koboldcpp-mac-arm64
./koboldcpp-mac-arm64 --help

Metal acceleration is built in.

Docker (community-maintained)

docker run --gpus all -p 5001:5001 \
    -v $(pwd)/models:/models \
    koboldai/koboldcpp:latest \
    --model /models/llama-3.1-8b.gguf --port 5001 --host 0.0.0.0

---

Your First Run

# Download a model
wget https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf

# Launch with sensible defaults
./koboldcpp \
    --model Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
    --usecuda \
    --gpulayers 999 \
    --contextsize 8192 \
    --port 5001

Open http://localhost:5001 in your browser. The Kobold Lite UI loads.

For Windows users without CLI: launch koboldcpp.exe and a GUI dialog asks for the model file, GPU acceleration option, and context size.

---

GPU Acceleration: CUDA, Vulkan, Metal, CPU

Most users want --usecuda (NVIDIA) or --usevulkan (AMD / Intel / mixed). Metal is auto-detected on Apple Silicon Macs. Every flag on this page was checked against the v1.124.2 source (koboldcpp.py at tag v1.124.2) on November 2, 2026; run --help on your build to confirm. Flags that older guides still print are gone or hidden: --usemetal and --useclblast no longer exist (Metal is automatic, CLBlast was removed), --flashattention is a hidden no-op because flash attention is now on by default (--noflashattention turns it off), --quantkv takes f16, bf16, q8_0, q5_1 or q4_0 (or 0-3), so --quantkv 8 fails, and there is no --ttsvoice flag (the voice is chosen per request).

For partial GPU offload on a card too small for the full model: set --gpulayers to a number lower than the model's layer count. See CUDA Optimization for layer counts per popular model.

---

Picking GGUF Quantization

For Llama 3.1 70B (24 GB RTX 4090 partial-offload reality; ceilings assume dual-channel DDR5-5600 system RAM at 89.6 GB/s for the layers that do not fit):

• Q4_K_M = 42 GB — partial offload, ~4 tok/s ceiling
• IQ3_XXS = 30 GB — partial offload, ~11 tok/s ceiling, with better quality than Q3_K_M
• Q2_K = 25 GB — fits with offload but quality cliff

For 70B fully on a single 24GB GPU, use ExLlamaV2 instead.

---

Kobold Lite Web UI

The built-in UI at http://localhost:5001 has four modes:

• Story — long-form continuation, no chat formatting.
• Adventure — text adventure / interactive fiction.
• Chat — chat with personas; supports world info / character cards.
• Instruct — single-turn instruction following.

Sidebar features:

• AI Settings — every sampler exposed
• Memory / Author's Note / World Info — persistent context injection
• Image generation — if Stable Diffusion is loaded
• Voice — if Whisper / TTS are loaded
• Saved presets — Balanced, Creative, Precise, plus custom

For roleplay-specific features (character cards, lorebooks, advanced UI), pair with SillyTavern instead — see SillyTavern Integration.

---

OpenAI-Compatible API

Endpoints mirror OpenAI plus KoboldAI legacy:

Example:

curl http://localhost:5001/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "llama-3.1-8b",
        "messages": [{"role":"user","content":"Hello"}],
        "temperature": 0.7,
        "min_p": 0.05,
        "dry_multiplier": 0.8
    }'

Note the non-OpenAI fields (min_p, dry_multiplier) — KoboldCpp accepts the full sampler stack as extension fields.

---

Sampler Reference (with DRY and XTC)

KoboldCpp supports every sampler in our LLM Sampling Parameters guide. Key flags / API fields:

{
  "temperature": 0.7,
  "top_k": 0,
  "top_p": 0.9,
  "min_p": 0.05,
  "typical_p": 1.0,
  "smoothing_factor": 0.0,
  "smoothing_curve": 1.0,
  "rep_pen": 1.05,
  "rep_pen_range": 1024,
  "rep_pen_slope": 0.7,
  "presence_penalty": 0.0,
  "frequency_penalty": 0.0,
  "dynatemp_range": 0.0,
  "dynatemp_exponent": 1.0,
  "mirostat": 0,
  "mirostat_tau": 5.0,
  "mirostat_eta": 0.1,
  "dry_multiplier": 0.8,
  "dry_base": 1.75,
  "dry_allowed_length": 2,
  "dry_penalty_last_n": 0,
  "dry_sequence_breakers": ["\n", ":", "\"", "*"],
  "xtc_threshold": 0.1,
  "xtc_probability": 0.5,
  "grammar": "..."
}

Built-in UI presets:

• Balanced — temp 0.7, min_p 0.05, rep_pen 1.05
• Creative — temp 1.1, min_p 0.05, dry 0.8, xtc 0.5/0.1
• Precise — temp 0.3, min_p 0.05, no XTC, no DRY

---

Context Shifting and Long Context

KoboldCpp's "Context Shifting" feature reuses KV cache when the conversation grows past max_seq_len — it drops the oldest tokens and shifts remaining KV cache without full re-prefill. This makes long-running chat sessions much faster than vanilla llama.cpp.

./koboldcpp \
    --model model.gguf \
    --contextsize 32768 \
    --noshift              # disable context shifting (rarely needed)

For RoPE-scaled extension on models that need it (older Llama 2 to 16K, etc.):

./koboldcpp \
    --ropeconfig 0.5 32000   # rope_freq_scale rope_freq_base

---

Speculative Decoding (Draft Models)

./koboldcpp \
    --model llama-3.1-70b.Q4_K_M.gguf \
    --draftmodel llama-3.2-1b.Q4_K_M.gguf \
    --draftgpulayers 999 \
    --draftamount 8

Pair models with the same tokenizer (Llama 3.1 70B + Llama 3.2 1B = same vocab). The speedup depends on how often the draft's guesses are accepted, so it varies by model pair and prompt; measure with and without the draft on your own workload. See CUDA Optimization for theory.

---

Vision Models (LLaVA, Llama 3.2 Vision, Qwen 2 VL)

./koboldcpp \
    --model llama-3.2-11b-vision.Q4_K_M.gguf \
    --mmproj llama-3.2-11b-vision-mmproj.gguf \
    --usecuda --gpulayers 999

The --mmproj flag points to the multimodal projector file (CLIP-style image encoder). In Kobold Lite chat mode, drag-and-drop images into the conversation.

Compatible vision models: Llama 3.2 11B / 90B Vision, Qwen 2 VL, LLaVA-1.5 / 1.6, MiniCPM-V, Pixtral.

---

Stable Diffusion Image Generation

./koboldcpp \
    --model llama-3.1-8b.gguf \
    --sdmodel sdxl-base-1.0.safetensors \
    --sdllm t5xxl_fp16.safetensors      # text encoder for SD 3.5 / Flux (replaces the old --sdt5xxl)

In Kobold Lite chat, type /img a cyberpunk samurai and the model generates an image inline.

API endpoint:

curl http://localhost:5001/sdapi/v1/txt2img \
    -H "Content-Type: application/json" \
    -d '{"prompt": "a cat", "steps": 25, "cfg_scale": 7.0}'

Supports SD 1.5, SDXL, SD 3.5, Flux Schnell. Memory budget: SD models share VRAM with the LLM, so plan accordingly. For dedicated image-gen workloads use ComfyUI instead.

---

Whisper Speech-to-Text

./koboldcpp --model llm.gguf --whispermodel ggml-medium.bin

API:

curl http://localhost:5001/api/extra/transcribe \
    -F "file=@audio.wav" -F "model=whisper"

Whisper models: tiny / base / small / medium / large-v3. medium is the sweet spot for English; large-v3 for accuracy or non-English. See Whisper Local Speech-to-Text for the standalone Whisper guide.

---

Text-to-Speech (OuteTTS)

./koboldcpp --model llm.gguf --ttsmodel oute-tts-0.3-1b.gguf --ttswavtokenizer wavtokenizer.gguf

OuteTTS needs its WavTokenizer file passed with --ttswavtokenizer. For voice cloning, point --ttsdir at a folder of reference voices; add --ttsgpu to run TTS on the GPU. v1.124 also added a set of TTS models ported from audio.cpp, including Qwen3-TTS, Chatterbox, Chatterbox Turbo, CosyVoice 3 and VoxCPM2 — the v1.124.2 release notes link a ready GGUF for each.

---

Embeddings Server

./koboldcpp --model llm.gguf --embeddingsmodel nomic-embed-text-v1.5.Q5_K_M.gguf

OpenAI-compatible /v1/embeddings endpoint. Compatible with Nomic Embed, BGE, GTE, E5, and most modern embedding GGUF models.

---

SillyTavern Integration

SillyTavern is the most-used roleplay / character-chat frontend; KoboldCpp is its preferred backend.

1. Launch KoboldCpp: ./koboldcpp --model X.gguf --port 5001 --host 0.0.0.0
2. Install SillyTavern (https://docs.sillytavern.app/installation/)
3. In SillyTavern: API → "Text Completion" → API type "KoboldCpp" → URL http://localhost:5001
4. Tick "Streaming"
5. Save and connect

SillyTavern uses KoboldCpp's full sampler stack and gives you persistent character cards, world info, lorebooks, group chats, and per-character preset overrides.

---

Multi-User and Concurrency

KoboldCpp queues requests from several users by default (--queuelimit, default 10; the old --multiuser flag is deprecated). To actually process requests in parallel, use the experimental --parallelrequests:

./koboldcpp --model llm.gguf --parallelrequests 4 --contextsize 8192

This enables up to 4 concurrent generations sharing the model. Throughput is much lower than vLLM (no PagedAttention), so for >4 concurrent users use vLLM. For 1-4 users on a desktop, KoboldCpp is fine.

---

Tuning Recipes by GPU

RTX 3060 12 GB

./koboldcpp \
    --model llama-3.1-8b-Q5_K_M.gguf \
    --usecuda --gpulayers 999 \
    --contextsize 16384 \
    --quantkv q8_0

RTX 4090 24 GB

./koboldcpp \
    --model llama-3.1-8b-Q5_K_M.gguf \
    --usecuda --gpulayers 999 \
    --contextsize 32768 \
    --quantkv q8_0

For 70B partial offload:

./koboldcpp \
    --model llama-3.1-70b-IQ3_XXS.gguf \
    --usecuda --gpulayers 65 \
    --contextsize 8192 \
    --quantkv q4_0

Apple M4 Max

./koboldcpp \
    --model llama-3.1-70b-Q4_K_M.gguf \
    --gpulayers 999 \
    --contextsize 16384

AMD RX 7900 XTX (Vulkan)

./koboldcpp \
    --model llama-3.1-8b-Q5_K_M.gguf \
    --usevulkan --gpulayers 999 \
    --contextsize 16384

---

Troubleshooting

---

FAQ

See answers to common KoboldCpp questions below.

---

Sources: KoboldCpp GitHub | Kobold Lite UI | SillyTavern docs | KoboldCpp v1.124.2 release notes. Speed figures are bandwidth ceilings (vendor memory bandwidth ÷ model size), not measurements.

Related guides:

• Complete Ollama Guide
• LLM Sampling Parameters Explained
• Best Ollama Clients
• Open WebUI Setup Guide
• Whisper Local Speech-to-Text
• Flux Local Image Generation
• ExLlamaV2 + TabbyAPI Guide