Ollama Load Balancing with Nginx and HAProxy

Ollama has no load balancer of its own — it is a single-node inference server. You scale it horizontally by putting a reverse proxy in front of several instances: Nginx or HAProxy with a least-connections policy for plain fan-out, or LiteLLM's router when different backends hold different models. The two settings that break streaming if you get them wrong are response buffering (must be off) and the idle timeout (must be minutes, not the 60-second defaults). Everything else is ordinary reverse-proxy work.

This guide gives you working Nginx and HAProxy configs, when each one is the right pick, how sticky sessions interact with streaming, model-aware routing with LiteLLM, and — instead of somebody else's benchmark numbers — the arithmetic to size a cluster before you spend anything.

What does the smallest working setup look like?

Two Ollama instances on different ports, one upstream block, five minutes.

OLLAMA_HOST=0.0.0.0:11434 OLLAMA_KEEP_ALIVE=24h ollama serve &
OLLAMA_HOST=0.0.0.0:11435 OLLAMA_KEEP_ALIVE=24h ollama serve &

Drop this at /etc/nginx/sites-available/ollama:

upstream ollama_backends {
    least_conn;
    server 127.0.0.1:11434 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:11435 max_fails=3 fail_timeout=30s;
}

server {
    listen 8080;
    location / {
        proxy_pass http://ollama_backends;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_buffering off;
        proxy_read_timeout 600s;
        proxy_send_timeout 600s;
    }
}

Reload Nginx, point clients at port 8080, and you are load balanced. Note the two lines that are not optional: proxy_buffering off (Nginx otherwise holds the whole response before forwarding it, which kills token streaming) and the 600-second timeouts (the Nginx default for proxy_read_timeout is 60 seconds, which cuts long generations off mid-token).

The rest of this guide is the engineering you do after the first incident.

Table of Contents

1. Why Ollama needs a load balancer
2. Nginx vs HAProxy vs LiteLLM
3. Production Nginx configuration
4. Production HAProxy configuration
5. Streaming and sticky sessions
6. Model-aware routing with LiteLLM
7. Health checks done right
8. TLS, auth, and rate limiting
9. Sizing a cluster: the arithmetic
10. Common pitfalls
11. FAQ

---

Why does Ollama need a load balancer at all?

Because a single instance is bounded by three things at once, and none of them are fixed by a faster GPU:

• VRAM. One model resident at a time, maybe two if both are small and quantized. OLLAMA_MAX_LOADED_MODELS governs how many Ollama will keep in memory concurrently, and every extra one costs VRAM you no longer have for KV cache.
• Concurrency slots. OLLAMA_NUM_PARALLEL sets how many requests one instance handles simultaneously. Requests beyond that number queue — they do not fail, they just wait, which users experience as the service getting slower rather than breaking.
• Blast radius. One process, one host. It restarts, everybody stops.

Horizontal scaling — several Ollama instances behind a load balancer — addresses all three:

• Add a backend, add its slots and its VRAM. No GPU swap.
• Mix loadouts: some backends holding a small chat model, one holding something large for occasional heavy requests.
• Survive a backend death. One node reboots, traffic routes around it.
• Cap request explosion at the LB instead of in every client app.

For the single-backend hardening that comes before this, see our Ollama production deployment guide; Ollama on Kubernetes is the right step up once you outgrow systemd-managed VMs.

---

Nginx, HAProxy or LiteLLM — which should you use?

Three good options with genuinely different strengths. This table is a capability comparison drawn from each project's documentation, not a benchmark.

Which to pick:

• Single team, 2-4 backends, one model everywhere → Nginx. Least config, and you probably already run it.
• Stricter latency goals, active health checking without a paid tier, rolling restarts that must not drop streams → HAProxy.
• Several models with different VRAM footprints, or a mix of local and hosted models behind one endpoint → LiteLLM.

They stack, too: LiteLLM for model-aware routing with Nginx in front of it for TLS, auth and rate limiting is a common shape.

---

What does a production Nginx config look like?

Four backends, TLS, API-key gate, per-key rate limiting, streaming-safe:

upstream ollama_backends {
    least_conn;
    keepalive 32;

    server 10.0.1.10:11434 max_fails=3 fail_timeout=30s;
    server 10.0.1.11:11434 max_fails=3 fail_timeout=30s;
    server 10.0.1.12:11434 max_fails=3 fail_timeout=30s;
    server 10.0.1.13:11434 max_fails=3 fail_timeout=30s;
}

# Rate limit zone: 60 req/min per key
limit_req_zone $http_authorization zone=ollama_per_key:10m rate=60r/m;

server {
    listen 443 ssl http2;
    server_name ollama.internal.example.com;

    ssl_certificate     /etc/letsencrypt/live/ollama.internal.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ollama.internal.example.com/privkey.pem;
    ssl_protocols       TLSv1.3 TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;

    # API key gate
    if ($http_authorization !~ "^Bearer (sk-team-[a-z0-9]+)$") {
        return 401;
    }

    # Rate limit (60/min per API key)
    limit_req zone=ollama_per_key burst=20 nodelay;
    limit_req_status 429;

    # Streaming requirements
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_buffering off;          # critical for token streaming
    proxy_cache off;
    proxy_read_timeout 600s;      # long generations
    proxy_send_timeout 600s;
    proxy_request_buffering off;  # don't buffer uploads (file inference)

    # Health check endpoint (no auth required)
    location = /health {
        proxy_pass http://ollama_backends/;
        access_log off;
    }

    # Main proxy
    location / {
        proxy_pass http://ollama_backends;
    }

    # Generous body size for image inputs (vision models)
    client_max_body_size 50M;
}

server {
    listen 80;
    server_name ollama.internal.example.com;
    return 301 https://$host$request_uri;
}

What every directive is doing

• least_conn — send each new request to the backend with the fewest in-flight requests. This is the single most important choice on the page, and the reasoning is in the sizing section.
• keepalive 32 — reuse upstream connections instead of paying a TCP handshake per request.
• max_fails=3 fail_timeout=30s — after three failures inside the window, take the backend out of rotation for 30 seconds.
• proxy_buffering off — without it Nginx buffers the whole response before sending. Streaming stops working.
• proxy_http_version 1.1 + Connection "" — required for upstream keepalive to function.
• proxy_read_timeout 600s — the default is 60s. A long generation on a large model can exceed that easily.
• limit_req — per-API-key rate limit. burst=20 allows short spikes over the sustained 60/min.
• client_max_body_size 50M — Nginx defaults to 1MB, which rejects image uploads to vision models with a 413.

Reload with nginx -t && systemctl reload nginx. To confirm least-conn is distributing, log $upstream_addr and watch the spread under load.

---

What does a production HAProxy config look like?

HAProxy is the pick when you want active health checks and graceful reloads without a commercial tier:

global
    log /dev/log local0
    log /dev/log local1 notice
    maxconn 4096
    user haproxy
    group haproxy
    daemon

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    option  http-server-close
    option  forwardfor except 127.0.0.0/8
    option  redispatch
    retries 3
    timeout connect  5s
    timeout client   600s    # long for streaming
    timeout server   600s
    timeout http-keep-alive 30s
    timeout queue    60s

frontend ollama_frontend
    bind *:443 ssl crt /etc/haproxy/ollama.pem
    bind *:80
    redirect scheme https if !{ ssl_fc }

    # ACL: API key required
    acl has_valid_key hdr_reg(authorization) -i ^Bearer\ sk-team-[a-z0-9]+$
    http-request deny if !has_valid_key

    # Rate limit: 60 requests per minute per API key
    stick-table type string len 64 size 1m expire 1m store http_req_rate(60s)
    http-request track-sc0 hdr(authorization)
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 60 }

    default_backend ollama_backend

backend ollama_backend
    balance leastconn
    option httpchk GET /
    http-check expect status 200
    default-server inter 5s fall 3 rise 2 maxconn 100

    server ollama1 10.0.1.10:11434 check
    server ollama2 10.0.1.11:11434 check
    server ollama3 10.0.1.12:11434 check
    server ollama4 10.0.1.13:11434 check

frontend stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 5s
    stats admin if { src 10.0.0.0/8 }

Where HAProxy earns its extra config syntax

• Active health checks by default — inter 5s probes every five seconds, fall 3 marks a backend down after three failures, rise 2 needs two successes to bring it back. Open-source Nginx only does passive checks.
• Stick-tables give you rate limiting without an external store like Redis.
• The stats page at :8404/stats shows per-backend request rates, connection counts and response times with no extra tooling.
• Graceful reloads — -sf to the old PID hands off cleanly, so in-flight streams finish on the old process instead of being cut.
• option redispatch — if a backend fails mid-request and retries remain, send the retry to a different backend. This is what keeps rolling restarts from producing 5xx.

The trade-off is a config syntax that is more compact but harder to search for than Nginx's.

---

Do you need sticky sessions for streaming?

Usually not. Ollama's /api/generate and /api/chat are stateless from the load balancer's point of view: the client sends the full messages array on every call, so any backend can serve any request. There is no server-side conversation to pin to.

Two things do matter for streaming:

1. Buffering must be off. Covered in the Nginx config above. HAProxy does not buffer responses by default, so option http-server-close plus the long timeouts is enough.

2. Stickiness is a narrow optimisation, not a requirement. Pin a client to a backend only if you are relying on prompt-prefix cache reuse across turns, or on a per-backend cache in front of Ollama. Otherwise stickiness costs you balance: one heavy user pinned to one backend is exactly the imbalance least-connections exists to prevent.

If you do need it, in Nginx:

upstream ollama_backends {
    ip_hash;     # client IP based stickiness
    server 10.0.1.10:11434;
    server 10.0.1.11:11434;
}

In HAProxy:

backend ollama_backend
    balance source
    hash-type consistent
    server ollama1 10.0.1.10:11434 check

Note the failure mode: a sticky client that lands on a backend during a model swap stays there through the slow window. Enable it only when you can name the cache you are trying to preserve.

---

How do you route when backends hold different models?

Naive load balancing breaks the moment your backends are not identical. A request for a 70B model lands on a backend that only has an 8B loaded, and you get an on-demand pull — or an error.

LiteLLM Router solves this by knowing which backends serve which models:

pip install litellm[proxy]

config.yaml:

model_list:
  # Backends 1-3 serve the 8B chat model
  - model_name: chat
    litellm_params:
      model: ollama/llama3.1:8b
      api_base: http://10.0.1.10:11434
  - model_name: chat
    litellm_params:
      model: ollama/llama3.1:8b
      api_base: http://10.0.1.11:11434
  - model_name: chat
    litellm_params:
      model: ollama/llama3.1:8b
      api_base: http://10.0.1.12:11434

  # Backend 4 has the 70B model
  - model_name: chat-large
    litellm_params:
      model: ollama/llama3.1:70b
      api_base: http://10.0.1.13:11434

  # Coding model on backends 1-2 only
  - model_name: code
    litellm_params:
      model: ollama/qwen2.5-coder:14b
      api_base: http://10.0.1.10:11434
  - model_name: code
    litellm_params:
      model: ollama/qwen2.5-coder:14b
      api_base: http://10.0.1.11:11434

router_settings:
  routing_strategy: least-busy
  num_retries: 2
  timeout: 600
  cooldown_time: 30
  fallbacks:
    - chat-large: ["chat"]   # if 70B is down, fall back to 8B

litellm_settings:
  set_verbose: false
  drop_params: true

general_settings:
  master_key: sk-master-key-rotate-me
  database_url: "postgresql://litellm:pass@localhost/litellm"

Run it:

litellm --config config.yaml --port 8000

Clients then use the OpenAI SDK against http://lb-host:8000/v1 with model name chat, chat-large or code. LiteLLM handles routing, retries, fallback and per-key limits, and the same router can front hosted providers alongside Ollama if you need a hybrid. For the client side of an OpenAI-compatible endpoint, see our guide to the best Ollama clients.

---

What is the right health check for an Ollama backend?

Three layers, increasing in depth and cost:

1. TCP check (always on)

Both Nginx and HAProxy mark a backend down when the TCP connect fails. It is a free baseline and a weak signal — a process can accept connections while being stuck on inference.

2. HTTP check (the one to actually use)

GET / on port 11434 returns Ollama is running with HTTP 200 when the server is up. Prefer it over /api/tags, which does more work per call.

In Nginx:

location = /healthz {
    proxy_pass http://127.0.0.1:11434/;
    access_log off;
}

In HAProxy:

option httpchk GET /
http-check expect status 200

3. Inference smoke test (when the HTTP check is not enough)

An external probe that actually generates a token, so you find out whether the model is loadable rather than whether the HTTP listener is alive:

#!/bin/bash
# /usr/local/bin/ollama-deep-check.sh
RESPONSE=$(curl -s -m 10 -X POST http://localhost:11434/api/generate \
  -d '{"model":"llama3.1:8b","prompt":"ping","stream":false,"options":{"num_predict":1}}')
if [[ "$RESPONSE" == *"response"* ]]; then
    exit 0
fi
exit 1

Run it from the LB host on a timer and mark the backend down after several consecutive failures. Keep the interval well above the probe's own worst-case latency, or the check becomes the load.

---

How do you put TLS, auth and rate limits in front of it?

TLS termination

Terminate TLS at the load balancer. Ollama itself has no TLS support, so this is not optional if the endpoint is reachable by anything but localhost:

sudo certbot certonly --nginx -d ollama.internal.example.com
sudo systemctl enable certbot.timer  # auto-renewal

API key auth

Three patterns, simplest first:

Inline allowlist (small teams):

if ($http_authorization !~ "^Bearer (sk-team-eng|sk-team-product)$") {
    return 401;
}

Map-based (cleaner once you have many keys):

map $http_authorization $valid_key {
    default 0;
    "Bearer sk-team-eng-2026"     1;
    "Bearer sk-team-product-2026" 1;
}
server {
    if ($valid_key = 0) { return 401; }
}

External auth service: OAuth2-proxy or Pomerium as an ext-auth filter. Validates JWTs, supports key rotation without an Nginx reload, integrates with your IdP.

Rate limiting

Key choices:

• Per-API-key — fair across users, and the key is already required by the auth gate.
• Per-IP — a fallback for unauthenticated traffic, which arguably should not exist here at all.
• Per-route — protect /api/generate separately from cheap endpoints like /api/tags.

Nginx and HAProxy both limit on request rate only. If you need token-based or per-model quotas, that is a LiteLLM feature.

---

How many backends do you actually need?

Nobody else's benchmark predicts your cluster, because throughput moves with model size, quantization, prompt length, output length and card. What you can do before spending money is bound the answer with arithmetic. Here are the three formulas, stated so you can check them yourself.

1. How much VRAM the model takes.

model size (GB) ≈ 0.6 × parameters in billions      (at Q4_K_M)

An 8B model at Q4_K_M is about 4.8 GB; a 70B is about 42 GB. Add KV cache on top — and note the cache scales with OLLAMA_NUM_PARALLEL, because each concurrent slot keeps its own. This is why raising the parallel setting on a nearly-full card produces out-of-memory rather than more throughput.

2. The throughput ceiling for one backend.

tokens/sec ≤ memory bandwidth (GB/s) ÷ model size (GB)

Every generated token requires reading the whole model out of memory once, so bandwidth is a hard roof. Using NVIDIA's published specification of 1,008 GB/s for an RTX 4090 and the 4.8 GB figure above:

1,008 ÷ 4.8 ≈ 210 tokens/sec

That is an arithmetic upper bound for a single stream, not a measurement — real output lands well below it, because attention, sampling and framework overhead all cost time the formula ignores. Treat it as "this backend cannot possibly exceed X", which is exactly what you need for capacity planning.

Batching changes the shape usefully: with several sequences decoding together, one weight read serves all of them, so aggregate tokens/sec can exceed the single-stream ceiling — up to the point where the GPU's compute, not its bandwidth, becomes the limit.

3. How many requests can be in flight at once.

cluster slots = number of backends × OLLAMA_NUM_PARALLEL

Requests beyond that queue. This is the number that determines whether your users experience "slow" or "fine", and it is the reason horizontal scaling works: adding a backend adds slots and VRAM together.

Why least-connections rather than round-robin. Round-robin assumes every request costs the same. LLM requests do not — a one-line completion and a 2,000-token summary occupy a slot for wildly different durations, and the ratio between them is a property of your traffic, not a constant anyone can publish. Round-robin will happily hand a new request to a backend that is already grinding through four long generations. Least-connections looks at in-flight count instead, which is a live proxy for how busy each backend really is. Use least_conn in Nginx, leastconn in HAProxy, least-busy in LiteLLM.

Measure your own numbers. Point a load generator at one backend with your real prompt distribution, record tokens/sec and queue depth, then multiply. That single-backend measurement times N is a far better predictor than any published cluster benchmark, because it already contains your model, your quant and your prompts. For model-by-model memory planning before you get that far, our Ollama model RAM/VRAM table has the footprints.

---

What goes wrong most often?

1. Round-robin instead of least-connections. See above. Always least_conn / leastconn.

2. proxy_buffering left on. It is Nginx's default, and it silently converts your streaming endpoint into a batch one.

3. 60-second timeouts. Nginx's proxy_read_timeout defaults to 60s and AWS Application Load Balancers default to a 60-second idle timeout — either one cuts a long generation off mid-stream. Raise both.

4. Forgetting OLLAMA_HOST=0.0.0.0. Ollama binds to localhost by default, so the LB on another host simply cannot reach it.

5. No keep_alive on backends. Models unload after idle and every burst then pays a cold-start. Set OLLAMA_KEEP_ALIVE=24h on every backend.

6. TCP-only health checks. A hung Ollama still accepts TCP. Use the HTTP check at minimum.

7. Same model name, different versions across backends. Pin full tags (llama3.1:8b-instruct-q4_K_M, not llama3.1) and verify with ollama list on each host. Mismatched quants across a pool produce output quality that changes per request, which is a miserable bug to chase.

8. Missing client_max_body_size. Nginx's 1MB default rejects image inputs to vision models with a 413.

9. restart instead of reload. systemctl reload nginx is graceful; restart drops in-flight streams. HAProxy's -sf handoff does the same job.

10. No per-backend observability. Without per-backend request rate, latency and error rate you cannot tell which node is degrading. The Ollama production deployment guide has the Prometheus and Grafana setup.

Two references worth having open while you tune: the Nginx HTTP load balancing docs for the full algorithm list, and HAProxy's own write-up of load balancing strategies for how leastconn differs from the alternatives.

---

FAQ

Does Ollama support load balancing natively?

No. Ollama is a single-node inference server. To scale horizontally you put a reverse proxy (Nginx, HAProxy, Caddy) or a router (LiteLLM, Envoy) in front of several Ollama instances. Each instance keeps its own copy of the model in VRAM, and the load balancer is the piece that distributes traffic across them.

Should I use round-robin or least-connections for Ollama?

Least-connections. Round-robin assumes requests are equal cost, and LLM generations are not — a short completion and a long summary hold a slot for very different lengths of time. Least-connections routes each new request to the backend with the fewest in-flight requests, which tracks actual busy-ness. Nginx calls it least_conn, HAProxy calls it leastconn, LiteLLM calls it least-busy.

Do I need sticky sessions for Ollama load balancing?

Almost certainly not. Standard /api/generate and /api/chat calls are stateless — the client resends the whole conversation, so any backend can serve any request. Enable IP-hash or cookie stickiness only if you are deliberately relying on a per-backend cache, and accept that it degrades balance in exchange.

How do I handle backends that have different models loaded?

Three options. (1) Identical model set on every backend — simplest, scales linearly, wastes VRAM on models nobody uses. (2) Model-aware routing with LiteLLM, which knows which backend holds what. (3) Two-tier: hot models everywhere, specialty models on dedicated backends behind a path rule. Option 1 is fine until you have several models or run short on VRAM.

What is the right health check for Ollama backends?

GET / on port 11434 — Ollama returns Ollama is running with HTTP 200 while the server is alive. Prefer it over /api/tags, which does more work per probe. For a deeper check, periodically call /api/generate with a one-token prompt: that verifies the model actually loads, not just that the HTTP listener answers.

Can I use a cloud load balancer like ALB in front of Ollama?

Yes, but the idle timeout will bite you. AWS documents a 60-second default idle timeout on Application Load Balancers, which terminates any generation that runs longer. Raise it into the hundreds of seconds, keep HTTP/1.1 keep-alive on, and check that connection draining does not cut in-flight streams. Many teams still put Nginx or HAProxy behind the cloud LB because the streaming knobs are easier to reach.

How do I stop a freshly added backend from getting hammered cold?

Pre-warm it. Before adding a backend to the pool, send it a request per model you serve with a long keep_alive, and set OLLAMA_KEEP_ALIVE=24h so nothing unloads afterwards. Configure max_fails / fail_timeout so a backend that returns 5xx during warm-up is marked down rather than failing real user requests.

How many backends do I need for N users?

Work it from slots, not from someone else's throughput number. Cluster capacity in concurrent requests is backends × OLLAMA_NUM_PARALLEL; everything past that queues. Measure tokens/sec on one backend with your own prompts and model, then multiply — a single-backend measurement you took is a much better predictor than a published cluster benchmark, because it already reflects your quantization and prompt lengths.

---

Conclusion

Load balancing Ollama is the difference between a hobby install and a shared service. Two backends behind Nginx with least-connections gets you most of the way. Add HAProxy for active health checks and clean reloads, or LiteLLM when backends stop being interchangeable. Layer TLS, API keys, rate limits and a real health check on top and you have an internal AI gateway people can depend on.

The part nobody warns you about is that the load balancer config is the small half of the job. The rest is operational hygiene: pre-warming new backends, rolling restarts that do not drop streams, and per-backend metrics so you notice a degrading node before your users do.

When this stops being enough — roughly when you are managing more backends by hand than you want to — Kubernetes is the natural next step. The Ollama on Kubernetes guide picks up exactly where this one ends, with StatefulSets, autoscaling and ingress handling the above declaratively.

---

Want the next infrastructure deep dive (multi-region failover, GPU-aware Envoy filters, request-level cost attribution)? Subscribe to the Local AI Master newsletter — production playbooks, weekly.