Private OpenAI-Compatible API: Self-Hosted Setup

Published April 23, 2026 · Updated August 23, 2026

"OpenAI-compatible" means a server that speaks the same HTTP contract as api.openai.com — /v1/chat/completions, /v1/embeddings, /v1/models, same JSON shapes, same SSE streaming. Ollama exposes that contract natively on port 11434, so any OpenAI client connects by changing two lines: base_url and api_key. Put LiteLLM in front when you need per-user keys, budgets and an audit trail, and vLLM underneath when you need real concurrency. Nothing in your application code has to change.

Teams usually reach this point for one of two reasons: an API bill that has stopped being a rounding error, or a security review that has landed on somebody's desk. Either way the codebase is full of openai.chat.completions.create(...) calls and rewriting them all is not the answer. Pointing them at an endpoint you control is.

By the end of this guide you have an API at https://ai.yourcompany.com/v1 that any OpenAI client can hit, backed by your hardware, with per-user keys, rate limits and an audit trail. Three deployment shapes are covered — Ollama alone, LiteLLM in front of Ollama, and LiteLLM in front of vLLM — along with the tradeoffs between them.

Why run your own OpenAI-compatible API?

Three reasons come up repeatedly:

1. Cost. Hosted inference is priced per token; your own hardware is priced once. Whether that trade works depends on volume, and it is arithmetic — the break-even formula is in the performance section, and Ollama vs ChatGPT API cost at scale works through the same maths in more detail.
2. Compliance. GDPR, HIPAA and SOC 2 all treat third-party processors as a risk to be documented and controlled. An endpoint inside your own network removes the data egress rather than papering over it. See GDPR-compliant local AI for the shape of that argument.
3. Vendor independence. Pricing changes, deprecation notices and rate limits are not yours to control when the endpoint is not yours. Model swaps behind a stable interface are.

What makes this practical is that the OpenAI HTTP contract has become a de facto standard. Ollama, vLLM, llama.cpp's server, LM Studio and most hosted providers all expose something that behaves like /v1/chat/completions. Building against that contract means your client code does not care which backend wins next year.

Architecture: three layers

A production-grade private API has three distinct layers. Skipping any one of them creates an outage waiting to happen.

┌──────────────────────┐
│ Edge layer           │  TLS + WAF + per-key auth
│ nginx / Caddy        │  rate limiting, IP allowlist
└──────────┬───────────┘
           │
┌──────────▼───────────┐
│ Gateway layer        │  Virtual keys, per-team budgets
│ LiteLLM              │  Routing, fallbacks, retries
│                      │  Audit logging to Postgres
└──────────┬───────────┘
           │
┌──────────▼───────────┐
│ Inference layer      │  Ollama  (general purpose)
│                      │  vLLM    (high throughput)
│                      │  llama.cpp (max efficiency)
└──────────────────────┘

The edge layer is non-optional. The gateway layer is optional for a single team and mandatory the moment two teams share infrastructure. The inference layer is where the cost and throughput tradeoff lives.

What is the smallest thing that works?

Single host, single user — useful for learning the contract.

# 1. Install Ollama on a server you control
curl -fsSL https://ollama.com/install.sh | sh

# 2. Pull a model
ollama pull llama3.1:8b

# 3. Bind to all interfaces (NOT public — see below)
sudo systemctl edit ollama.service
# Add:
#   [Service]
#   Environment="OLLAMA_HOST=0.0.0.0:11434"
#   Environment="OLLAMA_KEEP_ALIVE=24h"
sudo systemctl restart ollama

# 4. Test the OpenAI-compatible endpoint
curl http://YOUR_HOST:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role":"user","content":"hello"}]
  }'

That endpoint speaks the OpenAI dialect. Any SDK or tool that accepts a baseURL connects to it. Do not expose port 11434 to the internet directly. Bind to a private network, or put nginx in front with TLS and a token check before anything is reachable externally. This Quick Start is the inner core; the next section wraps it correctly.

How do I build the production stack?

The configuration below suits teams of up to roughly 50 users sharing a couple of GPU boxes.

1. Run Ollama on a private interface

Edit /etc/systemd/system/ollama.service.d/override.conf:

[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_KEEP_ALIVE=24h"
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_MAX_LOADED_MODELS=2"

OLLAMA_NUM_PARALLEL=4 lets a single model serve four concurrent streams; OLLAMA_MAX_LOADED_MODELS=2 keeps two models resident if you have the VRAM headroom. Both settings cost memory — parallel slots each need their own KV cache — so raise them only as far as your card allows.

2. Install and configure LiteLLM

pip install "litellm[proxy]"

Create config.yaml:

model_list:
  - model_name: gpt-4o-mini   # the alias clients use
    litellm_params:
      model: ollama/llama3.1:8b
      api_base: http://127.0.0.1:11434

  - model_name: gpt-4o        # route the heavier alias to a bigger model
    litellm_params:
      model: ollama/qwen2.5:32b
      api_base: http://127.0.0.1:11434

  - model_name: text-embedding-3-small
    litellm_params:
      model: ollama/nomic-embed-text
      api_base: http://127.0.0.1:11434

litellm_settings:
  drop_params: true            # silently drop unsupported params
  set_verbose: false
  cache: true
  cache_params:
    type: redis
    host: 127.0.0.1
    port: 6379

general_settings:
  master_key: sk-master-CHANGE-ME
  database_url: "postgresql://litellm:pass@127.0.0.1:5432/litellm"
  ui_username: admin
  ui_password: CHANGE-ME

The model_name field is what clients send. Aliasing gpt-4o-mini to llama3.1:8b means every existing line of code that says model="gpt-4o-mini" keeps working. No client edits required.

litellm --config config.yaml --port 4000

3. Issue per-user keys

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-master-CHANGE-ME" \
  -d '{
    "models": ["gpt-4o-mini", "text-embedding-3-small"],
    "max_budget": 5.00,
    "duration": "30d",
    "metadata": {"user_id": "engineer-42", "team": "platform"}
  }'

You get back sk-... keys that look exactly like OpenAI keys. Distribute them through your secrets manager.

4. Put nginx in front with TLS

server {
  listen 443 ssl http2;
  server_name ai.yourcompany.com;
  ssl_certificate     /etc/letsencrypt/live/ai.yourcompany.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/ai.yourcompany.com/privkey.pem;

  # 60s headers; long body for streaming completions
  proxy_read_timeout 600s;
  proxy_buffering off;

  location / {
    proxy_pass http://127.0.0.1:4000;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
  }
}

Issue the cert with certbot --nginx -d ai.yourcompany.com. The full hardening recipe (fail2ban, log rotation, healthchecks) is in Ollama in production.

5. Point any OpenAI client at it

from openai import OpenAI
client = OpenAI(
    base_url="https://ai.yourcompany.com/v1",
    api_key="sk-engineer-42-XXXXXXXX",
)
r = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role":"user","content":"summarize: ..."}],
    stream=True,
)
for chunk in r:
    print(chunk.choices[0].delta.content or "", end="")

That same code worked against api.openai.com yesterday. Today it runs on your hardware.

When should I swap Ollama for vLLM?

Ollama is excellent for development and small teams. Its concurrency is bounded by OLLAMA_NUM_PARALLEL and by how many KV caches fit in VRAM, so throughput plateaus early. For high-concurrency production — dozens of simultaneous users on one model — vLLM is built for a different job: PagedAttention manages the KV cache in pages rather than contiguous blocks, so many requests share GPU memory efficiently and continuous batching keeps the GPU busy between them. The vLLM project publishes its own throughput comparisons in the repository — read those rather than any second-hand multiplier, including one in a blog post.

Stand it up beside Ollama:

pip install vllm
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.92

Add it to LiteLLM:

  - model_name: gpt-4o-mini-fast
    litellm_params:
      model: openai/Meta-Llama-3.1-8B-Instruct
      api_base: http://127.0.0.1:8000/v1
      api_key: any

The costs of that move are real: vLLM wants the model in HuggingFace format, expects full GPU residency with no CPU offload, and gives up Ollama's one-command model management. That is why Ollama stays the default for mixed and low-traffic workloads, and vLLM earns its place only once concurrency is the binding constraint.

How do I add auth, rate limits and audit logs?

The three controls a security review will ask about:

Authentication. LiteLLM virtual keys (sk-team-...) bound to specific models, with expirations and budgets. The master key is for administration only and never belongs in client code. Put MFA in front of the LiteLLM admin UI by proxying it through Cloudflare Access or Tailscale.

Rate limiting. Per-key requests-per-minute and tokens-per-minute, configured in config.yaml:

  - model_name: gpt-4o-mini
    litellm_params: { model: ollama/llama3.1:8b, api_base: http://127.0.0.1:11434 }
    rpm: 60                  # 60 requests per minute per key
    tpm: 100000              # 100K tokens per minute per key

For burst control beyond what LiteLLM offers, layer nginx's limit_req module in front. The Ollama rate limiting guide goes deeper on multi-tenant patterns.

Audit logging. LiteLLM writes to Postgres by default. Enable Langfuse for prompt-level tracing:

litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]

Set LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_HOST (self-hosted Langfuse for full data residency). Every request, response, latency, token count and user id becomes queryable in one place.

Budget for the disk. Logging full request and response bodies at an average 10 KB per request and 10,000 requests a day is 100 MB a day — about 9 GB in three months, before indexes. Decide up front whether you retain bodies or only metadata, because changing your mind later means a retention policy applied retroactively to data you already promised to keep.

Ollama, LiteLLM or vLLM?

How fast will it be, and when does it pay off?

Both questions are arithmetic. Do them before you buy anything.

Generation speed has a hard ceiling set by memory bandwidth. Producing one token requires reading every weight out of memory once, so:

tokens/second ceiling = memory bandwidth (GB/s) / model size in memory (GB)

Model size at Q4_K_M is roughly 0.6 GB per billion parameters. An 8B model is about 4.8 GB; a 14B is about 8.4 GB; a 70B is about 42 GB. Worked example: an RTX 4090 is specified at 1,008 GB/s, so an 8B model tops out at 1008 ÷ 4.8 ≈ 210 tokens/second for a single stream.

Those are ceilings, not forecasts — attention overhead, prompt processing and sampling all pull real output below them, frequently by a third. Their value is elimination: a configuration whose ceiling is 2 tokens/second will never feel like an API, no matter how it is tuned. Note also that the ceiling applies per stream; batching many requests raises total throughput well above it, which is precisely what vLLM is optimised for.

Latency to first token is usually lower than a hosted API for the simple reason that there is no internet hop. Prompt processing then scales with input length, so long-context requests are dominated by the prompt, not the answer.

Break-even is one division:

break-even months = hardware cost / (current monthly API spend - monthly electricity)

monthly electricity = (average watts / 1000) x 720 hours x price per kWh

A GPU box averaging 250 W at $0.15/kWh costs (250 ÷ 1000) × 720 × 0.15 = $27/month to run. So $1,500 of hardware against a $500/month API bill breaks even in 1500 ÷ (500 − 27) ≈ 3.2 months; against a $100/month bill, 1500 ÷ 73 ≈ 21 months. Put your own invoice into the numerator — the reason this is presented as a formula rather than a table is that the answer swings by an order of magnitude with volume, and anyone quoting you a single payback figure has guessed at yours.

The OpenAI HTTP contract is documented at platform.openai.com/docs/api-reference — every endpoint LiteLLM and Ollama implement matches that schema.

What goes wrong?

• Binding 0.0.0.0 without a firewall. Internet-wide port scanners index open services continuously; an Ollama instance on a public IP is discoverable by anyone running one, and it will be used. Bind to localhost or a private interface, always.
• Forgetting model aliases. If your client sends gpt-4o-mini and your config does not alias it, LiteLLM returns a 400. Set drop_params: true and define every alias your code paths use.
• Streaming buffering at nginx. Without proxy_buffering off the user waits and then receives the whole response at once instead of watching it stream. Always disable buffering for AI endpoints.
• No connection limits in nginx. A single client opening 200 streaming connections starves everyone else. Add limit_conn_zone and limit_conn perip 10.
• Underprovisioned VRAM for embeddings. Embedding models share GPU memory with chat models. Either keep a CPU-only embedding model (nomic-embed-text runs fine on CPU) or use a second GPU.
• Cache leaks between users. LiteLLM's cache key does not include the API key by default. For multi-tenant deployments set cache_params.namespace: "{api_key}" so a query from user A never returns user B's cached response.
• Token accounting drift. Self-hosted servers count tokens with their own tokenizer; OpenAI counts with tiktoken. Server-side budgets will not match client-side estimates. Document the delta or field tickets about it every Friday.

Common questions

What does "OpenAI-compatible" actually mean? The server speaks the same HTTP contract as api.openai.com — /v1/chat/completions, /v1/completions, /v1/embeddings and /v1/models, with matching JSON shapes and SSE streaming. Any client written for OpenAI (the official SDKs, LangChain, LlamaIndex, Continue.dev, the Vercel AI SDK) connects by changing only the base URL and key.

Should I use Ollama, LiteLLM or vLLM? Ollama for single-machine, single-team setups under about ten users. vLLM when you have a dedicated GPU server and concurrency is the constraint. LiteLLM as the gateway in front of either — it is what adds auth, rate limits, virtual keys and per-team budgets. Most production deployments run LiteLLM in front of Ollama or vLLM rather than instead of them.

Does Ollama need LiteLLM to be OpenAI-compatible? No. Ollama exposes /v1/chat/completions natively on port 11434. LiteLLM is only needed for a single endpoint fronting several backends, multi-tenant keys, or per-user rate limits. For a hobby setup, Ollama alone is enough.

How do I add real authentication? Ollama ships no auth at all. Put nginx or Caddy in front with an Authorization header check, or run LiteLLM, which provides per-key authentication, expirations and budgets out of the box. Never expose port 11434 to the internet directly.

Can I keep audit logs of every prompt and response? Yes — LiteLLM logs to Postgres, MongoDB, OpenTelemetry, Langfuse or Helicone. For an Ollama-only setup, add logging middleware in nginx that captures request and response bodies, and size the disk with the arithmetic above before you turn it on.

How fast is this compared to a hosted API? Single-stream generation speed is capped by memory bandwidth divided by model size — the table in the performance section gives the ceiling for common hardware. Time to first token is typically better than a hosted API because there is no network hop. Aggregate throughput across many concurrent users is where hosted providers still win unless you add GPUs, which is the gap vLLM's batching is designed to close.

Will my existing OpenAI-based app work without changes? Usually, yes — change base_url and api_key in the client constructor. What breaks: organization headers, dedicated-capacity features, vision and audio modalities that only exist on specific hosted models, and anything depending on a particular model's exact reasoning behaviour. Llama, Mistral and Qwen instruct models are the usual drop-in replacements for small hosted models.

Is this safe for SOC 2, HIPAA or GDPR workloads? A self-hosted OpenAI-compatible API is the practical way to keep prompts and responses inside your compliance boundary. You still need TLS in transit, encryption at rest (LUKS or ZFS), access logging, MFA on the bastion and a documented retention policy. The compliance-relevant controls sit below the model, in your infrastructure.

What you actually get

• A single base URL that any OpenAI client speaks, including tools you did not write (Continue.dev, Cursor, Cline, Open WebUI, LangChain, LlamaIndex, the Vercel AI SDK).
• A virtual key system with budgets, expirations and an admin UI.
• A complete audit trail of who asked what, with token counts and latencies.
• The option to swap models invisibly — replace llama3.1:8b with qwen2.5:14b tomorrow and zero clients break.
• Compliance-friendly architecture: no data leaves your boundary, every request is logged, every key is revocable.

If your team currently shares one API key, this stack is a strict upgrade regardless of the economics. If your bill is large enough that the break-even formula above returns a small number, it is also cheaper. And if you have a security review coming, it is the architecture that gives you the controls auditors actually ask for.

The next thing to build on top is retrieval over your private documents — once the API surface is yours, embeddings, retrieval and prompt assembly all happen inside your network. Pair this with the private AI knowledge base guide or Ollama semantic search and you have replaced a per-seat enterprise assistant with infrastructure you own.