Local AI Deployment: From Laptop to Production

Running a model and operating one are different jobs. A single command gives you a chat prompt on your own machine, and that experience is genuinely good. It is also why the next step catches people out. What that command started is a single-process, single-user, best-effort program with no admission control, no memory budget and no story for what happens when the host reboots at 3am.

The problem becomes easier to reason about when you see local deployment as three decisions stacked on one another, because each constrains the others.

Weight format and precision. Safetensors in BF16, a GGUF file at some quantization level, AWQ or GPTQ integer weights, or MLX-converted weights for Apple hardware. This choice is usually made casually, by downloading whatever the model card links to first, and it silently rules out most of your runtime options before you have started.

The inference runtime. llama.cpp, vLLM, SGLang, TensorRT-LLM, ExLlamaV2, MLX. Each reads a different subset of formats and each is tuned for a different shape of workload.

The serving layer. The HTTP surface, request queueing, batching policy, authentication, rate limits, timeouts, metrics, and the supervisor that brings the process back.

Two properties make this harder than deploying an ordinary web service.

The first is that memory does not scale the way you are used to. A stateless API server absorbs one more concurrent request with a small, bounded slice of memory. An LLM server absorbs it with however much its key/value cache requires, and that figure grows with the length of the conversation. Capacity planning stops being a CPU question and becomes an explicit allocation problem.

The second is that latency is not one number. Time to first token is dominated by the prefill pass, which is compute-bound and scales with prompt length. The gap between subsequent tokens is dominated by streaming weights out of memory once per step, which is bandwidth-bound and largely independent of how long the prompt was. Changes that improve one often damage the other: larger batches raise total throughput and per-request latency simultaneously. Optimize against a single averaged figure and you will degrade the experience while the dashboard reports an improvement.

There is a third property that gets less attention. The interesting failure mode is frequently not an error at all. A silently truncated context, a mismatched chat template or an over-aggressive quantization raises no exception. It produces slightly worse answers, and nothing in the logs mentions it.

Almost every capacity question about local inference reduces to two numbers that people conflate: the memory the weights occupy, which is fixed, and the memory the conversations occupy, which is not.

Weights are the easy half

Parameter count multiplied by bytes per parameter gives the floor. BF16 is two bytes per parameter, eight-bit integer formats are roughly one, and four-bit formats are roughly half a byte before overhead. Real quantized files come in above the nominal figure because modern schemes are mixed precision. A GGUF K-quant keeps attention and embedding tensors at higher precision than the feed-forward blocks, and the suffix in names such as Q4_K_M or Q5_K_S encodes which tensors received the extra bits. Read the file size on disk rather than trusting the multiplication.

The KV cache is what actually breaks you

Every token already processed leaves a key vector and a value vector behind in every layer, and those have to stay resident for as long as the sequence is alive. The size is the product of two (for key and value), the layer count, the number of key/value heads, the head dimension, bytes per element, the sequence length and the number of concurrent sequences. Three of those terms are yours to set at deploy time.

This produces the single most common local deployment surprise. A model that loads happily and answers a short question falls over on a long document, or on the fourth simultaneous user. The weights did not change. The cache did. Serving one person at two thousand tokens of context and serving eight people at thirty-two thousand tokens are genuinely different hardware requirements for the same file.

Two mitigations matter. Grouped-query attention, used by most current open-weight models, shares key/value heads across query heads and cuts cache size considerably. The reduction factor is not a mystery: it is the ratio of attention heads to key/value heads in the model's config file, so you can read it rather than estimate it. Separately, the cache itself can be quantized independently of the weights. Eight-bit or four-bit KV is its own decision with its own quality cost, usually smaller than the equivalent weight quantization, but not free.

What lower precision actually costs

Quantization damage is not uniform across capabilities. Fluent conversational output survives aggressive quantization far better than exact instruction-following, schema-constrained output, arithmetic and long-context recall. A model that still sounds articulate at three bits can quietly stop honoring a JSON schema, and if your application depends on parsing that output, the model has failed even though every sample you skim-read looks fine.

The llama.cpp project publishes perplexity comparisons across its quantization types, and those are the right reference for relative ordering. They are not a substitute for measuring on your own task, because perplexity on a general corpus is only loosely connected to whether your extraction prompt still returns parseable output. Writing a small task-specific check before you commit to a quantization level is the cheapest insurance in this entire subject.

There is no best runtime. There is a best match between the shape of your workload and the hardware in front of you.

Two ideas explain most of the throughput gap between the top and bottom of that table.

Continuous batching lets the server admit a new request into the running batch the moment any sequence finishes, instead of holding a fixed batch open until its slowest member completes. Because generation lengths vary enormously between requests, this alone changes the economics of multi-user serving.

Paged attention, introduced with vLLM by the Berkeley group that published the PagedAttention paper, stores the KV cache in fixed-size blocks rather than one contiguous reservation per sequence. Memory is no longer wasted on the difference between the context length you reserved and the context length the conversation actually used, and blocks can be shared between sequences with identical prefixes.

The decision tree is shorter than the debate suggests. One user, or a handful, on whatever hardware happens to be available: the llama.cpp family, with Ollama on top if you want model management handled for you. A team or an application with a genuine load pattern on NVIDIA: vLLM, accepting the operational weight that comes with it. A Mac: MLX or llama.cpp with Metal, planning around the fact that prompt processing rather than generation is the weak point. A fixed model where you control the whole stack and need the last increment of latency: TensorRT-LLM, and not before.

One warning about numbers found online. Throughput figures for local inference mean almost nothing without the batch size, prompt length, generation length, quantization, context limit, driver version and power limit attached, and they shift between releases of the same runtime. The only figure that describes your deployment is the one you produce on your hardware with your own prompt distribution. Treat published benchmarks as a hint about ordering and never as a capacity plan.

The problems that take a local deployment down are rarely the ones people prepare for.

Memory exhaustion under concurrency. The model loaded, so the weights fit. Then several long conversations arrive together and the cache does not. Runtimes handle this differently: vLLM reserves a fraction of the device up front and preempts or queues requests when blocks run out, while llama.cpp-based servers are more likely to fail the allocation outright. Neither behavior is wrong, but you need to know which one you have before it happens in front of users.

Cold starts and model thrash. Loading tens of gigabytes of weights from disk is slow. If your server unloads the model between requests, or if you route across several models on one GPU, the eviction cost dominates everything else. Ollama's keep-alive setting exists precisely for this, and leaving it at the default is a common cause of "it was fast yesterday".

Silent context overflow. When a conversation exceeds the configured context window, something has to go, and the default is often middle truncation. That is exactly where the retrieved documents live in most RAG applications, and sometimes where the system prompt ends up after a long exchange. The response is coherent and wrong.

Chat template mismatch. Every instruction-tuned model expects a specific arrangement of role markers and special tokens. Serve a model through a runtime that applies a different template and you get output that is subtly off: ignored system prompts, invented turns, refusal loops. It looks like a bad model. It is a bad wrapper.

Streaming that dies at the proxy. Server-sent events pass through reverse proxies badly by default. Response buffering in nginx will hold the whole generation until it completes, which converts a nicely streaming interface into a long stare at a blank box, and aggressive idle timeouts will cut long generations off entirely.

Orphaned generations. A client disconnecting does not necessarily stop the work. Without cancellation wired through, an abandoned browser tab can keep a slot occupied for the full generation length, which is a quiet capacity leak under load.

Reproducibility that is not. Even at temperature zero, output can differ between runs because floating-point reduction order changes with batch composition, and because model tags get republished. Pin digests, not tags, if you want the thing you validated to be the thing you serve.

Thermal and power limits. Laptops, small-form-factor builds and dense multi-GPU rigs all throttle. Sustained generation is a very different thermal load from a burst, and a rig that looks quick in a short test may not hold that rate over a long job.

The metrics worth exporting from day one are time to first token, inter-token latency, queue depth, KV cache utilization, prompt and completion token counts per request, and the rate at which requests are rejected or preempted. Average response time on its own tells you nothing useful about an LLM server.

Once a model stops fitting on one accelerator, or once the accelerator is not an NVIDIA card, the shape of the problem changes.

More than one GPU

There are three distinct things people mean by multi-GPU, and they are not interchangeable.

Layer offload, which llama.cpp and Ollama do by default, splits the model's layers across devices and runs them in sequence. This is what makes an oversized model fit at all, but a single request only ever occupies one device at a time, so it buys capacity rather than speed.

Pipeline parallelism formalizes that split and keeps every stage busy by having several requests in flight at different stages. It raises throughput and does nothing for single-request latency.

Tensor parallelism splits individual matrices across devices so all of them work on the same token simultaneously. It is the option that genuinely reduces latency, and it is the one with real constraints: it needs high interconnect bandwidth, so a rig with cards on narrow PCIe lanes may see gains disappear into communication overhead, and the parallel degree has to divide the model's attention head count cleanly.

Mixed VRAM sizes complicate everything, since most splitting schemes assume uniform devices and end up bounded by the smallest card. Power is its own trap: several GPUs drawing transient peaks simultaneously can trip a supply rated comfortably above their nominal combined draw.

Apple Silicon

Unified memory changes the calculation entirely. There is no separate VRAM pool, so a large fraction of system memory is addressable by the GPU, and models that would need a datacenter card on a PC fit on a well-specified Mac. The limiting factor moves to memory bandwidth for generation and to compute for prefill, which is why Macs feel comparatively strong on long conversational generation and comparatively weak when you paste in a large document. MLX targets the hardware directly; llama.cpp with Metal is the more portable option. The wired memory limit is adjustable, and knowing that is often the difference between a model loading and not.

Edge and embedded

At the edge, the constraint set inverts. Power envelope, thermal headroom, and the absence of swap matter more than raw throughput. Storage is frequently flash with finite write endurance, so logging habits from server work become a hardware problem. Updates may have to work offline. Most importantly, the model selection changes: a small model with a narrow task definition and constrained structured output will beat a general chat model that barely fits, because the failure you cannot afford at the edge is the one nobody is present to notice.

A reasonable self-check before spending time here. Can you explain, roughly, why a seven-billion-parameter model at four bits needs less memory than the same model at sixteen bits, and why that difference is not the whole story? Have you already hit an out-of-memory error you could not explain? Do you have a use case where sending text to a hosted API is genuinely a problem, whether for contractual, regulatory, latency or cost reasons? Two yes answers is enough. Three means the material will pay for itself quickly.

If none of those apply, a hosted API is very likely the right answer for now, and there is no shame in it. Local deployment earns its keep when data cannot leave, when per-token pricing has become the dominant cost line, when the workload is embarrassingly steady rather than spiky, or when you need the system to keep working without a network. It costs you the operational burden that the API provider was absorbing on your behalf.

The material is also useful in reverse. Understanding prefill against decode, batching and cache pressure makes hosted API behavior legible: why a long system prompt is expensive on every call, why prompt caching exists, why streaming changes perceived latency more than actual latency.

Three adjacent skills compound with this one. Evaluation comes first, because without a task-specific test you cannot tell whether a quantization level, a runtime change or a model upgrade helped or hurt, and every optimization decision in this field is a trade. Retrieval comes second, since most useful local deployments end up feeding private documents into the context and the retrieval quality dominates the model choice more often than people expect. Security comes third: the moment a local model is reachable over the network and holds a tool, you have inherited a different threat model.

The course carries a public GitHub repository, so the configuration files, serving scripts and quantization workflows are readable before you commit to anything, and reusable afterwards.