Ollama Mastery: From Install to Daily Driver

Ollama bundles three separate things behind one command, and most of the confusion people have with it comes from not knowing which of the three is misbehaving.

The first is a package manager. Models are stored content-addressed: a manifest for each tag points at blob layers, so two tags that share a base do not store the weights twice, and an interrupted download resumes instead of restarting. That design also explains a common surprise — deleting a model sometimes frees less disk than expected, because a blob it shared with another tag stays behind.

The second is an inference runner. For most of its life Ollama has used llama.cpp underneath, which is why models arrive as GGUF files and why llama.cpp concepts leak straight through into Ollama's behavior: layer offload, k-quants, flash attention, KV cache handling. More recent releases added an in-house engine for some model families, particularly multimodal ones, so the runner you get is not always the one you assume from an old blog post.

The third is an HTTP server, and this is the part people forget exists. The moment Ollama is installed there is a daemon listening on a port, holding models resident in memory, and answering anything that can reach it. The CLI is a client to that server. Nearly every "it works in the terminal but not in my app" question resolves once that separation is clear.

Three jobs it is wrong for

Serving many users at once. Ollama is built around one person, or one application, at a time. It does not implement the continuous batching and paged key-value cache that vLLM and SGLang exist to provide. Point a room full of concurrent users at a single GPU running Ollama and you get a queue, not throughput. That is not a defect; it is a different category of software.

Weights that are not GGUF. If a model ships as AWQ or GPTQ for a GPU serving stack, or you want an ExLlama kernel, or you want a native MLX pipeline on Apple hardware, you are outside the format Ollama is organized around. Conversion is sometimes possible and frequently not worth the afternoon.

Training anything. Ollama can consume a LoRA adapter through a Modelfile. It cannot produce one. Fine-tuning belongs to an entirely different toolchain, and any course that blurs that line is teaching you to reach for the wrong tool under deadline.

Knowing the boundary early prevents the expensive class of mistake: building a product on a runtime that was never designed for the load you are about to put on it.

Three things compete for the same memory, and only one of them is the model.

Weights come first, and the arithmetic is simple enough to do in your head. Parameter count multiplied by bytes per parameter: a four-bit quantization stores roughly half a byte per weight, an eight-bit one roughly a byte, sixteen-bit floating point two bytes. Run that multiplication for the model you are eyeing before you start the download. It tells you more than any recommendation thread will.

The key-value cache comes second, and it is the part people forget. Every token held in context keeps a key vector and a value vector for every layer, and that memory is reserved according to the context length you configured, not the length of the conversation you have had so far. Doubling the context roughly doubles the cache. Architectures using grouped-query attention need dramatically less of it than older designs at the same context length, which is one reason two models with identical parameter counts can behave completely differently on the same card.

Runtime overhead comes third: the compute context itself, activations during a forward pass, and whatever the front end holds. Individually small, collectively enough to turn a model that "should just fit" into one that does not.

What happens when it does not fit

Ollama does not refuse. It splits — some layers on the GPU, the remainder on the CPU. The model still answers, which is precisely why people fail to notice: they conclude the hardware is weak when in fact only part of the work ever reached the accelerator. ollama ps prints that split as a percentage, and checking it before forming any theory about performance is the single most useful habit in this course.

CPU-only inference is genuinely usable at small model sizes and genuinely painful at large ones, and the reason is bandwidth rather than arithmetic. Producing each token requires streaming the active weights out of memory. System RAM delivers a fraction of the bandwidth a discrete GPU's dedicated memory does — both figures are published by the manufacturers, and the gap is wide enough that the difference is qualitative rather than marginal.

Disk is the quiet constraint. Models are large, they all land in one directory, and on Windows that directory defaults to the system drive. OLLAMA_MODELS relocates the store, and setting it before the first pull is far easier than migrating a blob directory afterwards.

A tag looks like a version string and behaves like a contract. Everything after the colon determines how much memory the file needs, how fast it runs and how good the output is, and none of that is obvious from the name.

Pulling :latest is the most common first mistake. It resolves to whatever the publisher pointed it at, which is usually a mid-sized variant at a middling quantization — a sensible default for a machine with plenty of memory and an instant disappointment on a small card. Naming the size and the quantization explicitly costs three extra keystrokes and removes the guesswork.

What the suffixes mean

Quantization suffixes such as q4_K_M, q5_K_M and q8_0 describe how the weights were compressed. The number is the nominal bit width. The K-series formats mix precision across tensor types rather than applying one width uniformly, and the trailing S, M or L indicates how aggressive that mixing is. Higher bit widths cost memory and bandwidth, and because decoding is bandwidth-bound, they cost speed as well. There is a genuine trade-off here, it is task-dependent, and anyone quoting a single universal "best quant" is guessing.

Beyond size and quantization, the library mixes several kinds of artifact that are not interchangeable:

• Instruct versus base. Base models continue text. Instruct models follow instructions. Asking a base model a question and being unimpressed by the answer is a category error.
• Reasoning models emit an internal thinking trace before the answer. Handling that trace correctly is a front-end and API concern, and ignoring it produces output that looks like the model is rambling.
• Vision models need a projector component alongside the language weights, and will fail in confusing ways if the front end sends images the runner is not configured to accept.
• Embedding models produce vectors, not conversation. They will not chat, and no amount of prompting fixes that.

The command that prevents most of this

ollama show reports the architecture, parameter count, quantization, the context length the file was built for, and the chat template and parameters baked into the manifest. Reading that output takes a few seconds and routinely prevents a download measured in tens of gigabytes.

The chat template deserves particular attention. Each model family expects its turns wrapped in a specific structure, and a mismatch between the template and the way a client formats requests produces output that reads like a broken model when the model is fine. When something behaves strangely, compare the template before blaming the weights.

A Modelfile is a short recipe that produces a new local tag from an existing one. FROM names the parent. SYSTEM bakes in a system prompt. PARAMETER sets sampling and runtime defaults. TEMPLATE overrides the prompt structure. ADAPTER attaches a LoRA. MESSAGE seeds example turns. ollama create compiles the lot into a tag you can run like any other.

The obvious use is convenience: stop pasting the same system prompt into every conversation. The more important use is control. Parameters set in a Modelfile become defaults for every caller, including the editor extension or automation script that has no settings panel of its own. If you want a code assistant that is deterministic and a writing assistant that is not, two Modelfiles are the clean way to get there.

num_ctx is the parameter that surprises people

Ollama applies a default context length that is smaller than the maximum most modern models advertise, and that default has moved between releases. Rather than trusting a figure from a tutorial, check the model with ollama show and set the value deliberately. Two consequences follow from raising it, and only one is the benefit:

• The model can hold more of the conversation, which is what you wanted.
• The key-value cache is reserved up front, so memory is consumed before you type a single word. Push it too far and you tip a model that fit into a CPU split, at which point the longer context has made the assistant worse rather than better.

This is the mechanism behind "the model forgets what I told it". The context has not been forgotten so much as never held. Older turns fall out of the window silently, and nothing in the interface announces it.

The other parameters worth understanding

temperature, top_p, top_k and min_p shape which token gets chosen from the distribution. repeat_penalty discourages repetition and interacts badly with structured output, where legitimate repetition — closing braces, recurring field names, indentation — is exactly what you need. num_predict caps output length. stop sequences terminate generation and are how you keep a model from continuing past the answer into an imagined next question.

Finally, keep-alive. A loaded model occupies memory until an idle timeout unloads it, after which the next request pays a load from disk again. OLLAMA_KEEP_ALIVE controls that window. Anyone timing responses without accounting for a cold load is measuring their storage device, not their model.

The most common support question about Ollama is some version of "it is running on the CPU and I do not know why". The causes are boringly repetitive.

• The runtime is invisible to the service. A driver installed for your user session, a CUDA or ROCm runtime the daemon cannot see, or a service account without device permissions. The model runs; the accelerator is simply not in the picture.
• Container isolation. Docker without the NVIDIA container toolkit and the appropriate device flags gives you a perfectly functional CPU container.
• WSL without GPU passthrough, or a Windows install where the service and the shell disagree about environment variables.
• A service unit that does not inherit your shell. Variables exported in a terminal are invisible to a systemd-managed daemon. On Linux that needs a unit override; on macOS a launch agent setting; on Windows the system environment panel and a restart.
• An AMD card outside the supported matrix. The silicon is often capable; the support list is the constraint, and the workaround paths are unofficial.
• Not enough free memory. The runner falls back to a partial or complete CPU split rather than erroring, so the symptom is slowness rather than failure.

Diagnose in this order

Start with ollama ps to see the actual GPU/CPU split — this converts a vague complaint into a measurable fact. Then read the server log, which records which device the runner selected and why. Only then reach for the vendor tool, watching utilization while a generation is in flight rather than while the machine is idle.

Speed levers, ranked by how much they matter

1. Fit the whole model in accelerator memory. Nothing else on this list comes close. A smaller model entirely on the GPU almost always beats a larger model split across GPU and CPU, because every token that touches system memory crosses the bus.
2. Drop one quantization level if that is what fitting requires. The quality difference between neighboring levels is usually smaller than the difference between resident and split.
3. Reduce the context to what you actually use. Reserved cache you never fill is memory you could have spent on layers.
4. Enable flash attention and cache quantization where the build and the model support them.
5. Then, and only then, consider hardware. Most people who conclude they need a bigger card have not yet done steps one to four.

Ollama binds to the loopback interface by default, which is a deliberate and sensible choice. The moment you change that binding to reach it from a laptop, a phone or a container on another host, you should understand exactly what you have published.

There is no authentication. No API keys, no user accounts, no rate limiting, no per-caller isolation. Anyone who can route a packet to the port can list your models, load them, generate with them and consume the memory of the machine. Internet-wide scanning for exposed inference endpoints is a documented, ongoing phenomenon, not a hypothetical, and an accidentally public instance is found quickly.

Patterns that work

• Keep the bind on loopback and reach it over a private overlay network. A WireGuard tunnel or a mesh VPN gives you device-level identity and encryption, and the server keeps its safe default.
• Put a reverse proxy in front. Terminate TLS, require an API key or an identity provider, and forward to loopback. This is also where rate limits and request logging belong.
• Bind to a specific private interface behind a firewall rule you have actually tested from another machine. An untested rule is a belief, not a control.

The cross-origin case catches out anyone building a browser front end: requests from a web page origin are rejected until that origin is allowed, and the resulting 403 is easy to misread as an authentication error in a system that has no authentication.

Two things people underrate

Pulling a model is a supply-chain action. A model brings a chat template and a system prompt that will execute on your machine as part of every conversation. Preferring publishers you can identify is basic hygiene, and it costs nothing.

Concurrency is a resource decision. Environment variables govern how many requests run in parallel and how many models stay loaded simultaneously. Left unbounded on a single GPU, several resident models will evict each other repeatedly and every user will experience the thrash as unexplained latency. Setting explicit limits produces slower peak numbers and a far better experience.

Errors look varied and are not. Almost everything clusters into five families, and recognizing the family is most of the fix.

Connectivity. Connection refused on the API port means one of three things: the service is not running, it is bound to an interface you are not calling, or you are inside a container addressing the host incorrectly. Check the process, then the bind address, then the network namespace, in that order.

Memory. The runner process terminating part-way through loading is nearly always an out-of-memory condition, on the accelerator or on the host, and the message almost never says so. If a very small model loads and a large one does not, you have your answer and no further debugging is warranted.

Compatibility. An unknown model architecture means your Ollama build predates the model you are pulling. New architectures require runner support; no configuration change substitutes for it. Update before investigating anything else, because a stale build produces failures that look like a dozen unrelated problems.

Storage and transfer. Stalled pulls, digest mismatches and permission errors on the model directory. A digest mismatch usually means a corrupted or partial blob rather than anything sinister; removing the affected blob and re-pulling resolves it. Permission errors typically mean the service account cannot read a directory your user owns.

Environment. Variables set in one context and read in another. This deserves its own family because it produces the most misleading symptoms: settings that clearly work when you run the binary by hand, and clearly do not when the daemon runs it.

A method that generalizes

Reproduce with the smallest model you have. That single step separates installation problems from capacity problems, and those two categories have almost nothing in common. If the small model works, stop reinstalling and start counting memory. If it does not, stop tuning parameters and fix the installation.

Then read the server log rather than the client output. The CLI reports that generation failed; the server records which device was selected, how many layers were offloaded, and what the runner said before it died. Almost every genuinely confusing Ollama problem becomes obvious within ten lines of that log, and learning to read it is worth more than memorizing any list of error strings.