Ollama Out of Memory: CUDA and RAM Crash Fixes

Every one of these errors is the runner asking for more memory than the machine will hand over at that instant — and the size of the ask is under your control, so almost none of them need new hardware. Work down six levers, cheapest first: unload whatever is still resident, cap OLLAMA_MAX_LOADED_MODELS, shrink the context you are actually paying for (num_ctx multiplied by OLLAMA_NUM_PARALLEL), quantise the KV cache with OLLAMA_KV_CACHE_TYPE=q8_0, push layers off the GPU with num_gpu, and only then drop a quant level. One symptom jumps the queue entirely: if the model loaded fine an hour ago and fails now, nothing about the model changed and lever one is your answer.

This page is for unbreaking a model that has already run once. If you have never got it running and want to know what your machine can hold, that is a different question with a better answer elsewhere: the Ollama model RAM and VRAM table lists what each model needs, the VRAM calculator does the arithmetic for a specific card, and Ollama system requirements covers the headroom per tier. Nothing below repeats those numbers.

Every error string, environment variable and default here is read from Ollama's source on main, its official docs, or a linked GitHub issue. No figure is quoted that we did not take from one of those.

Which out-of-memory error do you actually have

They are not interchangeable. Two of them come from the GPU driver, one is Ollama's own pre-flight refusal, and one is the operating system killing the process from outside.

The relabelling in row four is worth understanding, because it changes what you should search for. Ollama keeps a substring list in llm/status.go, verbatim from main:

var outOfMemorySubstrings = []string{
	"out of memory",
	"out of device memory",
	"cudaMalloc failed",
	"hipMalloc failed",
	"failed to allocate",
	"allocation failed",
	"not enough memory",
	"insufficient memory",
	"vk_error_out_of_device_memory",
	"erroroutofmemory",
}

When a captured stderr line contains any of those, llm/llama_server.go returns a different error entirely:

if IsOutOfMemoryMessage(msg) {
    return fmt.Errorf("llama-server reported out-of-memory during startup: %s", msg)
}

So on a current build a memory failure may not mention a crash or an exit code at all. If yours does arrive wrapped in a terminated-runner message, our guide to Ollama runner exit codes decodes that outer layer first.

One detail that will save you a false alarm. The same file carries a second, much shorter list:

var recoverableOutOfMemorySubstrings = []string{
	"retrying without pipeline parallelism",
}

An out-of-memory line matching that one is not fatal — Ollama keeps waiting and the load continues. An alarming memory message in your log is not proof of the failure you are chasing, so match it against the message that actually came back to your client.

The six levers, in the order that costs you least

Work down. Stop when it loads. Each row costs you something real, and the rows are ordered so you pay the smallest price that works.

Notice what is not on that list: reinstalling Ollama, updating your GPU driver, adding swap, and clearing ~/.ollama. None of them change the size of the request, and the first three of those are the most common things people try first.

Why did it work an hour ago and not now

Because the previous model is probably still loaded, and Ollama measures free memory at the moment of the next load.

This is the single most under-diagnosed case in the tracker. Issue #13556, still open, is titled almost exactly the way people search for it: "Successfully run llama3.3 only for the first time. Subsequent run hits model requires more system memory (39.4 GiB) than is available (30.0 GiB)". Same model, same machine, same command — the first invocation works and the second does not.

Three defaults combine to produce it, and all three are documented:

• Models stay loaded after you stop talking to them. OLLAMA_KEEP_ALIVE is described in envconfig/config.go as "The duration that models stay loaded in memory", with a documented default of five minutes. Five minutes of a large model sitting in RAM is five minutes during which a second model has less to work with.
• Ollama will happily load more than one. Per the official FAQ, OLLAMA_MAX_LOADED_MODELS is "The maximum number of models that can be loaded concurrently provided they fit in available memory. The default is 3 * the number of GPUs or 3 for CPU inference."
• Loads are measured against what is free right now. The docs are explicit that when there is not enough, "new requests will be queued until the new model can be loaded" — but an explicit memory refusal surfaces as an error rather than a queue.

The fix takes ten seconds:

# What is actually loaded, and when it expires
ollama ps

# Evict one immediately
ollama stop llama3.3

# Or make everything unload the moment it goes idle
OLLAMA_KEEP_ALIVE=0 ollama serve

Through the API, the same thing is a request field: send "keep_alive": 0 to unload after the response, or -1 to pin a model in memory deliberately. Our complete Ollama guide covers the keep-alive semantics in full, including when pinning is the right call.

If you are running a shared or multi-user Ollama, set OLLAMA_MAX_LOADED_MODELS=1 and treat model switching as an explicit cost. Three concurrent large models is a sensible default for a server with headroom and a bad one for a workstation.

How much context are you actually paying for

More than the number you set, if you have ever touched parallelism. This is the lever people miss because the multiplication is invisible.

The FAQ states it directly for OLLAMA_NUM_PARALLEL: "The maximum number of parallel requests each model will process at the same time, default 1. Required RAM will scale by OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH." And with an example: "a 2K context with 4 parallel requests will result in an 8K context."

So a config you set months ago to make an agent framework feel snappier is quietly multiplying every context allocation. If you have OLLAMA_NUM_PARALLEL=4 and you raise num_ctx, you are raising it four times over.

The base default has also moved. In a log captured on Ollama 0.12.9 in issue #12982, the server config line reads OLLAMA_CONTEXT_LENGTH:4096. On main today the variable is described as "Context length to use unless otherwise specified (default: 4k/32k/256k based on VRAM)". A model that fit on an older Ollama can fail on a newer one purely because the default context it gets handed is larger — which is why "it broke after I updated" is so often a context problem rather than a regression.

How to cut it, from most to least persistent:

# Per session, inside ollama run
/set parameter num_ctx 4096

# Per request, via the API
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1",
  "prompt": "hello",
  "options": { "num_ctx": 4096 }
}'

# Server-wide
OLLAMA_CONTEXT_LENGTH=4096 ollama serve

Two real reports show how wide this lever's range is. Issue #5949 is an out-of-memory error running an 8B model at Q8_0 with num_ctx=120000 on ROCm — the weights were never the problem. Issue #4985 is a 4GB GPU handling Phi-3 Mini fine until a 20k-token prompt arrives. In both, the model fits and the conversation does not.

Do not go lower than the work needs. A context so short that the model forgets the question is not a fix, it is a different failure. If you need long context and it genuinely will not fit, that is the point at which the sizing pages become the right read.

Does quantising the KV cache actually help

Yes, and it is the best value-for-money lever on this page — but it has a prerequisite that quietly disables it.

Per the official FAQ, OLLAMA_KV_CACHE_TYPE takes three values: f16 (the default, high precision), q8_0 ("8-bit, ~1/2 memory of f16") and q4_0 ("4-bit, ~1/4 memory of f16"). Those ratios are the documented ones, and they apply to the cache — not to the weights, which are unaffected.

The prerequisite is flash attention. The FAQ describes the setting as the quantisation type "when Flash Attention is enabled", and says "Ollama uses Flash Attention automatically when the selected backend and devices support it". Automatic is not the same as always. In that same 0.12.9 config dump from #12982, the two relevant fields read OLLAMA_FLASH_ATTENTION:false and OLLAMA_KV_CACHE_TYPE: — flash attention off, cache type unset. On a build or a backend in that state, setting the cache type alone achieves nothing.

# Halve the KV cache. Set both, in this order.
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

Which value to pick. Start at q8_0. It is the setting most people can leave on permanently, and it buys back roughly half the cache. Reach for q4_0 only when q8_0 was not enough — cutting the cache to a quarter of its f16 size is a real precision loss, and it shows up on long contexts and long generations before it shows up on short chats, which is exactly the workload that made you need it.

Check it took effect. Restart the server, then look for the flash-attention and cache lines in the startup log. If the backend refused, they will say so, and you will have your answer before you spend a session wondering.

How do you force layers off the GPU

With num_gpu, which is a layer count and not a boolean — a fact the name hides, and the reason so many people set it to 1 and wonder why it got slower rather than smaller.

The semantics are readable in server/sched.go:

• num_gpu = 0 means no GPU at all. The scheduler skips GPU discovery: if pending.opts.NumGPU == 0 { gpus = []ml.DeviceInfo{} }.
• A negative value means automatic. The file carries the comment "Don't reload runner if num_gpu=-1 was provided", and the mmap logic branches on opts.NumGPU < 0 as the auto case.
• A positive value is a number of layers, compared against the model's block count: uint64(opts.NumGPU) < f.KV().BlockCount()+1. Anything below that threshold is a partial offload, with the remainder on the CPU.

# Inside ollama run — put 20 layers on the GPU, the rest on CPU
/set parameter num_gpu 20

# Per request
"options": { "num_gpu": 20 }

# Force CPU-only, to prove the GPU is the constraint
"options": { "num_gpu": 0 }

How to search for the right number: halve it. If the model has 32 layers and num_gpu 32 crashes, try 16. If 16 loads, try 24. Three loads will get you within a couple of layers of the maximum, and each one takes seconds. Guessing downward one layer at a time is the slow way to spend an afternoon.

That num_gpu 0 line is also the fastest diagnostic on this page. If CPU-only loads and works, the constraint is VRAM specifically, and levers 4 and 5 are where your remaining wins are. If CPU-only also fails, you are out of system RAM and no amount of offload arithmetic will help.

One side effect worth knowing about, from the same file: a partial offload changes Ollama's default for memory-mapping. disableMmapDefaultReason returns "metal_partial_offload" on Apple Silicon when NumGPU is positive but below the block count, "windows_cuda" on Windows with CUDA, and "cpu" when NumGPU is zero. So switching to partial offload is not purely a layer decision — it can change how the weights are held in memory too, which is occasionally why a partial offload behaves differently from what the arithmetic predicted.

Why does it run out with free VRAM on the meter

Because Ollama does not spend everything the driver reports as free, and because the number it looked at may be stale by the time the allocation happens.

The reservation is explicit in server/sched.go:

available := gpu.FreeMemory - envconfig.GpuOverhead() - gpu.MinimumMemory()
if gpu.FreeMemory < envconfig.GpuOverhead()+gpu.MinimumMemory() {
    available = 0
}

Two subtractions before anything is offered to the model: a per-GPU minimum, and OLLAMA_GPU_OVERHEAD, described in envconfig/config.go as "Reserve a portion of VRAM per GPU (bytes)" with a default of 0. Right after that, the scheduler logs both views. These are the lines to find, as they appeared in a real log posted to issue #16506:

msg="system memory" total="13.5 GiB" free="9.6 GiB" free_swap="48.0 GiB"
msg="gpu memory" id=0 library=CUDA available="2.8 GiB" free="3.2 GiB" minimum="457.0 MiB" overhead="0 B"

available is the number that decides your load, not free. In that captured example the gap between them is the per-GPU minimum. If your available is far below what your GPU monitor shows, something else on the machine is holding VRAM — a compositor, a browser with hardware acceleration, another inference process — and Ollama measured after it.

That measurement is a snapshot, which is the mechanism behind a whole family of reports: #14632 ("Ollama under utilizes available GPU VRAM causing out of memory", open) and #9782 (out of memory "while having some idling GPUs") are both this shape.

When to raise OLLAMA_GPU_OVERHEAD deliberately: when the crash happens partway through generation rather than at load. That pattern means Ollama's estimate was right at load time and something took VRAM afterwards. Reserving a slice up front trades a little capacity for a load that survives:

# Hold back 1 GiB per GPU (value is in bytes)
OLLAMA_GPU_OVERHEAD=1073741824 ollama serve

Multi-GPU boxes have their own version of this, where the problem is distribution rather than total capacity — #10113 ("CUDA out of memory: Mixed VRAM Cards") and #15033 (an 8-GPU Windows setup) are both about which card got which share. The Ollama multi-GPU setup guide covers device ordering, OLLAMA_SCHED_SPREAD and the split behaviour that decides it.

When no lever will save you

Some gaps are not closeable, and recognising one immediately is worth more than an evening of tuning. The tell is the ratio in Ollama's own refusal message.

Issue #10920 is the archetype: ollama run deepseek-v2.5 on a 16GB Arm board, and after what the reporter describes as "nearly 4 hours" of downloading a 132 GB model, Ollama answers model requires more system memory (164.8 GiB) than is available (13.4 GiB). Do the division that Ollama is implicitly doing: 164.8 ÷ 13.4 ≈ 12.3. The ask is over twelve times the machine. There is no context setting, no cache quantisation and no offload split that closes a 12× gap — and num_gpu cannot help at all, because there is no GPU in that story.

Issue #8667 is the same shape at a larger scale — deepseek-r1:671b at Q4_K_M reporting a requirement of 446.3 GiB — and #8571 is the version where the kernel does the refusing instead, with signal: killed on 64GB and 128GB Macs.

The rule of thumb: if the requirement is under about 1.5× your available memory, the levers above will usually close it. If it is 2× or more, you need a smaller model, a smaller quant, or a bigger machine — and that is a sizing decision, which our RAM and VRAM table and the best Ollama models for 8GB of VRAM pages are built to answer. Pick the model that fits rather than tuning one that never will.

Worth noting as a matter of expectations rather than technique: that memory check runs after the download, which is why #10920's reporter asked for it to run before. Check the model size against your machine before you start a pull, not after.

What this page cannot tell you

• We did not reproduce these crashes. Every error string, default and log line is read from Ollama's source on main, its published docs, or the linked issue. Nothing here is a claim about our own hardware.
• We deliberately quote no gigabyte figures of our own. The only sizes on this page are ones a reporter or a log printed. What a given model needs is a measurement, and it lives on the sizing pages, not here.
• Defaults move between releases. OLLAMA_CONTEXT_LENGTH alone has gone from a flat 4096 to a VRAM-dependent 4k/32k/256k. Check ollama serve's startup config line for what your build is really using before trusting any article, including this one.
• Several of the linked issues are open. #13556, #14632 and #15033 were open when this was written. Click through before you rearrange a workflow around a workaround.
• Lever six is last for a reason. Dropping a quant level is the only change here that permanently costs output quality, and it is the one most guides put first.

FAQ

What does "cudaMalloc failed: out of memory" mean in Ollama

The NVIDIA driver refused an allocation while the runner was loading the model. It is not a driver bug and reinstalling CUDA will not touch it. Ollama treats the string as an out-of-memory condition explicitly — cudaMalloc failed is one of ten substrings in the list in llm/status.go that make it relabel the failure as "llama-server reported out-of-memory during startup". Start by checking what is already resident with ollama ps, then cut context, then cap num_gpu.

Ollama says "signal: killed". Is that the same thing

Related, but it comes from outside the process. SIGKILL cannot be raised by a process on itself, so the kernel or another process ended it — on Linux that is normally the OOM killer, and dmesg or journalctl -k will name the victim. In the reports where it appears, the model was far too large for the machine rather than marginally over, so treat it as a sizing signal rather than a tuning one.

Why does it run out of memory when nvidia-smi shows free VRAM

Because Ollama does not offer the model everything the driver reports. Its scheduler computes available = FreeMemory - GpuOverhead - MinimumMemory, logs both numbers side by side, and decides on available. If that figure is much lower than your monitor's, something else took VRAM before Ollama measured. If the crash happens during generation rather than at load, the opposite is true — something took VRAM afterwards, and raising OLLAMA_GPU_OVERHEAD to reserve a slice up front is the fix.

Does OLLAMA_KV_CACHE_TYPE=q4_0 hurt output quality

It costs precision in the attention cache, and the documented saving is roughly a quarter of f16's memory versus about half for q8_0. The loss is felt most on long contexts and long generations, which is unfortunately the workload that usually drives people to set it. Try q8_0 first and only drop to q4_0 if that was not enough. Both need flash attention active — a build with OLLAMA_FLASH_ATTENTION:false in its config line will ignore the cache setting entirely.

Will adding more system RAM fix a CUDA out-of-memory error

Not directly, but it changes what is possible. A CUDA out-of-memory error is about VRAM, so more system RAM does not add capacity where the failure happened. What it does is make partial offload viable: with num_gpu set below the layer count, the layers that no longer fit on the GPU are held in system memory instead, and more of it means you can push more layers off the card without the CPU side becoming the new constraint.

How do I stop Ollama loading two models at once

Set OLLAMA_MAX_LOADED_MODELS=1. The documented default is three per GPU (or three for CPU inference), which is reasonable on a server and aggressive on a workstation. Pair it with a shorter OLLAMA_KEEP_ALIVE — the default holds a model in memory for five minutes after its last request — or evict on demand with ollama stop.

Sources

• Ollama — official FAQ (OLLAMA_MAX_LOADED_MODELS, OLLAMA_NUM_PARALLEL, OLLAMA_KV_CACHE_TYPE values and ratios, flash attention, concurrency and memory)
• ollama/ollama — envconfig/config.go (variable descriptions and defaults, quoted verbatim)
• ollama/ollama — server/sched.go (the available calculation, the memory log lines, and num_gpu semantics)
• ollama/ollama — llm/status.go (outOfMemorySubstrings and recoverableOutOfMemorySubstrings)
• ollama/ollama — llm/llama_server.go (the out-of-memory relabelling branch)
• ollama/ollama issues #4985, #5949, #8571, #8667, #9782, #10113, #10920, #12982, #13556, #14632, #15033, #16506