AI Models for 16GB RAM: What Fits, What Swaps

Published April 23, 2026 • Updated August 2026 • 12 min read

A 16GB machine runs any 7B-9B open-weight model at Q4_K_M quantization — roughly 4 to 6 GB of weights — with a 4k to 8k context window. A 13B fits only at Q3 or lower. Anything at 30B or above pages to disk and stops being interactive. The ceiling is not "16GB minus the model." It is 16GB minus the operating system, minus whatever else is open, minus a KV cache that grows with every token you generate.

This page gives you the arithmetic rather than a leaderboard, because the arithmetic is what transfers to the machine actually in front of you.

Quick start: which model should I pull first?

# 1. Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# 2. Pull a 7B that fits 16GB with headroom to spare
ollama pull qwen2.5:7b-instruct-q4_K_M

# 3. Run
ollama run qwen2.5:7b-instruct-q4_K_M

Qwen 2.5 7B at Q4_K_M is about 4.6 GB of weights, which leaves the rest of a 16GB system to the OS, your browser and the KV cache. If you only ever pull one model on this tier, pull that one. Everything below explains when a different pick is defensible.

Table of Contents

1. What actually eats your 16GB?
2. How much RAM does a model need?
3. How big does the KV cache get?
4. Which quantization should I pick?
5. Which models fit on 16GB?
6. How fast will it run?
7. Can I run a 13B on 16GB?
8. Is a GPU worth it here?
9. What trips people up?
10. FAQ

---

What actually eats your 16GB?

Three things compete for the same pool, and only one of them is the model:

1. The model weights. Fixed once you pick a quantization. See the next section.
2. The KV cache. Grows linearly with context length. This is the one that surprises people.
3. Everything else. The OS plus your open apps. A desktop OS with a browser, an editor and a chat client open is realistically several gigabytes before a model loads at all.

Measure your own baseline before you plan anything else — on Linux read the available column of free -h, on macOS read Activity Monitor's memory pressure graph. Whatever number that gives you is your real budget, and it is always smaller than 16.

For the tier-by-tier upgrade picture, our hardware requirements complete guide covers what changes at 32GB and with a discrete GPU, and RAM requirements for local AI works the same budget for other memory sizes.

---

How much RAM does a model need?

Weight footprint is arithmetic, not a mystery. A quantized model stores roughly a fixed number of bits per parameter, so:

weights (GB) ≈ parameters (billions) × bits-per-weight ÷ 8

Q4_K_M averages about 4.5 bits per weight once you include the higher-precision layers K-quants keep, which collapses to a rule you can do in your head:

Q4_K_M weights (GB) ≈ 0.6 × parameters (billions)

So a 7.6B model is about 4.6 GB, an 8B is about 4.8 GB, a 12B is about 7.2 GB. Check any of those against the file size Ollama reports with ollama list — the formula lands within a few hundred megabytes, and where it does not, the gap is usually a vision tower or an unquantized embedding layer.

One reconciliation, since you will notice the two numbers: pure bit arithmetic gives about 4.3 GB for a 7.6B model at Q4_K_M, while the 0.6-per-billion rule of thumb gives about 4.6 GB. The rule rounds up deliberately, because a real GGUF also carries the token embedding table and file metadata. Budget with the higher figure and you will not be surprised.

If you want a defensible number for how much quality each quant costs, the llama.cpp maintainers publish perplexity deltas per quant type in the repo itself. Read llama.cpp's quantize tool documentation and the linked perplexity discussions rather than trusting a percentage copied out of a blog post — the numbers move as the quant formats are revised. Our AWQ vs GPTQ vs GGUF comparison explains which format to reach for; on 16GB, GGUF Q4_K_M served by Ollama is the right default.

---

How big does the KV cache get?

Every token you generate is appended to a key/value cache that stays resident for the whole conversation. Its size is fully determined by the model's published architecture:

KV cache bytes per token = 2 × layers × KV-heads × head-dim × bytes-per-element

Worked example, Llama 3.1 8B (32 layers, 8 KV heads, head dim 128, fp16 cache — all from its config.json on Hugging Face):

2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes ≈ 0.125 MB per token

That last row is the honest reason Llama 3.1's advertised 128k context is unusable on this tier: the cache alone exceeds the whole machine. Models with fewer KV heads are much cheaper here — Qwen 2.5 7B uses 28 layers and 4 KV heads, which works out to roughly 0.055 MB per token, less than half of Llama's. That architectural difference matters more for long-context work on 16GB than any benchmark score.

Run the formula for your model before you raise num_ctx. It takes ten seconds and it is the single most useful calculation on this page.

---

Which quantization should I pick?

Q4_K_M, unless you have a specific reason. It is where the size curve flattens: dropping to Q3 saves under a gigabyte on a 7B and costs real reasoning quality, while climbing to Q5 costs about a gigabyte and buys a difference most people cannot detect in chat.

The two situations that justify Q5_K_M or Q6_K on 16GB:

• Strict structured output. If your application depends on valid JSON or exact arithmetic, test the higher quant explicitly against your own prompts before assuming Q4 is fine.
• Small models with room left over. A 3B at Q8 still only costs about 3.2 GB. If the model is small enough that the higher quant fits comfortably, take it.

---

Which models fit on 16GB?

Sizes below are computed from published parameter counts using the Q4_K_M formula above. Release dates and specialties are from each model's own card.

Picking by job, based on what each model was trained and released for rather than a score we invented:

• Code: Qwen 2.5-Coder 7B. It is the code-specialized variant of a strong general model, which is the whole reason it exists. Compare options in best local AI models for programming.
• Multi-step reasoning: DeepSeek-R1-Distill-Qwen 7B. The reasoning trace costs you tokens, so it feels slower in wall-clock time even at identical throughput.
• Tool and function calling: Granite 3.1 8B was trained for it explicitly; Qwen 2.5 7B is the general-purpose alternative. Evaluate both against your actual tool schema.
• Long context on tight memory: favour models with fewer KV heads. Run the cache formula, not the marketing number.
• Battery and speed: Llama 3.2 3B or Phi-4 mini. Under half the weights to stream per token means proportionally more headroom for everything else.

---

How fast will it run?

Single-stream generation is memory-bandwidth bound. Producing one token requires reading every weight once, so there is a hard arithmetic ceiling:

tokens/sec ceiling = memory bandwidth (GB/s) ÷ model size (GB)

The table below applies that to a 4.6 GB model (a 7.6B at Q4_K_M) using each platform's published memory bandwidth. These are upper bounds, not predictions. Real output always lands below them, because attention over the KV cache, sampling, and imperfect memory access all cost time the formula ignores. Treat a row as "this machine cannot beat this," and treat any guide quoting a higher figure as wrong.

Two consequences fall straight out of this. First, CPU cores are almost never the bottleneck on this tier — bandwidth is, which is why a slower CPU with faster memory beats the reverse. Second, Apple Silicon competes with entry discrete GPUs not because of clever software but because unified memory gives the GPU the full memory bandwidth of the machine.

Want your own numbers instead of a ceiling? Our benchmarking guide walks through measuring generation rate, prompt processing and time-to-first-token so the result is reproducible.

---

Can I run a 13B on 16GB?

At Q4 a 13B is about 7.8 GB, which fits in isolation but leaves almost nothing for cache and apps. Three legitimate approaches:

Drop the quantization

A 13B at Q3_K_S lands near 5.5 GB. Whether that beats a 7B at Q4 depends entirely on whether the larger model's underlying capability survives the harsher quant — for most everyday tasks it does not, which is why the 7B remains the default recommendation.

Split the model across GPU and CPU

If you have a small discrete GPU, Ollama will run some layers on it and the rest on CPU. Estimate how many fit:

layers on GPU ≈ (VRAM GB − 1 GB overhead) ÷ (model GB ÷ total layers)

For a 7.8 GB 13B with 40 layers on an 8GB card: 7.8 ÷ 40 = 0.195 GB per layer, and (8 − 1) ÷ 0.195 ≈ 35 layers. Set that explicitly rather than guessing:

OLLAMA_LLM_LIBRARY=cuda_v12 ollama run llama2:13b --num-gpu 35

Let it page from disk

Ollama mmaps weights, so the OS can stream parts in on demand. The ceiling formula still applies, just with SSD bandwidth substituted for RAM bandwidth — and a fast NVMe drive is roughly an order of magnitude slower than system memory. Usable for overnight batch jobs, not for conversation.

OLLAMA_KEEP_ALIVE=24h OLLAMA_NUM_PARALLEL=1 ollama run llama2:13b

Above 13B — Mixtral 8x7B, Llama 70B — you want 32GB minimum or a 24GB GPU. Our budget local AI machine and used GPU buying guide cover the upgrade path.

---

Is a GPU worth it here?

Compare the bandwidth column, not the price tag. An RTX 3060 12GB moves memory several times faster than dual-channel DDR5, and the ceiling table above turns that ratio directly into a throughput ratio. The catch is capacity: a 12GB card holds a 7B at Q4 with room for cache, but a 13B at Q4 will not fit entirely, and the moment layers spill to CPU your effective bandwidth is dominated by the slow half.

That gives a simple rule for this tier. Buy VRAM capacity first, bandwidth second, and ignore core counts. A card that holds your whole model at a modest bandwidth beats a faster card that forces an offload split. RTX 4060 vs RTX 3060 for AI compares the two most common picks at this budget.

The same logic explains why an older Intel Mac is a poor local AI machine regardless of its CPU: without unified memory the GPU cannot address the full memory pool at full bandwidth, so the ceiling formula runs against a much smaller number.

---

What trips people up?

1. Misreading free memory. free -h can report most of your RAM as free when it is actually page cache. Read the available column. On macOS, watch the memory pressure graph — once it turns yellow, throughput is being destroyed by swap and no configuration change will fix it.

2. Background apps. They come out of the same 16GB budget as the model. Closing a browser with dozens of tabs is the cheapest performance change available on this tier.

3. Context window inflation. num_ctx defaults low in Ollama for a reason. Raising it to 32k because "more context is better" allocates a cache you computed the size of two sections ago. Set it to what your prompts actually need.

4. Assuming flash attention always helps. It reduces KV cache memory, but the throughput effect varies by backend. Toggle OLLAMA_FLASH_ATTENTION=1 and compare on your own machine rather than assuming.

5. Running out of disk, not RAM. Ollama stores models in ~/.ollama/models. Check with du -h ~/.ollama/models and prune with ollama rm.

6. Unpinned model tags. ollama pull qwen2.5:7b resolves to whatever the tag points at today. For anything reproducible, pin the full tag: qwen2.5:7b-instruct-q4_K_M.

If your setup is slower than the ceiling table suggests it should be, why is my local LLM slow works through the causes in order of frequency.

The official Ollama model library lists every quant variant with its file size, and the Qwen 2.5 7B model card publishes the layer and head counts the KV cache formula needs.

---

Frequently Asked Questions

Is 16GB really enough for serious local AI?

For one person running a 7B-class model at Q4 with a sensible context window, yes. For serving several users at once, indexing a large RAG corpus, or running anything above 13B, no. The upgrade that unlocks the next tier is 32GB, not a faster CPU.

Should I buy more RAM or a faster CPU on a fixed budget?

More RAM. A model that fits runs at the bandwidth ceiling; a model that spills to disk runs at SSD speed no matter how fast the CPU is, and the gap between those two is roughly an order of magnitude.

Will Llama 70B run on 16GB using disk swap?

It loads, and then it crawls. At Q4 a 70B is around 42 GB, so most of it streams from the SSD on every single token. Divide a fast NVMe drive's sequential bandwidth by 42 GB and the ceiling comes out below a fifth of a token per second — before accounting for the random-access penalty. That is a batch job at best, never an assistant.

Why does Q4_K_M behave better than Q4_0 at the same nominal bit width?

K-quants allocate bits unevenly: layers that are more sensitive to rounding keep more precision, less sensitive ones keep less. The average bits per weight stays about the same while the damage is concentrated where it matters least.

Does a 16GB Apple Silicon Mac beat a 16GB Intel Mac for this?

Yes, and the reason is architectural rather than generational. Apple Silicon's unified memory lets the GPU address the whole pool at full bandwidth; an Intel Mac's integrated graphics cannot, so the ceiling formula runs against a much smaller bandwidth number.

Can I keep two models loaded at once on 16GB?

Two 7B models at Q4 is roughly 9 GB of weights before either one has a KV cache. It is technically possible with OLLAMA_MAX_LOADED_MODELS=2 and short contexts, but swapping between models on demand costs a couple of seconds and is almost always the better trade.

How much context can I afford?

Run the KV cache formula for your specific model, then subtract from your measured available memory. For a Llama-architecture 8B on a typical 16GB desktop, 4k to 8k is comfortable; models with fewer KV heads stretch considerably further on the same budget.

Which is better at 7B-8B, Mistral or Llama?

They target different things. Mistral 7B is smaller and therefore has a higher bandwidth ceiling on identical hardware; Llama 3.1 8B has broader tooling support and a larger ecosystem of fine-tunes. Neither wins universally — run both against your own prompts, which takes an afternoon and beats any third-party ranking.

---

Conclusion

The 16GB tier is where most people meet local AI, and the honest summary is that it is a 7B-to-9B machine. Pick Qwen 2.5 7B at Q4_K_M as the default, swap in Qwen 2.5-Coder for code and DeepSeek-R1-Distill for multi-step reasoning, keep the context window matched to what you actually use, and you have a setup that handles the bulk of practical work without sending anything to a cloud API.

When you outgrow it, the hardware requirements guide covers the path to 32GB and a dedicated GPU, and best local AI models widens the model shortlist.

Want new 16GB-friendly models flagged as they land? Subscribe to the LocalAIMaster newsletter.