Lightweight AI Models: Sub-7B LLMs That Fit in 8GB

Which lightweight AI models actually fit on a laptop?

Any model up to about 8B parameters fits in 8GB once it is quantized to Q4_K_M, because that format costs roughly 0.6GB per billion parameters. A 3B model is about 1.8GB of weights, a 1.1B model is under a gigabyte, and that leaves room for the operating system and a working context window. The list below is eight models that clear that bar, with the arithmetic shown so you can size anything not on the list.

What you will not find on this page is a tokens-per-second league table. We do not own the hardware to produce one honestly, and a made-up number is worse than no number. What you get instead is the memory maths, the published specifications, and the throughput ceiling your hardware imposes — which is the part that actually decides whether a model is usable for you.

How much memory does a small model need?

One rule of thumb covers almost every case:

weights (GB) ≈ parameters in billions × bytes per weight

At Q4_K_M, bytes per weight ≈ 0.6. So a 3B model ≈ 1.8GB, a 7B model ≈ 4.2GB.

Q4_K_M is not literally four bits per weight — the k-quant format keeps attention and embedding tensors at higher precision, which is why the working constant is 0.6 rather than 0.5. Here is the same arithmetic across the quantization ladder:

Then add the KV cache, which grows with how much context you actually use. It is small for these models — a few hundred megabytes at typical chat lengths — but it is not zero, and it is what bites you when you paste in a long document. Budget 1–2GB on top of the weights and you will not be surprised.

Two consequences worth internalising:

• Q4_K_M is the default for a reason. Going to Q8 nearly doubles the memory for a quality difference most people cannot pick out in a blind comparison; going below Q4 on an already-small model degrades it fast, because a 1B model has less redundancy to spare than a 70B one.
• Fitting matters more than raw speed. A model that fits entirely in VRAM or unified memory runs at memory-bandwidth speed. One that spills to disk runs at storage speed, which is one to two orders of magnitude worse. The whole game on a small machine is fitting.

The sub-7B models worth knowing

Parameter counts and context windows below are the figures published on each model's card. The memory column is the 0.6GB-per-billion arithmetic from the section above, not a measurement.

Note the context column, because it is the specification people skip. TinyLlama's 2K window is the real constraint on that model — it is not a quality problem, it is that you cannot paste a long file into it. Phi-3 Mini ships as two separate builds and picking the wrong one is a common cause of "why did it forget what I said".

What each one is for

• Llama 3.2 3B — the sensible default in 2026 if you have 8GB. General instruction following, a 128K window, and the most tooling built around it. Start here and only move if it disappoints you at something specific.
• Llama 3.2 1B — for when 3B is still too heavy: background tasks, classification, mobile-class hardware. It follows instructions far better than 1B models did two years ago.
• Gemma 2 2B — strong writing quality for its weight class, held back by the 8K window. Good for chat and drafting, poor for long-document work.
• Phi-3 Mini — Microsoft trained it heavily on synthetic textbook-style data, which shows up as reasoning that punches above 3.8B on structured problems. Take the 128K build unless disk is critical.
• Qwen 2.5 3B — the best non-English coverage in this size band, and a 32K window. Check the licence on the model card before commercial use; the Qwen family does not use one licence for every size.
• TinyLlama 1.1B — the extreme-edge option. Genuinely runs on a Raspberry Pi. Do not expect it to reason.
• SmolLM2 1.7B — Hugging Face's small-model line, Apache-2.0, designed for on-device use.
• DeepSeek-Coder 1.3B — code completion specifically, not chat. A specialised small model beats a general small model on its own task.

How fast will these run on my hardware?

Generation speed on a local LLM is bounded by memory bandwidth: to produce one token, the runtime streams the whole model out of memory. That gives an upper bound you can calculate before buying anything.

ceiling (tokens/sec) = memory bandwidth (GB/s) ÷ model size in memory (GB)

This is an arithmetic upper bound, not a prediction. Real output lands well below it — the formula ignores prompt processing, attention over the KV cache, sampling, and the fact that no runtime reaches a vendor's peak bandwidth figure. Use it to rule hardware out, never to promise a number.

Worked through for a 3B model at Q4_K_M (~1.8GB), using manufacturers' published bandwidth specifications:

The shape of that table is the useful part. Human reading speed is roughly 5–8 tokens per second, so even the CPU row is comfortably faster than you can read — which is why small models on ordinary laptops feel fine, and why buying a bigger GPU to run a 3B model is money wasted. The Pi row is where the ceiling starts to matter.

Which quantization should I pick?

• Q4_K_M — the default. Best size-to-quality trade in the k-quant family, and what ollama pull gives you unless you ask for something else.
• Q5_K_M — worth it if you have the memory headroom and the task is precision-sensitive (structured output, code).
• Q8_0 — for evaluation, when you want to know how much the quantization is costing you rather than guessing.
• Below Q4 — a last resort on models this small. Quantization damage is proportionally worse on fewer parameters.

The mechanics of what these formats actually do to the weights are covered in Quantization Explained.

How do I install them?

Ollama is the shortest path on all three platforms:

# Install (macOS / Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Pull and chat
ollama run llama3.2:3b

# Check what a model costs on disk before you commit
ollama list

Windows users can follow the Ollama Windows installation guide; on Apple Silicon, Run Llama 3 on Mac covers Metal acceleration. If the machine needs to work with no network at all, Run AI Offline covers pre-caching models and blocking outbound calls.

Frequently asked questions

How much RAM do I need for a sub-7B model?

8GB of system RAM is enough for any 3B model at Q4_K_M, with room left for the OS. 16GB is comfortable for 7B. On a GPU, you need the weight size plus 1–2GB of headroom for context — so a 6GB card handles 3B models easily and a 7B model at Q4_K_M needs 8GB. Use the 0.6GB-per-billion rule to check any specific model.

Are small models good enough for coding?

For completion, boilerplate and explaining unfamiliar code, yes — and a specialised model like DeepSeek-Coder 1.3B beats a general model twice its size at exactly that. For architectural decisions, multi-file refactors, or anything where a subtle error is expensive, they are not a substitute for review. Treat the output as a draft, always.

Which small model is best for a Raspberry Pi?

TinyLlama 1.1B or Llama 3.2 1B, both under a gigabyte at Q4_K_M. The binding constraint on a Pi is memory bandwidth, not capacity — see the ceiling table above. Running an LLM on a Raspberry Pi 5 covers the setup end to end.

How do lightweight models compare to GPT-4 or Claude?

They are not competitive on reasoning depth or breadth of world knowledge, and no honest framing of a 3B model claims otherwise. What they are competitive on is latency, cost, and the fact that nothing leaves your machine. For summarising, rewriting, extraction, classification and routing, that trade is often the right one. For hard multi-step reasoning it is not.

Can small models handle images or audio?

Some. Llama 3.2 ships vision variants (11B and up, so outside this size band), and Qwen-VL handles text-image tasks. In the sub-7B range, multimodal support is the exception rather than the rule — check the model card rather than assuming.

Does a bigger context window make a model better?

No, and it costs memory. A 128K window means the model accepts that much input; quality of recall across that span is a separate question, and small models degrade earlier than large ones. Pick the window you actually need.

Where to go next

• Outgrown 8GB? Best GPUs for Local AI covers the next tier up.
• Want the compression theory? Quantization Explained.
• Optimising a small model rather than swapping it? Small Language Models efficiency guide.
• Assembling a full local stack? Top Free Local AI Tools.