Llama 3.1 8B: Local Deployment Guide

About 7 GB at Q4_K_M with an 8K context, so an 8 GB GPU is the floor: a 4.9 GB file, 1.1 GB of context cache and roughly 1 GB of buffers. A 12 GB card covers Q8_0 or a 32K context; 16 GB of system RAM is the equivalent floor without a GPU; the unquantised fp16 build needs a 24 GB card. The arithmetic is parameters × bits per weight ÷ 8: 8.03B × 4.90 bits ÷ 8 = 4.92 GB at Q4_K_M, which matches the published file.

File sizes are from bartowski/Meta-Llama-3.1-8B-Instruct-GGUF and Ollama’s tag list. The context cache is arithmetic from the model’s config.json: 32 layers × 8 key-value heads × 128 dimensions, stored twice at 2 bytes, is 128 KB per token, or 1.07 GB for 8,192 tokens. “Memory needed” is file + that cache + a 1 GB allowance for buffers.

• System RAM, no GPU: the file loads into RAM instead, so the same totals apply. 16 GB of RAM covers every build up to Q8_0 with room for the operating system; on an 8 GB machine only the Q2_K and Q3 builds leave enough free. See running an LLM on CPU only.
• Mac: unified memory follows the same arithmetic; an 8 GB Mac is the same squeeze as an 8 GB machine with no GPU, and a 16 GB Mac is comfortable. See running Llama 3 on a Mac.
• Disk: the download size in the table, once per build you pull.
• The 128K window: the table assumes 8K of context. The next section shows what the full window adds.

For the wider ladder across model sizes see the 8 GB RAM model guide, best LLM for 8 GB of VRAM, best LLM for 12 GB of VRAM and the Ollama RAM and VRAM table.

The default llama3.1:8b is 4.9 GB. Ollama carries fifteen instruct builds of the 8B model, from 3.2 GB at Q2_K to 16 GB at fp16, all with the 128K context. The sizes below are copied from the tag list; the 8b-text-* tags are the same ladder for the base (pretrained) model, and the 70B and 405B tags are on the Llama 3.1 70B and Llama 3.1 405B pages.

Note the 4.7 GB vs 4.9 GB difference between llama3:8b and llama3.1:8b: both models have the same 8,030,261,248 parameters, but the Llama 3 default tag is the older q4_0 format while the Llama 3.1 default is Q4_K_M. The build names are explained in quantization explained.

About 17 GB of KV cache at the full 131,072 tokens, on top of the 4.9 GB of weights, so a 24 GB card is the floor for the whole window. This is the number missing from most Llama 3.1 8B write-ups. The weights are only half the memory bill; the cache grows linearly with how many tokens are in the window:

So the honest version of “128K context on an 8 GB card” is: the model supports it, your GPU does not. Ollama’s documentation says it uses a 4,096-token window by default, so you also do not get the long window unless you ask for it: /set parameter num_ctx 32768 inside ollama run, OLLAMA_CONTEXT_LENGTH=32768 ollama serve, or num_ctx in an API request. Quantising the cache to 8-bit roughly halves the figures above, at some cost to long-context recall.

We have measured this model on our own Apple M3 Pro and on a rented NVIDIA L4 (the tables further down); for any other card or Mac, what you can do before buying anything is compute the ceiling. Token generation on a local LLM is bandwidth-bound: to emit one token the runtime streams every weight it is using out of memory, so:

Two things follow. First, a bigger GPU only helps until the model already fits; two cards that both hold the 4.9 GB file differ in bandwidth, not capacity. Second, if the model does not fit and layers spill to system RAM, the much lower bandwidth of system memory becomes the effective ceiling for the spilled portion. Fitting the model is worth more than raw GPU speed, and the 8-bit Q8_0 build on a 12 GB card will be slower than Q4_K_M on the same card simply because there are more bytes to move.

Read the measured row against the ceiling: 27.2 tok/s on a 150 GB/s machine with a 4.92 GB file is 89% of the bound, which is as close as a dense model gets. On the rented NVIDIA L4 (24 GB, 300 GB/s) the same tag measured 49.2 tok/s, 81% of its 61 tok/s bound (Ollama 0.40.13, 3 November 2026). Scale that share, not the raw number, to your own hardware’s bandwidth.

Llama 3.1 8B Instruct scores 69.4 on MMLU (5-shot) on Meta’s model card, 73.0 with 0-shot chain-of-thought; the base model scores 66.7. These are Meta’s own results from its internal evaluation library. The card compares Llama 3.1 with Llama 3 and with the larger 3.1 models; it contains no figures for any other company’s models. The biggest moves over Llama 3 8B are in maths (MATH 29.1 → 51.9), code (HumanEval 60.4 → 72.6) and tool use (API-Bank 48.3 → 82.6); GPQA is the one row that went down.

Source: Meta’s Llama 3.1 model card. Other models’ scores on the same benchmarks are on the AI model leaderboard.

Llama 3.1 8B is a dense decoder-only transformer with grouped-query attention. The figures below are from the model’s config.json; they are identical to Llama 3 8B except for the maximum position count and the added RoPE scaling, which is how the same weights layout reaches 128K.

From config.json (ungated mirror of Meta’s repository). The full generation-by-generation table of Llama dimensions is on the Llama 3 8B page.

Comparing the two model cards, four things changed that matter for local use:

• Context 8K → 128K. The headline change, with the memory bill described above.
• Tool calling. The 3.1 card reports tool-use benchmarks for the instruct model; on API-Bank the 8B went from 48.3 to 82.6.
• Eight supported languages rather than English alone, with the multilingual MMLU scores above.
• Knowledge cutoff December 2023 instead of March 2023, with roughly the same 15T-token pretraining budget.

Same parameter count, same architecture, and the same memory footprint at a given quantisation, so upgrading from Llama 3 8B costs you nothing but the download. The one caveat: the default Ollama tags differ (4.7 GB q4_0 for Llama 3, 4.9 GB Q4_K_M for 3.1), which is a format change, not a size change.

Llama 3.1 is released under the Llama 3.1 Community License (version release date July 23, 2024). Meta describes it as a custom commercial licence; it is not an open-source licence such as MIT or Apache 2.0. The terms that matter in practice:

• Commercial use is permitted, along with modification and redistribution.
• If you distribute the model, a derivative, or a product or service that contains it, you must include the licence and “prominently display ‘Built with Llama’” on a related website, interface, blog post, about page or documentation.
• A model you train or fine-tune from it and distribute must “include ‘Llama’ at the beginning of any such AI model name”.
• If, on the release date, your products or services had more than 700 million monthly active users in the preceding month, you must request a separate licence from Meta.
• Use is also subject to Meta’s Acceptable Use Policy, and the card lists use outside the eight supported languages as out of scope unless you fine-tune for it.

This is a summary, not legal advice; read the licence before building a product on it. Other model licences are compared in LLM licences for commercial use.

Nothing at the same size from Meta. Llama 3.1 8B is still the most recent 8B model in the Llama line, which is why it remains the default “small Llama” on Ollama:

• Llama 3.2 added 1B and 3B text models and 11B and 90B vision models (its two model cards list those four sizes), with no 8B. See Llama 3.2 1B and 3B and Llama 3.2 11B Vision.
• Llama 3.3 is a 70B-only release (its model card lists one size); there is no Llama 3.3 8B. See Llama 3.3 70B.
• Llama 4 Scout (April 2025) is Meta’s next generation: a 109B mixture-of-experts model with 17B active parameters that accepts images. Its Ollama tag is 67 GB, so it is not a replacement for anything that runs on an 8 GB card.

If you want the strongest model that fits 8 or 12 GB rather than this model specifically, the best Ollama models list is kept current.