Llama 2 70B: architecture, file sizes and hardware requirements

How many layers does Llama 2 70B have?

80 layers, each 8192 wide. The values below are copied from the config.json shipped with the weights. Meta's own repositories are gated behind a licence form, so they were read from two ungated fp16 mirrors, TheBloke/Llama-2-70B-fp16 and NousResearch/Llama-2-70b-hf, which agree on every field.

Where the 68.98 billion parameters come from

The count follows from the config. In each layer the query and output projections are 8192 × 8192 and the key and value projections are 8192 × 1024 (8 key/value heads of 128 dimensions), which comes to 151.0 million attention weights. The MLP has three 8192 × 28672 matrices, 704.6 million weights. That is 855.6 million per layer and 68.45 billion across 80 layers. The input embedding and the output head are each 32,000 × 8192, adding 0.52 billion. The total, 68.98 billion, matches the 68,976,653,312 that Hugging Face reports for the safetensors files.

Grouped-query attention is the one architectural difference between the 70B and its smaller siblings. Meta's model card marks GQA as used only on the 70B, “for improved inference scalability”. Sharing 8 key/value heads across 64 query heads makes the KV cache an eighth of what full multi-head attention would need. The Llama 2 13B uses standard attention.

Llama 2 70B hardware requirements

Plan for about 44 GB of GPU or unified memory to run the Q4_K_M file with its full context: two 24 GB cards, one 48 GB card, or a 64 GB Mac. The smallest published file, Q2_K, is 29 GB and still does not fit a 24 GB card. The arithmetic is parameters × bits per weight: 68.98 billion × 16 bits ÷ 8 is 138 GB at fp16, and the 41.4 GB Q4_K_M file works out to 4.8 bits per weight.

File sizes are from Ollama's llama2 tag list and agree with the GGUF files in TheBloke/Llama-2-70B-Chat-GGUF. The memory column is the file plus 1.3 GB of KV cache for the full 4,096-token context (80 layers × 8 KV heads × 128 dimensions × 2 × 2 bytes is 328 KB per token) plus a 1 GB allowance for compute buffers. It is arithmetic, not a measurement.

With one 24 GB card, llama.cpp and Ollama will put as many layers as fit on the GPU and run the rest from system RAM. That works with 64 GB of RAM, but the layers left on the CPU set the pace, so expect it to be slow. On a Mac, macOS limits how much unified memory the GPU may use, which is why 64 GB is the practical floor for the 4-bit files; the Mac memory pressure guide explains the limit.

If you have a single 24 GB or 32 GB card, a newer and smaller model is a better use of it than a 2-bit Llama 2 70B: see the best LLM for 24 GB of VRAM and the 32 GB tier. The VRAM calculator runs the same sums for other models, and quantization explained covers what the K-quant names mean.

Ollama tags and install

llama2:70b is the chat model at q4_0, a 39 GB download. It shares a digest with llama2:70b-chat and llama2:70b-chat-q4_0, so all three names pull the same file. The 70b-text tags are the pretrained completion model with no chat tuning. Every 70B tag is listed with a 4K context.

# Chat model, q4_0, 39 GB
ollama pull llama2:70b
ollama run llama2:70b

# Q4_K_M, 41 GB
ollama pull llama2:70b-chat-q4_K_M

# Smallest published file, Q2_K, 29 GB
ollama pull llama2:70b-chat-q2_K

# Pretrained completion model (not chat-tuned), 39 GB
ollama pull llama2:70b-text

The original weights are on Hugging Face at meta-llama/Llama-2-70b-chat-hf, behind Meta's licence form. Memory by model size for everything else on Ollama is in the Ollama system requirements guide.

Benchmarks Meta reported

These are Meta's numbers for the pretrained 70B, copied from the table in the Llama 2 model card and produced with Meta's internal evaluation library. This site has not run them. Most rows are averages over several benchmarks, as the labels say. On TruthfulQA the card reports 50.18 for the pretrained 70B and 64.14 for the chat model.

Training details from the same card: 2.0 trillion pretraining tokens, trained between January and July 2023, pretraining data cut off in September 2022, and 1,720,320 GPU hours on A100-80GB hardware for the 70B. The full method is in the paper, Llama 2: Open Foundation and Fine-Tuned Chat Models (arXiv:2307.09288).

What replaced Llama 2 70B

Meta has shipped three generations since. If you have the hardware for Llama 2 70B you have the hardware for either 70B successor, and the 4K context is the main reason to move: it is small enough that a long document or a multi-file coding session will not fit. Sizes are from the Ollama tag lists linked under Sources.

Related pages from the same era: CodeLlama 70B, the code-tuned model built on Llama 2, and Mixtral 8x7B. For what to run today on two 24 GB cards or one, start with the best local models for a 24 GB GPU.

Frequently asked questions

Sources

• Llama 2 model card — Meta: sizes, context, GQA, training data, benchmark table, licence
• Llama 2: Open Foundation and Fine-Tuned Chat Models — the paper, arXiv:2307.09288, 18 July 2023
• config.json (fp16 mirror) — layers, hidden size, heads, vocabulary, context
• Ollama llama2 tags — every 70B tag with its download size
• TheBloke/Llama-2-70B-Chat-GGUF — GGUF files for llama.cpp
• Ollama llama3.1 tags, llama3.3 tags and llama4 tags — successor sizes