Google Gemma 2 9B: VRAM, Ollama Tags, Benchmarks and Licence

What hardware does Gemma 2 9B need?

A 12 GB card runs every tag up to q6_K at the full 8K context. An 8 GB card runs q2_K at full context, or q3_K_M and the default q4_0 at Ollama's default 4,096-token context. No 9B file fits a 4 GB card with room for its context; use Gemma 2 2B there. With no GPU at all, 16 GB of system RAM holds the 5.4 GB default tag.

“VRAM needed” is arithmetic, not a measurement: the file size, plus the key-value cache for the whole 8,192-token context, plus 1 GB for compute buffers. The cache figure comes from config.json: 42 layers × 8 key-value heads × 256 dimensions × 2 (keys and values) × 2 bytes = 344,064 bytes per token, which is 2.82 GB at 8,192 tokens and 1.41 GB at 4,096. That assumes a full cache on every layer; Gemma 2 uses a 4,096-token sliding window on every other layer, so a runtime that exploits it will use somewhat less.

Hugging Face's launch post puts the unquantised bfloat16 weights at “about 18 GB”, which matches the 18 GB fp16 tag on Ollama. Add the cache and you are at a 24 GB card for full precision. For the Transformers library the same 18 GB applies in GPU memory, before activations.

With no GPU, Ollama loads the file into system RAM instead, so the question becomes whether the machine has roughly the download size plus the cache free. A 16 GB laptop can hold the default tag with the operating system still running; an 8 GB laptop cannot hold it comfortably, and the 2B model is the honest choice there. Speed on a processor depends on the machine and has not been measured on this site; the general picture is in running LLMs on CPU only.

Where a card is marginal, the two levers are a smaller quantisation (one row up in the table) and a shorter context. Ollama's default context is 4,096 tokens, which already halves the cache figure above; see the troubleshooting section for how to set it explicitly. What the quantisation levels mean is covered in the quantisation guide, and the general RAM and VRAM rules in Ollama system requirements.

What Gemma 2 9B is: the model card in one table

Gemma 2 is the second generation of Google DeepMind's open-weight Gemma family. Google announced the 9B and 27B sizes on June 27, 2024, and the 2B followed. The official model card for the instruction-tuned 9B is at huggingface.co/google/gemma-2-9b-it; the paper is Gemma 2: Improving Open Language Models at a Practical Size (arXiv:2408.00118, July 31, 2024). The table below is drawn from those two documents and the configuration file shipped with the weights.

Two architectural details are worth knowing because they show up in practice. The first is logit soft-capping: attention logits are capped at 50 and final logits at 30 with a tanh, which is why early Flash Attention builds could not train Gemma 2 and why some inference engines needed updates before it ran correctly. The second is the alternating sliding-window attention, which is what keeps the cache small enough for an 8 GB card at a reduced context.

On distillation, the paper says exactly this: “We also train the 2B and 9B models with knowledge distillation instead of next token prediction.” Google does not publish which model served as the teacher or any percentage of its capability that was “retained”; an earlier revision of this page printed such numbers, and they have been removed because there is no source for them.

The licence is the Gemma Terms of Use. Google describes it as commercially friendly, and it does allow commercial use, but it is not Apache 2.0: it incorporates a Prohibited Use Policy, requires you to pass the same terms on if you redistribute the weights or a fine-tune, and requires a notice file for non-hosted distribution. The terms page states that Gemma 4 is licensed separately. How the Gemma terms compare with Llama's and Apache 2.0 is covered in which local models you can legally use at work.

gemma-2-9b vs gemma-2-9b-it: which one do you want?

Google publishes two 9B checkpoints. google/gemma-2-9b is the pre-trained base model: it continues text and is the one you fine-tune. google/gemma-2-9b-it is the instruction-tuned version that follows a chat format, and it is the one almost everyone means when they say “Gemma 2 9B”. If you plan to type questions at it, you want -it.

On Ollama the naming is different and catches people out. The plain gemma2:9b tag is the instruct model (its full name is 9b-instruct-q4_0). The base model is published under 9b-text- tags, for example gemma2:9b-text-q4_0, at the same file sizes. There is no “9b-it” tag; searching Ollama for it finds nothing because the instruct build is the default.

One more naming point: the benchmark table on the model card is labelled “Gemma PT 9B”, where PT means pre-trained. The MMLU and GSM8K numbers people quote for Gemma 2 9B are therefore scores for the base checkpoint, measured by Google, and not scores for the -it model you download by default.

Gemma 2 9B MMLU, GSM8K and the other published scores

These are Google's figures from the three Gemma 2 model cards, for the pre-trained models. This site has not reproduced them. The 2B and 27B columns are filled in only for the rows those cards were checked for; a dash means not checked here, not that the figure does not exist.

Reading the 9B column: 71.3 on MMLU and 68.6 on GSM8K put it well above Gemma 1 7B (64.3 and 46.4) and within a few points of the 27B on the knowledge benchmarks, while the gap to the 27B is wider on code (40.2 vs 51.8 on HumanEval). Natural Questions at 29.2 is a reminder that a 9B model with no retrieval is a weak encyclopaedia; it reasons better than it recalls.

Sources: 9B card, 2B card, 27B card. The paper reports further results for the instruction-tuned models, including human-preference comparisons, which are not reproduced here.

Ollama tags and install

ollama pull gemma2 with no tag downloads the 9B instruct model at q4_0, 5.4 GB. Name a quantisation if you want a different size.

# Gemma 2 9B instruct, q4_0, 5.4 GB (the default)
ollama pull gemma2:9b

# Smaller file for an 8 GB card, 3.8 GB
ollama pull gemma2:9b-instruct-q2_K

# Higher-quality quantisation for a 16 GB card, 9.8 GB
ollama pull gemma2:9b-instruct-q8_0

# The pre-trained base model (continues text; does not chat)
ollama pull gemma2:9b-text-q4_0

# Chat
ollama run gemma2:9b

• Every 9B tag lists an 8K context on Ollama, matching the model card.
• Tag families: 9b-instruct- and 9b-text-, each followed by one of q2_K, q3_K_S, q3_K_M, q3_K_L, q4_0, q4_1, q4_K_S, q4_K_M, q5_0, q5_1, q5_K_S, q5_K_M, q6_K, q8_0 or fp16.
• Sizes: q3_K_S 4.3 GB, q3_K_L 5.1 GB, q4_1 6.0 GB, q5_0 and q5_K_S 6.5 GB, q5_1 7.0 GB, in addition to the rows in the hardware table.
• No tool calling. The Ollama library page has no tools badge, so agent frameworks that send tool definitions will be refused. If you need that, see the related models below.

Full tag list: ollama.com/library/gemma2/tags. The step-by-step install for the whole Gemma family, including Gemma 3 and 4 and Apple Silicon, is in run Gemma locally with Ollama.

Running gemma-2-9b-it with a Transformers pipeline, and API options

Outside Ollama, the model card's own example uses the Transformers pipeline API. Because the repository is gated, you must accept Google's terms on Hugging Face and log in (huggingface-cli login) before the download will start. The example, quoted from the card:

import torch
from transformers import pipeline

pipe = pipeline(
    "text-generation",
    model="google/gemma-2-9b-it",
    model_kwargs={"torch_dtype": torch.bfloat16},
    device="cuda",  # replace with "mps" to run on a Mac device
)

messages = [
    {"role": "user", "content": "Who are you? Please, answer in pirate-speak."},
]

outputs = pipe(messages, max_new_tokens=256)

In bfloat16 this needs about 18 GB of GPU memory, so it is a 24 GB-card or an A100/L4 job; on a smaller card, load with bitsandbytes 4-bit or 8-bit quantisation, or use a GGUF through llama.cpp or Ollama instead. Note that the messages list has only a user turn: Gemma 2's chat template has no system role, so put instructions in the first user message.

If you would rather call it than host it, the model card links Google's own hosted routes: Vertex AI Model Garden and Kaggle. Several third-party API gateways also list google/gemma-2-9b-it, some with a rate-limited free tier; those listings change often, so check the provider's current page rather than this one. For an API that stays on your machine, Ollama exposes the model at http://localhost:11434/api/chat once pulled.

Gemma 1 7B vs Gemma 2 9B: what changed

The first Gemma shipped as 2B and 7B. Gemma 2 replaced the 7B with a 9B, trained on a third more data with a different objective. The rows below come from the two model cards; the architecture row for Gemma 2 comes from its configuration file.

The maths and code rows are where the generation gap is largest: GSM8K went from 46.4 to 68.6 and MATH from 24.3 to 36.6. Everyday knowledge (HellaSwag) barely moved. If you still have Gemma 7B installed, the 9B is a straight upgrade at a similar download size.

Gemma 2 model sizes: 2B, 9B and 27B

All three share the 8,192-token context and the Gemma Terms of Use. The 2B and 9B were trained by distillation; the 27B was pre-trained from scratch. The 9B is the size that fits a 12 GB card with its full context, which is why it is the one most people run; the 27B at its 16 GB default tag wants a 24 GB card, and the 2B at 1.6 GB fits almost anything, including a 4 GB card.

Troubleshooting: context, memory and the settings that do not exist

# for every model the server loads
OLLAMA_CONTEXT_LENGTH=8192 ollama serve

# or inside one session
ollama run gemma2:9b
>>> /set parameter num_ctx 8192

Is Gemma 2 9B still worth running?

This section is opinion. Google released Gemma 3 on March 12, 2025 in 1B, 4B, 12B and 27B sizes with a 128K-token context, image input from the 4B up and training across more than 140 languages, and Gemma 4 has followed under its own licence. Both are covered on this site: Gemma 4: every model and what hardware runs it and the Gemma setup guide, which includes the Gemma 3 4B and 12B tags. There is no Gemma 3 at exactly 9B: the 12B wants a larger card than the Gemma 2 9B and the 4B a smaller one, so the 9B's size class has no direct successor.

That is the case for keeping it: a 9B at q4 is the largest Gemma that fits a 12 GB card with its full context, it has no tool calling and no vision to pay for, and the 8K context is enough for chat and short documents. If you need a longer context, tool use or more languages, move to Gemma 3 or 4. If you want an alternative at the same size with a 128K context and tool calling, Llama 3.1 8B is the usual comparison.

Frequently asked questions

Sources

• Gemma 2 9B instruct model card: parameters, training data, benchmark table, pipeline example
• Gemma 2: Improving Open Language Models at a Practical Size (arXiv:2408.00118): distillation, architecture
• Google's Gemma 2 announcement: release date, licence positioning
• Hugging Face: Welcome Gemma 2: context, sliding window, soft-capping, memory
• Gemma Terms of Use
• Ollama tag list for gemma2
• bartowski GGUF files: the file sizes in the hardware table
• Ollama FAQ: default context, environment variables