Run Gemma Locally with Ollama: Setup and VRAM

Published April 10, 2026 • Updated August 2026 • 18 min read

Two commands put Gemma on your machine: ollama pull gemma3:4b then ollama run gemma3:4b. The 4B variant is the right default for most people — it fits an 8GB machine at Q4, handles images as well as text, and carries a 128K context window. Step up to 12B if you have 16GB and want stronger reasoning; step down to 1B only if you are memory-starved and working with text alone.

Quick start: how do I get Gemma running?

Pull and run Gemma with two commands:

1. Pull the model: ollama pull gemma3:4b (2-3 minutes on broadband)
2. Start chatting: ollama run gemma3:4b

Want the newest generation instead? Swap in ollama pull gemma4:e4b -- Gemma 4 (released April 2026, Apache 2.0) is covered in detail below.

---

What this guide covers:

• Every Gemma variant from 270M to 31B and which to pick
• How to work out the VRAM each size and quantization needs
• The throughput ceiling your hardware imposes, and how to compute it
• MLX optimization for M-series Macs
• Fine-tuning Gemma on your own data with Unsloth
• How Gemma compares with Phi-4 and Llama 3.2 on specs and licensing

Google's Gemma family has become one of the strongest options for local AI. The models punch well above their weight class -- Gemma 3 4B matches or beats many 7-8B models from other families on reasoning and instruction following, and the newer Gemma 4 line (April 2026) pushes that further with multimodal audio/video and Apache 2.0 licensing. Google trains these on their TPU infrastructure with the same data pipeline used for Gemini, then releases the weights for commercial use -- under the custom Gemma License for Gemma 1-3 and the standard Apache 2.0 license starting with Gemma 4.

If you're new to running models locally, start with our Mac local AI setup guide or check the RAM requirements guide to confirm your hardware can handle the model size you want.

Table of Contents

1. The Gemma Model Family
2. How much VRAM does Gemma need?
3. Ollama Setup Step by Step
4. How fast will Gemma run on my machine?
5. MLX on Apple Silicon
6. Which quantization should I pick?
7. Multimodal Capabilities
8. Fine-Tuning with Unsloth
9. Gemma vs Phi-4 vs Llama 3.2
10. Best Use Cases
11. Frequently Asked Questions

---

The Gemma Model Family {#gemma-family}

Google has released four generations of Gemma. Here's the full lineup as of June 2026:

Gemma 4 (Latest)

Released April 2, 2026 under the Apache 2.0 license -- a notable shift from the custom Gemma License used by earlier generations. Apache 2.0 is a standard permissive open-source license, so Gemma 4 is cleaner to use commercially with no separate Google agreement. Every Gemma 4 size is multimodal (text + image), and the E2B and E4B edge variants add native video and audio input. The "E" in E2B/E4B stands for effective parameters -- these are edge-tuned models. The 26B is a Mixture-of-Experts model (about 4B active parameters per token); the 31B is dense.

Pull them in Ollama with ollama pull gemma4:e2b, gemma4:e4b, gemma4:26b, or gemma4:31b (ollama pull gemma4 defaults to E4B). The E2B is the smallest of the family and targets 8GB machines, while the 26B MoE activates only about 4B parameters per token — so it streams roughly a 4B model's worth of weights per token while drawing on 26B of knowledge, which is exactly why MoE architectures beat their size class on speed.

Gemma 3 (Previous Generation)

Still excellent and widely deployed. Released March 2025 under the Gemma License.

Gemma 2

Gemma 1 and Specialized Variants

For new setups, Gemma 4 E4B is the place to start -- Apache 2.0 licensing, 128K context, and audio/video input in a model that fits 8GB. If you are already running Gemma 3 4B, it remains a strong, well-supported choice: solid reasoning, vision tasks, 128K context, and a comfortable fit on 8GB hardware. Step up to the Gemma 4 26B (MoE) on 24GB for a clear jump in quality — it delivers 26B-level answers while running nearly as fast as a 4B.

---

How much VRAM does Gemma need? {#vram-requirements}

Weight footprint is arithmetic, not a mystery. A quantized model stores a roughly fixed number of bits per parameter:

weights (GB) ≈ parameters (billions) × bits-per-weight ÷ 8

Q4_K_M averages about 4.5 bits per weight. The table below runs that arithmetic exactly; for mental maths, about 0.6 GB per billion parameters rounds it up a little to cover the embedding table and file metadata a real GGUF also carries.

Two caveats worth knowing, because they are where the formula and reality part company:

• Gemma's vocabulary is unusually large (roughly a quarter of a million tokens), so the embedding layer is a meaningful fraction of the smallest models. Expect the 1B in particular to weigh more on disk than the formula suggests.
• The vision tower on the multimodal variants is extra, and it is not always quantized alongside the language weights.

For the authoritative figure, Ollama's Gemma 3 library page lists the exact download size of every tag, and ollama list shows what actually landed on your disk. Use the formula for planning and the library page for precision.

Do not forget the KV cache

Weights are only half the budget. Context is stored in a key/value cache that grows with every token:

KV cache bytes per token = 2 × layers × KV-heads × head-dim × bytes-per-element

The counts come from each model's config.json on Hugging Face. Run this before you raise num_ctx — a 128K context window on a small model can cost more memory than the weights do, which is why a model that "fits" at 4K can fall over at 32K.

What This Means for Your Hardware

Wondering how much quality Q4_K_M actually costs? Do not trust a percentage copied from a blog post — the llama.cpp maintainers publish perplexity deltas per quantization type in the repo's quantize documentation, and those figures are revised as the formats change. Our AWQ vs GPTQ vs GGUF comparison covers which format to choose in the first place.

---

Ollama Setup Step by Step {#ollama-setup}

Install Ollama (If Needed)

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Start the service
ollama serve

Pull Gemma Models

# Gemma 4 - newest generation (April 2026, Apache 2.0)
ollama pull gemma4:e2b       # ~2B effective - edge, ~5GB RAM at 4-bit
ollama pull gemma4:e4b       # ~4B effective - best small balance (default)
ollama pull gemma4:26b       # 26B MoE (~4B active) - efficient large
ollama pull gemma4:31b       # 31B dense - maximum quality

# Gemma 3 - previous generation (still excellent)
ollama pull gemma3:1b        # 1B - ultra fast, basic tasks
ollama pull gemma3:4b        # 4B - best balance
ollama pull gemma3:12b       # 12B - strong reasoning
ollama pull gemma3:27b       # 27B - maximum quality

# Specific quantization
ollama pull gemma3:4b-q8_0   # Higher quality 4B
ollama pull gemma3:12b-q4_K_M # Fits in 16GB

# Gemma 2 (still excellent)
ollama pull gemma2:2b
ollama pull gemma2:9b
ollama pull gemma2:27b

# Code-specific
ollama pull codegemma:7b

Verify Installation

# Check model is downloaded
ollama list

# Quick test
ollama run gemma3:4b "What is the capital of France? Answer in one sentence."

# Check model details
ollama show gemma3:4b

Run with Custom Parameters

# Create a Modelfile for custom settings
cat > Modelfile << 'EOF'
FROM gemma3:4b
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 8192
PARAMETER repeat_penalty 1.1
SYSTEM "You are a precise, helpful assistant. Give concise answers with specific details. When you're unsure, say so."
EOF

# Create custom model
ollama create my-gemma -f Modelfile

# Run it
ollama run my-gemma

---

How fast will Gemma run on my machine? {#benchmarks}

You can work out the answer before you download anything. Single-stream generation is memory-bandwidth bound: producing one token requires streaming every weight through the compute units once, so there is a hard arithmetic ceiling.

tokens/sec ceiling = memory bandwidth (GB/s) ÷ model size (GB)

Take the Q4_K_M sizes computed in the previous section, divide each platform's published memory bandwidth by them, and you get the fastest that platform could possibly go:

Read every cell as a ceiling, never as a prediction. Attention over the KV cache, sampling, kernel overhead and imperfect memory access all cost time the formula ignores, so a real run always lands below its row — often well below. The value of the table is not the number, it is the ratio: it tells you which upgrade actually moves the needle, and it flags any published figure that exceeds it as wrong.

Two things fall straight out of the arithmetic. First, if a model does not fit in memory the row is meaningless — spilling to CPU or disk replaces GPU bandwidth with something an order of magnitude slower, which is why capacity beats speed when choosing a card. Second, model size and throughput trade off exactly one-for-one: the 4B is roughly three times faster than the 12B on identical hardware because it is roughly a third the size.

Anything above about 30 tokens/second reads as instant in interactive chat; below 10 starts to feel like waiting. Find your row, halve it as a working assumption, and you will know which Gemma size suits your machine before you pull it.

If you want a measured number rather than a bound, our benchmarking guide walks through capturing generation rate, prefill rate and time-to-first-token reproducibly.

---

MLX on Apple Silicon {#mlx-apple-silicon}

If you have an M-series Mac, MLX can squeeze extra performance out of Gemma. MLX is Apple's machine learning framework designed specifically for Apple Silicon's unified memory architecture.

Install MLX

pip install mlx-lm

Download and Run Gemma with MLX

# Download quantized Gemma 3 for MLX
mlx_lm.generate \
  --model mlx-community/gemma-3-4b-it-4bit \
  --prompt "Explain quantum computing in simple terms" \
  --max-tokens 500

# Interactive chat
mlx_lm.chat --model mlx-community/gemma-3-4b-it-4bit

Is MLX faster than Ollama on a Mac?

Usually, yes — both run against the same memory-bandwidth ceiling, so neither can beat the table above, but MLX generally gets closer to it. The advantage is architectural: MLX is written by Apple specifically for unified memory, so its kernels and memory access patterns are tuned for the hardware in a way a portable runtime cannot match. Apple publishes benchmarks and example scripts in the mlx-examples repository; run mlx_lm.generate and ollama run --verbose on your own Mac with the same prompt and you will have a number that applies to your machine rather than someone else's.

The tradeoff runs the other way: MLX has no API server, no model management, and no ecosystem of client apps. Use MLX when raw speed in a script matters; use Ollama when you need an API endpoint or tools like Open WebUI.

Converting Models for MLX

# Convert any HuggingFace model to MLX format
mlx_lm.convert \
  --hf-path google/gemma-3-4b-it \
  --mlx-path ./gemma-3-4b-mlx \
  --quantize --q-bits 4

---

Which quantization should I pick? {#quantization}

Quantization reduces the bits stored per weight to save memory. Because throughput is bandwidth-bound, a smaller file is also a faster file — the two move together, which is why the size column below doubles as a speed ranking.

Sizes are computed from the bits-per-weight formula above, so treat them as planning estimates and check ollama list for the byte-exact figure once you have pulled a tag.

Recommendation: Q4_K_M, unless you have a specific reason otherwise. It cuts memory by roughly 72% against FP16, which is where the size curve flattens — dropping to Q3 saves under half a gigabyte on a 4B and costs real reasoning quality, while climbing to Q5 costs about 0.6GB for a difference most people cannot detect in chat. Two cases justify going higher: strict structured output such as valid JSON or exact arithmetic, and small models where the higher quant fits comfortably anyway. For quality deltas per quant, read the perplexity data llama.cpp publishes rather than any single blog's percentage.

How to Choose

# Check available quantizations
ollama show gemma3:4b --modelfile

# Pull specific quantization
ollama pull gemma3:4b-q8_0     # Maximum quality
ollama pull gemma3:4b-q5_K_M   # Good balance
ollama pull gemma3:4b-q4_K_M   # Memory efficient (default)

For a deeper comparison of quantization formats, see our AWQ vs GPTQ vs GGUF comparison.

---

Multimodal Capabilities {#multimodal}

Gemma 3 4B, 12B, and 27B are multimodal -- they accept both text and images. This works out of the box in Ollama.

Image Analysis with Ollama

# Describe an image
ollama run gemma3:4b "What's in this image?" ./photo.jpg

# Extract text from a screenshot
ollama run gemma3:12b "Extract all text visible in this image" ./screenshot.png

# Analyze a chart
ollama run gemma3:4b "What trends does this chart show?" ./quarterly_revenue.png

Via the API

# Base64 encode an image and send to Ollama API
curl http://localhost:11434/api/generate -d '{
  "model": "gemma3:4b",
  "prompt": "Describe this image in detail",
  "images": ["'$(base64 -i photo.jpg)'"]
}'

Vision Performance

Gemma 3 4B handles basic image understanding -- object identification, text extraction, simple visual Q&A. For complex image reasoning (counting objects, spatial relationships, detailed chart analysis), the 12B or 27B variants perform noticeably better.

The 1B model is text-only. If you need vision on constrained hardware, the 4B is your only Gemma option under 8GB.

---

Fine-Tuning with Unsloth {#fine-tuning}

Gemma models respond extremely well to fine-tuning. With QLoRA, you can fine-tune Gemma 3 4B on a GPU with just 6GB VRAM.

Install Unsloth

pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
pip install --no-deps trl peft accelerate bitsandbytes

Fine-Tuning Script

from unsloth import FastLanguageModel

# Load Gemma with 4-bit quantization
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/gemma-3-4b-it-bnb-4bit",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)

# Add LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
)

# Prepare your dataset
from datasets import load_dataset
dataset = load_dataset("json", data_files="my_training_data.jsonl")

# Format: {"instruction": "...", "input": "...", "output": "..."}

from trl import SFTTrainer
from transformers import TrainingArguments

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset["train"],
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        max_steps=100,
        learning_rate=2e-4,
        fp16=True,
        logging_steps=1,
        output_dir="outputs",
    ),
)

trainer.train()

# Save the fine-tuned model
model.save_pretrained_merged("gemma-finetuned", tokenizer)

Export to Ollama

# Convert to GGUF
python llama.cpp/convert_hf_to_gguf.py gemma-finetuned \
  --outtype q4_K_M \
  --outfile gemma-finetuned.gguf

# Create Ollama model
cat > Modelfile << 'EOF'
FROM ./gemma-finetuned.gguf
TEMPLATE """<start_of_turn>user
{{ .Prompt }}<end_of_turn>
<start_of_turn>model
{{ .Response }}<end_of_turn>"""
PARAMETER stop "<end_of_turn>"
EOF

ollama create my-gemma-finetuned -f Modelfile
ollama run my-gemma-finetuned

For a comprehensive fine-tuning walkthrough beyond Gemma, see our LoRA fine-tuning local guide.

Unsloth publishes its own speed and memory claims — roughly 2x faster training against a standard HuggingFace loop — along with the benchmark scripts behind them on the Unsloth GitHub repository. Those are the vendor's figures on the vendor's hardware, so treat them as a ceiling and run the scripts yourself before planning a training budget around them.

---

Gemma vs Phi-4 vs Llama 3.2 {#comparison}

The honest comparison at this size class is on specifications and licensing, not on scores. Benchmark numbers for these models are published by the labs that trained them, on evaluation harnesses they chose, and they get revised — so the useful move is to read the current figures on each model card yourself rather than trust a table copied into a blog post months ago.

What does not drift is the spec sheet. VRAM figures below are computed with the 0.6 GB per billion parameters rule from earlier:

4B Class

12B Class

What the specs alone tell you:

• Phi-4's MIT license is the most permissive of the three. If your legal review is the bottleneck rather than quality, that is the deciding column — and Gemma 4 moves to Apache 2.0 for the same reason.
• Context window is the sharpest split at 12B. Phi-4's 16K against Gemma's 128K is not a marginal difference; it decides whether long-document work is possible at all.
• Vision is not universal. Llama 3.2 3B is text-only and Phi-4 14B is text-only, so if you need image input in a small model the field narrows fast.
• The three are within about 2GB of each other at Q4, so memory is rarely the tiebreaker at this size — capability and licensing are.

Read the current scores at the source: the Gemma 3 4B model card publishes Google's evaluation table, and each of the other model cards does the same. For a broader survey of small local models, check our small language models guide.

---

Best Use Cases {#use-cases}

Where Gemma Excels

Document analysis and summarization. The 128K context window combined with multimodal support means Gemma 3 can process long documents and images in a single pass. Feed it a 50-page PDF and ask for a structured summary.

Multilingual tasks. Google trained Gemma on data spanning 30+ languages. It handles translation, multilingual Q&A, and cross-lingual retrieval better than most open models its size.

Instruction following. Gemma's instruction-tuned variants follow complex, multi-step instructions with high reliability. This makes them excellent for structured output tasks like JSON generation, data extraction, and template filling.

Where Other Models Are Better

Pure coding tasks. If you write code all day, Phi-4 or Qwen2.5-Coder will serve you better. Gemma is competent at code but not a specialist.

Creative writing. Llama 3.2 and Mistral produce more varied, creative prose. Gemma tends toward factual, concise responses -- great for work, less great for fiction.

Constrained memory (<4GB). The Gemma 3 1B is decent but the Phi-4 Mini at 3.8B Q2_K provides meaningfully better quality in a similar memory footprint.

---

Troubleshooting

Model Won't Load

# Check available memory
nvidia-smi   # GPU
free -h       # System RAM

# Try smaller quantization
ollama pull gemma3:4b-q4_0

# Force CPU mode if GPU memory is full
CUDA_VISIBLE_DEVICES="" ollama run gemma3:4b

Slow Generation

# Reduce context window
ollama run gemma3:4b --num-ctx 4096

# Check if model is using GPU
ollama ps   # Shows GPU memory usage per model

# On Mac, verify Metal is active
system_profiler SPDisplaysDataType | grep Metal

Vision Not Working

# Only 4B, 12B, 27B support vision
# 1B is text-only

# Verify with API
curl http://localhost:11434/api/show -d '{"name": "gemma3:4b"}' | grep -i "vision"

---

Frequently Asked Questions {#faq}

How much VRAM does Gemma 3 4B need?

At Q4_K_M the weights come to roughly 2.4GB by the 0.6 GB per billion parameters rule, and Ollama's library page lists the byte-exact download size for the tag you pull. Budget above that for the KV cache, which grows with your context length. An 8GB GPU or 8GB Apple Silicon Mac handles the Q4 build with room for the operating system and a moderate context window.

Is Gemma 3 better than Llama 3.2 for local use?

It depends on which column you are optimizing. Gemma 3 4B accepts image input and carries a 128K context window; Llama 3.2 3B is text-only and about half a gigabyte smaller at Q4. Both labs publish evaluation tables on their model cards — read the current ones rather than a comparison copied from a blog, because they are revised as the models are updated.

Can I run Gemma on Apple Silicon with Metal acceleration?

Yes, and Ollama enables it automatically. For the speed you can expect, divide your Mac's published memory bandwidth by the model's file size in GB — an M2 at 100 GB/s against a 2.4GB Gemma 3 4B gives a ceiling near 42 tok/s, and real output lands below that. MLX generally gets closer to the ceiling than Ollama does, at the cost of the API server and tooling.

Does Gemma 3 support image input?

Gemma 3 4B, 12B and 27B are multimodal and accept both text and images. The 1B variant is text-only. In Ollama, pass an image path after your prompt. Via the API, send base64-encoded images in the images array.

Can I fine-tune Gemma locally on consumer hardware?

Yes. QLoRA with Unsloth trains against 4-bit quantized base weights with 16-bit LoRA adapters, which is what keeps a 4B fine-tune inside a small VRAM budget. Wall-clock time depends entirely on your dataset size, sequence length, step count and card, so run a short job first and extrapolate rather than trusting anyone's headline figure.

What is the Gemma license? Can I use it commercially?

Gemma 1 through 3 ship under Google's custom Gemma Terms of Use, which permit commercial use but attach a Prohibited Use Policy that you must pass on to anyone you redistribute to. Gemma 4 moves to standard Apache 2.0, which is why it is the cleaner starting point for commercial work. Read the current terms directly before shipping — a summary in a blog post is not legal review.

How does MLX compare to Ollama for Gemma on Mac?

Both are bound by the same memory-bandwidth ceiling; MLX typically runs closer to it because Apple wrote it for unified memory specifically. Ollama gives you a REST API, model management and client apps like Open WebUI, which MLX does not. Run both against the same prompt on your own Mac to see the size of the gap on your hardware.

Which Gemma model should I use for coding?

CodeGemma 7B is the code-trained member of the family. For general-purpose work that includes occasional code, Gemma 3 4B or 12B handle it competently. If coding is your primary use case, evaluate Qwen2.5-Coder and Phi-4 alongside it — both were built with code as the priority rather than a secondary capability.

---

Conclusion

Gemma 3 earns its spot as a top-tier local model family. The 4B variant delivers a rare combination: vision support, a 128K context window, and a weight footprint under 3GB at Q4. That is a lot of capability in a small package.

Start with ollama pull gemma3:4b (or the newer ollama pull gemma4:e4b) and run it for a week as your daily driver. If you hit quality ceilings on complex reasoning tasks, step up to a 12B model. If you need peak performance for production workloads, Gemma 3 27B -- or Gemma 4's 26B MoE / 31B dense -- stays competitive with models twice its parameter count. For brand-new setups, Gemma 4 is the better starting point: it adds audio/video input and the more permissive Apache 2.0 license.

The model weights and technical documentation are available on Google's Gemma page and the Google organization on HuggingFace.

---

Looking for a model comparison that covers the full local AI landscape? Our best local AI models for 8GB RAM guide ranks every major family by real-world usability on consumer hardware.