Run Google Gemma Locally: Ollama Setup Guide

Published on April 10, 2026 • 24 min read

Quick Start: Gemma Running in 60 Seconds

Pull and run Gemma with two commands:

1. Pull the model: ollama pull gemma3:4b (2-3 minutes on broadband)
2. Start chatting: ollama run gemma3:4b

You now have a Gemma model running on your hardware. No API key, no usage limits. Want the newest generation instead? Swap in ollama pull gemma4:e4b -- Gemma 4 (released April 2026, Apache 2.0) covered in detail below.

---

What this guide covers:

• Every Gemma variant from 270M to 27B and which to pick
• Exact VRAM requirements for each model size and quantization
• Real performance numbers on consumer GPUs and Apple Silicon
• MLX optimization for M-series Macs
• Fine-tuning Gemma on your own data with Unsloth
• Head-to-head comparison with Phi-4 and Llama 3.2

Google's Gemma family has become one of the strongest options for local AI. The models punch well above their weight class -- Gemma 3 4B matches or beats many 7-8B models from other families on reasoning and instruction following, and the newer Gemma 4 line (April 2026) pushes that further with multimodal audio/video and Apache 2.0 licensing. Google trains these on their TPU infrastructure with the same data pipeline used for Gemini, then releases the weights for commercial use -- under the custom Gemma License for Gemma 1-3 and the standard Apache 2.0 license starting with Gemma 4.

If you're new to running models locally, start with our Mac local AI setup guide or check the RAM requirements guide to confirm your hardware can handle the model size you want.

Table of Contents

1. The Gemma Model Family
2. VRAM Requirements
3. Ollama Setup Step by Step
4. Performance Benchmarks
5. MLX on Apple Silicon
6. Quantization Options
7. Multimodal Capabilities
8. Fine-Tuning with Unsloth
9. Gemma vs Phi-4 vs Llama 3.2
10. Best Use Cases

---

The Gemma Model Family {#gemma-family}

Google has released four generations of Gemma. Here's the full lineup as of June 2026:

Gemma 4 (Latest)

Released April 2, 2026 under the Apache 2.0 license -- a notable shift from the custom Gemma License used by earlier generations. Apache 2.0 is a standard permissive open-source license, so Gemma 4 is cleaner to use commercially with no separate Google agreement. Every Gemma 4 size is multimodal (text + image), and the E2B and E4B edge variants add native video and audio input. The "E" in E2B/E4B stands for effective parameters -- these are edge-tuned models. The 26B is a Mixture-of-Experts model (about 4B active parameters per token); the 31B is dense.

Pull them in Ollama with ollama pull gemma4:e2b, gemma4:e4b, gemma4:26b, or gemma4:31b (ollama pull gemma4 defaults to E4B). The E2B runs in roughly 5GB of RAM at 4-bit, making it a strong pick for 8GB machines, while the 26B MoE gives 26B-class quality at nearly 4B speed for agentic and multimodal work.

Gemma 3 (Previous Generation)

Still excellent and widely deployed. Released March 2025 under the Gemma License.

Gemma 2

Gemma 1 and Specialized Variants

For new setups, Gemma 4 E4B is the place to start -- Apache 2.0 licensing, 128K context, and audio/video input in a model that fits 8GB. If you are already running Gemma 3 4B, it remains a strong, well-supported choice: solid reasoning, vision tasks, 128K context, and a comfortable fit on 8GB hardware. Step up to the Gemma 4 26B (MoE) on 24GB for a clear jump in quality — it delivers 26B-level answers while running nearly as fast as a 4B.

---

VRAM Requirements {#vram-requirements}

These are measured VRAM numbers, not theoretical estimates. Tested with Ollama's default quantization (Q4_K_M for most sizes).

Gemma 3 VRAM Usage

What This Means for Your Hardware

The Q4_K_M quantization retains roughly 97% of full-precision quality for instruction following. You lose maybe 1-2% on complex reasoning benchmarks. For most practical tasks, you will not notice the difference.

---

Ollama Setup Step by Step {#ollama-setup}

Install Ollama (If Needed)

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Start the service
ollama serve

Pull Gemma Models

# Gemma 4 - newest generation (April 2026, Apache 2.0)
ollama pull gemma4:e2b       # ~2B effective - edge, ~5GB RAM at 4-bit
ollama pull gemma4:e4b       # ~4B effective - best small balance (default)
ollama pull gemma4:26b       # 26B MoE (~4B active) - efficient large
ollama pull gemma4:31b       # 31B dense - maximum quality

# Gemma 3 - previous generation (still excellent)
ollama pull gemma3:1b        # 1B - ultra fast, basic tasks
ollama pull gemma3:4b        # 4B - best balance
ollama pull gemma3:12b       # 12B - strong reasoning
ollama pull gemma3:27b       # 27B - maximum quality

# Specific quantization
ollama pull gemma3:4b-q8_0   # Higher quality 4B
ollama pull gemma3:12b-q4_K_M # Fits in 16GB

# Gemma 2 (still excellent)
ollama pull gemma2:2b
ollama pull gemma2:9b
ollama pull gemma2:27b

# Code-specific
ollama pull codegemma:7b

Verify Installation

# Check model is downloaded
ollama list

# Quick test
ollama run gemma3:4b "What is the capital of France? Answer in one sentence."

# Check model details
ollama show gemma3:4b

Run with Custom Parameters

# Create a Modelfile for custom settings
cat > Modelfile << 'EOF'
FROM gemma3:4b
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 8192
PARAMETER repeat_penalty 1.1
SYSTEM "You are a precise, helpful assistant. Give concise answers with specific details. When you're unsure, say so."
EOF

# Create custom model
ollama create my-gemma -f Modelfile

# Run it
ollama run my-gemma

---

Performance Benchmarks {#benchmarks}

Real-world measurements from our test hardware. All numbers are tokens per second for generation (not prompt processing).

GPU Benchmarks (Gemma 3)

*Partial GPU offload

Apple Silicon Benchmarks (Gemma 3)

*With limited context window

30+ tokens/second feels instant for interactive chat. Below 10 tok/s starts feeling sluggish. These numbers show Gemma 3 4B delivers a snappy experience on almost any modern hardware.

---

MLX on Apple Silicon {#mlx-apple-silicon}

If you have an M-series Mac, MLX can squeeze extra performance out of Gemma. MLX is Apple's machine learning framework designed specifically for Apple Silicon's unified memory architecture.

Install MLX

pip install mlx-lm

Download and Run Gemma with MLX

# Download quantized Gemma 3 for MLX
mlx_lm.generate \
  --model mlx-community/gemma-3-4b-it-4bit \
  --prompt "Explain quantum computing in simple terms" \
  --max-tokens 500

# Interactive chat
mlx_lm.chat --model mlx-community/gemma-3-4b-it-4bit

MLX vs Ollama Performance on Apple Silicon

MLX typically delivers 20-30% faster inference than Ollama on Apple Silicon. The advantage comes from tighter Metal integration and memory access patterns optimized for unified memory. The tradeoff: MLX lacks Ollama's API server, model management, and ecosystem of client apps. Use MLX when raw speed matters; use Ollama when you need an API or compatible tools like Open WebUI.

Converting Models for MLX

# Convert any HuggingFace model to MLX format
mlx_lm.convert \
  --hf-path google/gemma-3-4b-it \
  --mlx-path ./gemma-3-4b-mlx \
  --quantize --q-bits 4

---

Quantization Options {#quantization}

Quantization reduces model precision to save memory. Here's how different levels affect Gemma 3 4B:

Quantization Quality Comparison

Recommendation: Q4_K_M is the default for good reason. You lose about 1.3 points on MMLU compared to full precision -- barely noticeable in practice -- while cutting memory usage by 64%. Drop to Q3_K_M only if you absolutely need to fit in tight memory. Avoid Q2_K for anything beyond basic chat.

How to Choose

# Check available quantizations
ollama show gemma3:4b --modelfile

# Pull specific quantization
ollama pull gemma3:4b-q8_0     # Maximum quality
ollama pull gemma3:4b-q5_K_M   # Good balance
ollama pull gemma3:4b-q4_K_M   # Memory efficient (default)

For a deeper comparison of quantization formats, see our AWQ vs GPTQ vs GGUF comparison.

---

Multimodal Capabilities {#multimodal}

Gemma 3 4B, 12B, and 27B are multimodal -- they accept both text and images. This works out of the box in Ollama.

Image Analysis with Ollama

# Describe an image
ollama run gemma3:4b "What's in this image?" ./photo.jpg

# Extract text from a screenshot
ollama run gemma3:12b "Extract all text visible in this image" ./screenshot.png

# Analyze a chart
ollama run gemma3:4b "What trends does this chart show?" ./quarterly_revenue.png

Via the API

# Base64 encode an image and send to Ollama API
curl http://localhost:11434/api/generate -d '{
  "model": "gemma3:4b",
  "prompt": "Describe this image in detail",
  "images": ["'$(base64 -i photo.jpg)'"]
}'

Vision Performance

Gemma 3 4B handles basic image understanding -- object identification, text extraction, simple visual Q&A. For complex image reasoning (counting objects, spatial relationships, detailed chart analysis), the 12B or 27B variants perform noticeably better.

The 1B model is text-only. If you need vision on constrained hardware, the 4B is your only Gemma option under 8GB.

---

Fine-Tuning with Unsloth {#fine-tuning}

Gemma models respond extremely well to fine-tuning. With QLoRA, you can fine-tune Gemma 3 4B on a GPU with just 6GB VRAM.

Install Unsloth

pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
pip install --no-deps trl peft accelerate bitsandbytes

Fine-Tuning Script

from unsloth import FastLanguageModel

# Load Gemma with 4-bit quantization
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/gemma-3-4b-it-bnb-4bit",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)

# Add LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
)

# Prepare your dataset
from datasets import load_dataset
dataset = load_dataset("json", data_files="my_training_data.jsonl")

# Format: {"instruction": "...", "input": "...", "output": "..."}

from trl import SFTTrainer
from transformers import TrainingArguments

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset["train"],
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        max_steps=100,
        learning_rate=2e-4,
        fp16=True,
        logging_steps=1,
        output_dir="outputs",
    ),
)

trainer.train()

# Save the fine-tuned model
model.save_pretrained_merged("gemma-finetuned", tokenizer)

Export to Ollama

# Convert to GGUF
python llama.cpp/convert_hf_to_gguf.py gemma-finetuned \
  --outtype q4_K_M \
  --outfile gemma-finetuned.gguf

# Create Ollama model
cat > Modelfile << 'EOF'
FROM ./gemma-finetuned.gguf
TEMPLATE """<start_of_turn>user
{{ .Prompt }}<end_of_turn>
<start_of_turn>model
{{ .Response }}<end_of_turn>"""
PARAMETER stop "<end_of_turn>"
EOF

ollama create my-gemma-finetuned -f Modelfile
ollama run my-gemma-finetuned

For a comprehensive fine-tuning walkthrough beyond Gemma, see our LoRA fine-tuning local guide.

Unsloth claims 2x training speed over standard HuggingFace training. In our testing with Gemma 3 4B, we measured 1.7x speedup -- still significant. Full details on the Unsloth GitHub repository.

---

Gemma vs Phi-4 vs Llama 3.2 {#comparison}

The three strongest open model families for local use, compared head-to-head at similar sizes:

4B Class Models

12B Class Models

Key takeaways:

• Gemma 3 4B wins on general knowledge (MMLU, ARC) while being smaller than Phi-4 Mini
• Phi-4 wins on math and code (GSM8K, HumanEval) across both size classes
• Llama 3.2 is the most memory-efficient but trails on every benchmark
• Gemma 3 has the longest context (128K) at every size, which matters for document analysis
• Vision capability on the 4B model gives Gemma a unique advantage in its size class

For a broader comparison of small local models, check our small language models guide.

---

Best Use Cases {#use-cases}

Where Gemma Excels

Document analysis and summarization. The 128K context window combined with multimodal support means Gemma 3 can process long documents and images in a single pass. Feed it a 50-page PDF and ask for a structured summary.

Multilingual tasks. Google trained Gemma on data spanning 30+ languages. It handles translation, multilingual Q&A, and cross-lingual retrieval better than most open models its size.

Instruction following. Gemma's instruction-tuned variants follow complex, multi-step instructions with high reliability. This makes them excellent for structured output tasks like JSON generation, data extraction, and template filling.

Where Other Models Are Better

Pure coding tasks. If you write code all day, Phi-4 or Qwen2.5-Coder will serve you better. Gemma is competent at code but not a specialist.

Creative writing. Llama 3.2 and Mistral produce more varied, creative prose. Gemma tends toward factual, concise responses -- great for work, less great for fiction.

Constrained memory (<4GB). The Gemma 3 1B is decent but the Phi-4 Mini at 3.8B Q2_K provides meaningfully better quality in a similar memory footprint.

---

Troubleshooting

Model Won't Load

# Check available memory
nvidia-smi   # GPU
free -h       # System RAM

# Try smaller quantization
ollama pull gemma3:4b-q4_0

# Force CPU mode if GPU memory is full
CUDA_VISIBLE_DEVICES="" ollama run gemma3:4b

Slow Generation

# Reduce context window
ollama run gemma3:4b --num-ctx 4096

# Check if model is using GPU
ollama ps   # Shows GPU memory usage per model

# On Mac, verify Metal is active
system_profiler SPDisplaysDataType | grep Metal

Vision Not Working

# Only 4B, 12B, 27B support vision
# 1B is text-only

# Verify with API
curl http://localhost:11434/api/show -d '{"name": "gemma3:4b"}' | grep -i "vision"

---

Conclusion

Gemma 3 earns its spot as a top-tier local model family. The 4B variant delivers a rare combination: vision support, 128K context, strong benchmarks, and 3.3GB memory footprint. That's a lot of capability in a small package.

Start with ollama pull gemma3:4b (or the newer ollama pull gemma4:e4b) and run it for a week as your daily driver. If you hit quality ceilings on complex reasoning tasks, step up to a 12B model. If you need peak performance for production workloads, Gemma 3 27B -- or Gemma 4's 26B MoE / 31B dense -- stays competitive with models twice its parameter count. For brand-new setups, Gemma 4 is the better starting point: it adds audio/video input and the more permissive Apache 2.0 license.

The model weights and technical documentation are available on Google's Gemma page and the Google organization on HuggingFace.

---

Looking for a model comparison that covers the full local AI landscape? Our best local AI models for 8GB RAM guide ranks every major family by real-world usability on consumer hardware.