Run Google Gemma Locally: Complete Ollama Setup Guide

Published on April 10, 2026 • 24 min read

Quick Start: Gemma Running in 60 Seconds

Pull and run Gemma with two commands:

1. Pull the model: ollama pull gemma3:4b (2-3 minutes on broadband)
2. Start chatting: ollama run gemma3:4b

You now have Google's latest open model running on your hardware. No API key, no usage limits.

---

What this guide covers:

• Every Gemma variant from 270M to 27B and which to pick
• Exact VRAM requirements for each model size and quantization
• Real performance numbers on consumer GPUs and Apple Silicon
• MLX optimization for M-series Macs
• Fine-tuning Gemma on your own data with Unsloth
• Head-to-head comparison with Phi-4 and Llama 3.2

Google's Gemma family has become one of the strongest options for local AI. The models punch well above their weight class -- Gemma 3 4B matches or beats many 7-8B models from other families on reasoning and instruction following. Google trains these on their TPU infrastructure with the same data pipeline used for Gemini, then releases them under a permissive license that allows commercial use.

If you're new to running models locally, start with our Mac local AI setup guide or check the RAM requirements guide to confirm your hardware can handle the model size you want.

Table of Contents

1. The Gemma Model Family
2. VRAM Requirements
3. Ollama Setup Step by Step
4. Performance Benchmarks
5. MLX on Apple Silicon
6. Quantization Options
7. Multimodal Capabilities
8. Fine-Tuning with Unsloth
9. Gemma vs Phi-4 vs Llama 3.2
10. Best Use Cases

---

The Gemma Model Family {#gemma-family}

Google has released three generations of Gemma. Here's the full lineup as of April 2026:

Gemma 3 (Latest)

Gemma 2

Gemma 1 and Specialized Variants

For most users, Gemma 3 4B is the sweet spot. It delivers strong reasoning, handles vision tasks, supports 128K context, and runs comfortably on 8GB hardware. If you have 16GB+, the 12B variant is a significant step up in quality.

---

VRAM Requirements {#vram-requirements}

These are measured VRAM numbers, not theoretical estimates. Tested with Ollama's default quantization (Q4_K_M for most sizes).

Gemma 3 VRAM Usage

What This Means for Your Hardware

The Q4_K_M quantization retains roughly 97% of full-precision quality for instruction following. You lose maybe 1-2% on complex reasoning benchmarks. For most practical tasks, you will not notice the difference.

---

Ollama Setup Step by Step {#ollama-setup}

Install Ollama (If Needed)

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Start the service
ollama serve

Pull Gemma Models

# Gemma 3 - recommended variants
ollama pull gemma3:1b        # 1B - ultra fast, basic tasks
ollama pull gemma3:4b        # 4B - best balance (default)
ollama pull gemma3:12b       # 12B - strong reasoning
ollama pull gemma3:27b       # 27B - maximum quality

# Specific quantization
ollama pull gemma3:4b-q8_0   # Higher quality 4B
ollama pull gemma3:12b-q4_K_M # Fits in 16GB

# Gemma 2 (still excellent)
ollama pull gemma2:2b
ollama pull gemma2:9b
ollama pull gemma2:27b

# Code-specific
ollama pull codegemma:7b

Verify Installation

# Check model is downloaded
ollama list

# Quick test
ollama run gemma3:4b "What is the capital of France? Answer in one sentence."

# Check model details
ollama show gemma3:4b

Run with Custom Parameters

# Create a Modelfile for custom settings
cat > Modelfile << 'EOF'
FROM gemma3:4b
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 8192
PARAMETER repeat_penalty 1.1
SYSTEM "You are a precise, helpful assistant. Give concise answers with specific details. When you're unsure, say so."
EOF

# Create custom model
ollama create my-gemma -f Modelfile

# Run it
ollama run my-gemma

---

Performance Benchmarks {#benchmarks}

Real-world measurements from our test hardware. All numbers are tokens per second for generation (not prompt processing).

GPU Benchmarks (Gemma 3)

*Partial GPU offload

Apple Silicon Benchmarks (Gemma 3)

*With limited context window

30+ tokens/second feels instant for interactive chat. Below 10 tok/s starts feeling sluggish. These numbers show Gemma 3 4B delivers a snappy experience on almost any modern hardware.

---

MLX on Apple Silicon {#mlx-apple-silicon}

If you have an M-series Mac, MLX can squeeze extra performance out of Gemma. MLX is Apple's machine learning framework designed specifically for Apple Silicon's unified memory architecture.

Install MLX

pip install mlx-lm

Download and Run Gemma with MLX

# Download quantized Gemma 3 for MLX
mlx_lm.generate \
  --model mlx-community/gemma-3-4b-it-4bit \
  --prompt "Explain quantum computing in simple terms" \
  --max-tokens 500

# Interactive chat
mlx_lm.chat --model mlx-community/gemma-3-4b-it-4bit

MLX vs Ollama Performance on Apple Silicon

MLX typically delivers 20-30% faster inference than Ollama on Apple Silicon. The advantage comes from tighter Metal integration and memory access patterns optimized for unified memory. The tradeoff: MLX lacks Ollama's API server, model management, and ecosystem of client apps. Use MLX when raw speed matters; use Ollama when you need an API or compatible tools like Open WebUI.

Converting Models for MLX

# Convert any HuggingFace model to MLX format
mlx_lm.convert \
  --hf-path google/gemma-3-4b-it \
  --mlx-path ./gemma-3-4b-mlx \
  --quantize --q-bits 4

---

Quantization Options {#quantization}

Quantization reduces model precision to save memory. Here's how different levels affect Gemma 3 4B:

Quantization Quality Comparison

Recommendation: Q4_K_M is the default for good reason. You lose about 1.3 points on MMLU compared to full precision -- barely noticeable in practice -- while cutting memory usage by 64%. Drop to Q3_K_M only if you absolutely need to fit in tight memory. Avoid Q2_K for anything beyond basic chat.

How to Choose

# Check available quantizations
ollama show gemma3:4b --modelfile

# Pull specific quantization
ollama pull gemma3:4b-q8_0     # Maximum quality
ollama pull gemma3:4b-q5_K_M   # Good balance
ollama pull gemma3:4b-q4_K_M   # Memory efficient (default)

For a deeper comparison of quantization formats, see our AWQ vs GPTQ vs GGUF comparison.

---

Multimodal Capabilities {#multimodal}

Gemma 3 4B, 12B, and 27B are multimodal -- they accept both text and images. This works out of the box in Ollama.

Image Analysis with Ollama

# Describe an image
ollama run gemma3:4b "What's in this image?" ./photo.jpg

# Extract text from a screenshot
ollama run gemma3:12b "Extract all text visible in this image" ./screenshot.png

# Analyze a chart
ollama run gemma3:4b "What trends does this chart show?" ./quarterly_revenue.png

Via the API

# Base64 encode an image and send to Ollama API
curl http://localhost:11434/api/generate -d '{
  "model": "gemma3:4b",
  "prompt": "Describe this image in detail",
  "images": ["'$(base64 -i photo.jpg)'"]
}'

Vision Performance

Gemma 3 4B handles basic image understanding -- object identification, text extraction, simple visual Q&A. For complex image reasoning (counting objects, spatial relationships, detailed chart analysis), the 12B or 27B variants perform noticeably better.

The 1B model is text-only. If you need vision on constrained hardware, the 4B is your only Gemma option under 8GB.

---

Fine-Tuning with Unsloth {#fine-tuning}

Gemma models respond extremely well to fine-tuning. With QLoRA, you can fine-tune Gemma 3 4B on a GPU with just 6GB VRAM.

Install Unsloth

pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
pip install --no-deps trl peft accelerate bitsandbytes

Fine-Tuning Script

from unsloth import FastLanguageModel

# Load Gemma with 4-bit quantization
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/gemma-3-4b-it-bnb-4bit",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)

# Add LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
)

# Prepare your dataset
from datasets import load_dataset
dataset = load_dataset("json", data_files="my_training_data.jsonl")

# Format: {"instruction": "...", "input": "...", "output": "..."}

from trl import SFTTrainer
from transformers import TrainingArguments

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset["train"],
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        max_steps=100,
        learning_rate=2e-4,
        fp16=True,
        logging_steps=1,
        output_dir="outputs",
    ),
)

trainer.train()

# Save the fine-tuned model
model.save_pretrained_merged("gemma-finetuned", tokenizer)

Export to Ollama

# Convert to GGUF
python llama.cpp/convert_hf_to_gguf.py gemma-finetuned \
  --outtype q4_K_M \
  --outfile gemma-finetuned.gguf

# Create Ollama model
cat > Modelfile << 'EOF'
FROM ./gemma-finetuned.gguf
TEMPLATE """<start_of_turn>user
{{ .Prompt }}<end_of_turn>
<start_of_turn>model
{{ .Response }}<end_of_turn>"""
PARAMETER stop "<end_of_turn>"
EOF

ollama create my-gemma-finetuned -f Modelfile
ollama run my-gemma-finetuned

For a comprehensive fine-tuning walkthrough beyond Gemma, see our LoRA fine-tuning local guide.

Unsloth claims 2x training speed over standard HuggingFace training. In our testing with Gemma 3 4B, we measured 1.7x speedup -- still significant. Full details on the Unsloth GitHub repository.

---

Gemma vs Phi-4 vs Llama 3.2 {#comparison}

The three strongest open model families for local use, compared head-to-head at similar sizes:

4B Class Models

12B Class Models

Key takeaways:

• Gemma 3 4B wins on general knowledge (MMLU, ARC) while being smaller than Phi-4 Mini
• Phi-4 wins on math and code (GSM8K, HumanEval) across both size classes
• Llama 3.2 is the most memory-efficient but trails on every benchmark
• Gemma 3 has the longest context (128K) at every size, which matters for document analysis
• Vision capability on the 4B model gives Gemma a unique advantage in its size class

For a broader comparison of small local models, check our small language models guide.

---

Best Use Cases {#use-cases}

Where Gemma Excels

Document analysis and summarization. The 128K context window combined with multimodal support means Gemma 3 can process long documents and images in a single pass. Feed it a 50-page PDF and ask for a structured summary.

Multilingual tasks. Google trained Gemma on data spanning 30+ languages. It handles translation, multilingual Q&A, and cross-lingual retrieval better than most open models its size.

Instruction following. Gemma's instruction-tuned variants follow complex, multi-step instructions with high reliability. This makes them excellent for structured output tasks like JSON generation, data extraction, and template filling.

Where Other Models Are Better

Pure coding tasks. If you write code all day, Phi-4 or Qwen2.5-Coder will serve you better. Gemma is competent at code but not a specialist.

Creative writing. Llama 3.2 and Mistral produce more varied, creative prose. Gemma tends toward factual, concise responses -- great for work, less great for fiction.

Constrained memory (<4GB). The Gemma 3 1B is decent but the Phi-4 Mini at 3.8B Q2_K provides meaningfully better quality in a similar memory footprint.

---

Troubleshooting

Model Won't Load

# Check available memory
nvidia-smi   # GPU
free -h       # System RAM

# Try smaller quantization
ollama pull gemma3:4b-q4_0

# Force CPU mode if GPU memory is full
CUDA_VISIBLE_DEVICES="" ollama run gemma3:4b

Slow Generation

# Reduce context window
ollama run gemma3:4b --num-ctx 4096

# Check if model is using GPU
ollama ps   # Shows GPU memory usage per model

# On Mac, verify Metal is active
system_profiler SPDisplaysDataType | grep Metal

Vision Not Working

# Only 4B, 12B, 27B support vision
# 1B is text-only

# Verify with API
curl http://localhost:11434/api/show -d '{"name": "gemma3:4b"}' | grep -i "vision"

---

Conclusion

Gemma 3 earns its spot as a top-tier local model family. The 4B variant delivers a rare combination: vision support, 128K context, strong benchmarks, and 3.3GB memory footprint. That's a lot of capability in a small package.

Start with ollama pull gemma3:4b and run it for a week as your daily driver. If you hit quality ceilings on complex reasoning tasks, step up to 12B. If you need peak performance for production workloads, the 27B is competitive with models twice its parameter count.

The model weights and technical documentation are available on Google's Gemma page and the Google organization on HuggingFace.

---

Looking for a model comparison that covers the full local AI landscape? Our best local AI models for 8GB RAM guide ranks every major family by real-world usability on consumer hardware.