IBM Granite 3 Local Setup Guide (2026): Enterprise-Grade Open Models

IBM Granite 3 is the enterprise-grade open-weights model family for organizations that care about Apache 2.0 licensing, audited training data, and built-in safety. Sizes from 1B (mobile) to 8B (production) plus MoE variants for sparse-activation efficiency. Strong on code (Granite Code 3-34B), structured output, and instruction following. Pairs with Granite Guardian for input/output safety classification. For regulated industries — finance, healthcare, government — Granite is the cleanest open-source choice in 2026.

This guide covers the Granite 3 family, setup across runtimes, the Granite Guardian safety stack, Granite Code for coding assistants, fine-tuning workflows, watsonx integration, and benchmarks vs Llama 3.1 / Mistral / Phi-4.

Table of Contents

1. What Granite 3 Is
2. The Granite Family
3. Apache 2.0 + Audited Training Data
4. Hardware Requirements
5. Granite vs Llama / Mistral / Phi-4
6. Ollama Setup
7. llama.cpp / vLLM Setup
8. Chat Template
9. Granite Code Variants
10. Granite Guardian Safety Models
11. Function Calling and JSON Mode
12. Multilingual Support
13. Fine-Tuning
14. watsonx Integration
15. Real Benchmarks
16. Production Deployment
17. Troubleshooting
18. FAQ

---

What Granite 3 Is {#what-it-is}

IBM Granite 3 (ibm-granite/* on HuggingFace) is IBM's flagship open-weights LLM family. Released October 2024 (3.0), updated through 2025-2026 (3.1, 3.2, 3.3). Apache 2.0 license. 12-language coverage. 128K context (3.2+).

Key thesis: enterprise-grade quality with auditable data sources, strong on structured/agentic workloads, and a complete supporting stack (Guardian, Code, Embeddings).

---

The Granite Family {#family}

For most production deployments: Granite 3.2 8B Instruct + Granite Guardian 2B for input/output filter + Granite Embedding 278M for RAG.

---

Apache 2.0 + Audited Training Data {#license}

Apache 2.0 license — same commercial-friendly terms as Mistral Small 3. Plus IBM publishes Data Cards documenting:

• All training data sources
• Filtering and deduplication processes
• Bias detection and mitigation
• IP/copyright due diligence

For regulated industries this provenance matters — IBM provides legal cover that Llama / Qwen / DeepSeek do not.

---

Hardware Requirements {#requirements}

Granite 8B is faster than Mistral Small 3 24B and Phi-4 14B because it's smaller — useful for high-throughput agent/RAG workloads.

---

Granite vs Llama / Mistral / Phi-4 {#comparison}

Granite leads on RAG grounding and structured tasks. For pure reasoning, Phi-4. For pure chat, Mistral / Llama.

---

Ollama Setup {#ollama}

ollama run granite3.2
ollama run granite3.2:2b   # smaller variant
ollama run granite3.2:8b
ollama run granite-code:8b  # coding variant

---

llama.cpp / vLLM Setup {#runtimes}

llama.cpp

huggingface-cli download bartowski/granite-3.2-8b-instruct-GGUF \
    granite-3.2-8b-instruct-Q5_K_M.gguf --local-dir ./models

./llama-cli -m models/granite-3.2-8b-instruct-Q5_K_M.gguf -ngl 999 -c 32768 -fa

vLLM

vllm serve ibm-granite/granite-3.2-8b-instruct \
    --max-model-len 32768 \
    --enable-prefix-caching

---

Chat Template {#chat-template}

Granite uses its own format:

<|start_of_role|>system<|end_of_role|>{system}<|end_of_text|>
<|start_of_role|>user<|end_of_role|>{user}<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>

All major runtimes auto-load it from tokenizer_config.json.

Recommended sampling: temperature 0.5, min-p 0.05, top-p 0.95.

---

Granite Code Variants {#granite-code}

For coding assistants, use Granite Code 3:

ollama run granite-code:8b      # general coding
ollama run granite-code:20b     # higher quality
ollama run granite-code:34b     # max quality

Pair with Tabby for editor integration. HumanEval scores: 8B ~78%, 20B ~85%, 34B ~88%.

---

Granite Guardian Safety Models {#guardian}

from transformers import AutoModelForSequenceClassification, AutoTokenizer

guardian = AutoModelForSequenceClassification.from_pretrained(
    "ibm-granite/granite-guardian-3.2-2b"
).cuda()
tok = AutoTokenizer.from_pretrained("ibm-granite/granite-guardian-3.2-2b")

def classify(text, risk_dimension="harm"):
    inputs = tok(text, return_tensors="pt").to("cuda")
    logits = guardian(**inputs).logits
    return logits.argmax(-1).item()  # 0=safe, 1=unsafe

Risk dimensions: harm, social_bias, profanity, sexual_content, unethical_behavior, jailbreak, function_call_safety, groundedness (RAG), answer_relevance.

For RAG groundedness specifically (does the answer match the retrieved chunks?), Granite Guardian is currently the best open-source classifier. See prompt injection defense.

---

Function Calling and JSON Mode {#tools}

Granite was trained for structured agentic output. OpenAI-compatible:

{
  "model": "granite3.2",
  "messages": [...],
  "tools": [{"type": "function", "function": {...}}],
  "tool_choice": "auto"
}

JSON mode via response_format works in vLLM with xgrammar. Granite 3.2 is one of the more reliable open-source models for structured output — it rarely produces malformed JSON when given a schema.

---

Multilingual Support {#multilingual}

Strong: English, German, Spanish, French, Italian, Portuguese, Japanese, Chinese, Arabic, Czech, Dutch, Korean.

For mainland Chinese: Qwen 2.5 / 3 still wins. For most European business languages: Granite is a solid balanced choice.

---

Fine-Tuning {#fine-tuning}

QLoRA via Unsloth:

from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
    "unsloth/granite-3.2-8b-instruct-bnb-4bit",
    max_seq_length=4096,
    load_in_4bit=True,
)

Apache 2.0 license means derivatives can be redistributed under any license. For watsonx-style enterprise fine-tuning UI, use IBM's InstructLab framework.

---

watsonx Integration {#watsonx}

IBM's watsonx.ai is the paid managed platform. Same Granite weights you self-host run on watsonx with:

• SOC2 / HIPAA compliance
• Audit logging and lineage tracking
• watsonx.governance for AI policy
• RBAC, key management, networking
• Integration with IBM Cloud + on-premises

For "develop locally with Granite, deploy in regulated production via watsonx," the model weights are identical and migration is seamless.

---

Real Benchmarks {#benchmarks}

RTX 4090, Granite 3.2 8B Q5_K_M, single user:

---

Production Deployment {#production}

# vLLM with AWQ for max throughput
vllm serve ibm-granite/granite-3.2-8b-instruct-AWQ \
    --quantization awq \
    --max-model-len 32768 \
    --enable-prefix-caching --enable-chunked-prefill \
    --max-num-seqs 128 \
    --kv-cache-dtype fp8_e4m3

Pair with Granite Guardian behind a LiteLLM gateway for compliance-grade safety filtering.

---

Troubleshooting {#troubleshooting}

---

FAQ {#faq}

See answers to common Granite 3 questions below.

---

Sources: IBM Granite on Hugging Face | Granite 3 announcement | InstructLab | bartowski quants | Internal benchmarks RTX 4090.

Related guides:

• Mistral Small 3 Setup
• Phi-4 Local Setup
• Best Local AI Models 2025
• Prompt Injection Defense
• SOC2 Self-Hosted AI
• Tabby Self-Hosted Copilot Setup