Cohere Command R+ Local Setup Guide (2026): RAG and Tool-Use Specialist

Cohere's Command R+ is the open-weights model purpose-tuned for RAG and multi-step tool use. 104B parameters, native citation generation, structured tool-call output, and coverage of ten major languages. For research and personal RAG workloads where citations matter, it is one of the few open-weights models trained for exactly that. The catch: CC-BY-NC license restricts commercial use without a Cohere license, and 104B parameters need serious hardware. Command R7B (December 2024 release) brings the same training to a 7B model that fits on consumer GPUs.

This guide covers the full Command R family, setup across runtimes, the native RAG citation format, structured tool calling, multilingual usage, and the licensing considerations for commercial deployment.

Table of Contents

1. What Command R+ Is
2. Command R+ vs Command R 35B vs Command R7B
3. License (CC-BY-NC) Implications
4. Hardware Requirements
5. Command R+ vs Llama 3.1 70B for RAG
6. Ollama Setup
7. llama.cpp Setup
8. vLLM Setup
9. Native RAG Prompt Format
10. Structured Tool Calling
11. Multilingual Usage
12. Published Benchmarks
13. Tuning Recipes
14. Commercial Alternatives
15. Troubleshooting
16. FAQ

---

What Command R+ Is

Command R+ (CohereForAI/c4ai-command-r-plus on HuggingFace) is a 104B-parameter open-weights model from Cohere. Original release April 2024; revised "08-2024" version August 2024. License: CC-BY-NC-4.0.

Architecture: standard decoder-only with grouped-query attention. Context: 128K. Trained specifically for retrieval-augmented generation and multi-step tool use, not pure chat.

---

Command R+ vs Command R 35B vs Command R7B

For most local users: Command R7B is the right starting point. Command R 35B for 24GB cards. Command R+ for multi-GPU / H100 rigs.

---

License (CC-BY-NC) Implications

CC-BY-NC-4.0 = non-commercial only without separate license.

You can:

• ✅ Use in research / academic work
• ✅ Use in personal projects
• ✅ Use in internal company tools that don't directly generate revenue
• ✅ Modify and redistribute (with attribution and same license)

You cannot (without Cohere commercial license):

• ❌ Sell as a paid service / API
• ❌ Embed in commercial products
• ❌ Use to train competing models

For commercial RAG with permissive licensing, look at IBM's Granite 3 or Mistral Small 3, both released under Apache 2.0.

---

Hardware Requirements

Command R+ (104B)

• BF16: 210 GB → 4x A100 80GB or 4x H100
• AWQ-INT4: 60 GB → single H100 80GB or 2x RTX 4090 + offload
• Q4_K_M GGUF: 63 GB → 2x 48GB cards or 1x H100
• Q4_0: 59 GB (Ollama's default command-r-plus download) → 2x 48GB cards or 1x H100
• Q2_K GGUF: 38 GB → tight on Pro W7900 48GB

Command R 35B

• BF16: 70 GB → A100 80GB
• Q4_K_M: 21 GB → fits 24 GB card
• Q4_0: 19 GB (Ollama's default command-r:35b) → fits 24 GB card
• Q5_K_M: 25 GB → tight on 24 GB

Command R7B

• BF16: 14 GB → fits 16 GB card
• Q5_K_M: 5 GB → fits any modern GPU
• Q4_K_M: 5.1 GB (Ollama's command-r7b) → any 8 GB GPU

---

Command R+ vs Llama 3.1 70B for RAG

Speeds are bandwidth ceilings (memory bandwidth ÷ Ollama file size), not measurements. For RAG with built-in citations, Command R+ is the specialist. For general capability, chat and reasoning, Llama 3.1 70B is stronger. For commercial production, a permissively licensed model such as Granite 3 or Mistral Small 3 (both Apache 2.0) avoids the CC-BY-NC restriction.

---

Ollama Setup

# Command R7B (most practical)
ollama run command-r7b

# Command R 35B (24GB+ GPU)
ollama run command-r:35b

# Command R+ 104B (multi-GPU)
ollama run command-r-plus

For RAG-specific Modelfile:

FROM command-r7b
PARAMETER num_ctx 32768
PARAMETER temperature 0.3
PARAMETER min_p 0.05
SYSTEM """You are a helpful research assistant. Always cite sources using [1] [2] [3] format."""

---

llama.cpp Setup

huggingface-cli download bartowski/c4ai-command-r-plus-GGUF \
    c4ai-command-r-plus-Q4_K_M.gguf \
    --local-dir ./models

./llama-cli -m models/c4ai-command-r-plus-Q4_K_M.gguf \
    -ngl 999 -c 32768 -fa --tensor-split 24,24

For two 24GB GPUs (e.g., 2x RTX 4090): tensor split.

---

vLLM Setup

# AWQ for single-H100
vllm serve casperhansen/command-r-plus-104b-awq \
    --quantization awq --tensor-parallel-size 2 \
    --max-model-len 32768

# Command R7B (single card)
vllm serve CohereForAI/c4ai-command-r7b-12-2024 \
    --max-model-len 32768

---

Native RAG Prompt Format

Command R+ uses a custom RAG template:

<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>{system}<|END_OF_TURN_TOKEN|>
<|START_OF_TURN_TOKEN|><|USER_TOKEN|>{question}<|END_OF_TURN_TOKEN|>
<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>
{retrieved documents in <results> tags}
{model produces grounded answer with [N] citations}
<|END_OF_TURN_TOKEN|>

For Hugging Face Transformers:

documents = [
    {"title": "Doc 1", "snippet": "Local AI is..."},
    {"title": "Doc 2", "snippet": "Self-hosted means..."}
]

input_ids = tokenizer.apply_grounded_generation_template(
    [{"role": "user", "content": "What is local AI?"}],
    documents=documents,
    citation_mode="accurate",
    tokenize=False,
)

The model emits citations as [1] [2] markers tied to documents[] indices. This is the killer feature.

---

Structured Tool Calling

Tool definitions in the system prompt; the model emits a structured "Plan" + tool calls in JSON:

tools = [
    {"name": "search_web", "description": "Search the web", "parameters": {...}},
    {"name": "calculator", "description": "Math calc", "parameters": {...}}
]

input_ids = tokenizer.apply_tool_use_template(
    [{"role": "user", "content": "What is 2 to the 32nd power?"}],
    tools=tools,
    tokenize=False,
)

Output:

Plan: I will use the calculator tool.
Action: ```json
[{"tool_name": "calculator", "parameters": {"expression": "2^32"}}]

For OpenAI-compatible API integration, vLLM wraps this into standard `tool_calls`.

---

## Multilingual Usage {#multilingual}

Cohere's strongest open-weights multilingual model. Solid: English, French, Spanish, Italian, German, Portuguese, Japanese, Korean, Chinese, Arabic.

Cross-lingual RAG (retrieve in English, answer in French) is well-supported via the native template — pass documents in any language, ask question in any supported language.

---

## Published Benchmarks {#benchmarks}

| Test | Command R+ | Command R7B | Llama 3.1 70B | Granite 3.2 8B |
|------|------------|-------------|----------------|------------------|
| RAG MAP | — | — | — | — |
| Tool use accuracy | — | — | — | — |
| MMLU | 75.7 (Open LLM Leaderboard v1, via Cohere's model card) | — | 86.0 (Meta, 0-shot CoT) | — |
| HumanEval | — | — | 80.5 (Meta, 0-shot) | — |
| Multilingual MAP | — | — | — | — |
| Ollama download (Q4) | 59 GB | 5.1 GB | 43 GB | 4.9 GB |

"—" means we could not find a figure from the model's own publisher measured the same way, so we leave the cell empty rather than guess. Compare these models on your own documents: retrieval quality depends far more on your chunking and embeddings than on a leaderboard. Command R family is the RAG specialist by design; for general chat and reasoning, others are stronger.

---

## Tuning Recipes {#tuning}

### Single 24GB GPU (RTX 4090)

Command R7B Q5_K_M, 32K context, FlashAttention enabled.

### Dual 24GB (2x RTX 4090)

Command R 35B Q5_K_M with tensor split.

### H100 80GB

Command R+ 104B AWQ-INT4 fits with 30K context.

### Mac M4 Max 128GB

Command R 35B Q5_K_M comfortably; Command R+ Q3_K_M tight.

---

## Commercial Alternatives {#alternatives}

For commercial production with similar RAG quality:

1. **IBM Granite 3.2 8B** + Granite Embedding — Apache 2.0, comparable RAG MAP
2. **Mistral Small 3** + reranker — Apache 2.0, good general quality
3. **Llama 3.1 70B** + RAG prompt engineering — Meta license
4. **Qwen 2.5 72B** + Tongyi license — strong, EU restrictions

If RAG citations are mandatory and license must be permissive: Granite 3.2 + custom citation prompting is the cleanest path.

---

## Troubleshooting {#troubleshooting}

| Symptom | Cause | Fix |
|---------|-------|-----|
| Wrong chat template | Missing Cohere format | bartowski quants include template |
| Citations missing | Wrong RAG template | Use `apply_grounded_generation_template` |
| OOM at 104B | Hardware insufficient | Use Command R7B or rent H100 |
| Multilingual quality drop | Lower-resource language | Some languages stronger than others |

---

## FAQ {#faq}

See answers to common Command R+ questions below.

---

**Sources:** [Cohere Command R+ on HF](https://huggingface.co/CohereForAI/c4ai-command-r-plus) | [Command R7B](https://huggingface.co/CohereForAI/c4ai-command-r7b-12-2024) | [Cohere blog](https://cohere.com/blog/command-r-plus-microsoft-azure) | [Llama 3.1 model card](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct) | [Ollama library](https://ollama.com/library/command-r-plus)

**Related guides:**

- [RAG Local Setup Guide](/blog/rag-local-setup-guide)
- [Ministral 8B](/models/ministral-8b)
- [Llama 4 Local Setup](/blog/llama-4-local-setup-guide)
- [Ollama ChromaDB RAG Pipeline](/blog/ollama-chromadb-rag-pipeline)
- [Vector Databases Comparison](/blog/vector-databases-comparison)