Mistral 7B: Open Source Language Model

Mistral 7B system requirements

The default Mistral 7B download is a 4.4 GB Q4_K_M file that needs about 5.9 GB of memory with a 4K context: an 8 GB graphics card runs it comfortably, a 6 GB card only just, and a CPU-only machine needs 8 GB of RAM. Using the full 32K context adds about 3.8 GB, which moves the same file onto a 12 GB card. Q8_0 (7.7 GB) wants a 12 GB card and the unquantised fp16 file (14 GB) wants a 24 GB card.

Sizes are the download sizes on Ollama's mistral tag list. The memory columns are arithmetic, not measurements: file size, plus the key-value cache, plus a 1 GB allowance for compute buffers. With 32 layers and 8 key-value heads of dimension 128, the cache costs 2 × 32 × 1,024 × 2 bytes = 131 KB per token, which is 0.5 GB at 4,096 tokens and 4.3 GB at 32,768. Grouped-query attention is why that figure is a quarter of what a 32-head cache would cost.

By graphics card

• 8 GB card: Q4_K_M, Q5_K_M or Q6_K at a 4K context. This is the natural home for Mistral 7B.
• 12 GB card: Q8_0 at 4K, or Q4_K_M through Q6_K with the full 32K context.
• 16 GB card: Q8_0 with the full 32K context (about 13 GB needed).
• 24 GB card: fp16 fits, but a 24 GB card holds much larger models; see the successors section below.
• No GPU: 8 GB of system RAM covers q4_0 and Q4_K_M at 4K; 16 GB covers everything up to Q8_0.

Ollama's context-length documentation gives a 4K default below 24 GiB of VRAM, so the “Memory at 4K” column is what a default session uses. The Ollama system requirements guide covers the runtime itself, the VRAM calculator does this sum for other models, and quantization explained covers what each quantisation level gives up.

Mistral 7B release date, versions and licence

Mistral 7B was released on 27 September 2023. It is open-weight under the Apache 2.0 licence, which permits commercial use, modification and redistribution. Mistral AI's announcement post puts it this way: “We're releasing Mistral 7B under the Apache 2.0 license, it can be used without restrictions.” The paper followed on arXiv on 10 October 2023.

Is Mistral 7B open source? The weights and the reference inference code are Apache 2.0, which is an OSI-approved licence. The training data and training code were not published, so “open-weight” is the more precise description. Each version exists as a base (text completion) model and an Instruct model; the Instruct vs base section below explains which one you are getting. Note that the base model card states it “does not have any moderation mechanisms”.

Mistral 7B Instruct vs the base model

If you want to chat with Mistral 7B, you want the Instruct model, and that is what ollama run mistral gives you. The base model only continues text; it is the starting point for your own fine-tune or for plain completion tasks.

• What Instruct is: the v0.1 Instruct model card describes it as an instruct fine-tune of the base model “using a variety of publicly available conversation datasets”. The architecture and the memory it needs are the same as the base model, so the hardware table above applies to both.
• Prompt format: Instruct expects each user turn wrapped in [INST] and [/INST] tokens. Ollama applies this template for you; you only need it when you call the raw model through Transformers or llama.cpp.
• Moderation: neither variant has a built-in safety layer. Both model cards say the model “does not have any moderation mechanisms”, so add your own filtering if the output reaches end users.
• Reported chat score: the paper gives Mistral 7B Instruct an MT-Bench score of 6.84. The benchmark table further down is for the base model.

Mistral 7B architecture: exact specifications

Mistral 7B is a decoder-only transformer. The table is Table 1 of the Mistral 7B paper (Jiang et al., 2023), which describes v0.1; the last column notes where later versions differ, from their config.json files and model cards. The published weights hold 7,241,732,096 parameters, which Mistral AI rounds to 7.3B in the announcement.

Grouped-query attention (GQA)

The model has 32 query heads but only 8 key-value heads, so every four query heads share one set of keys and values. The announcement lists this as the change made “for faster inference”. For anyone running the model locally the practical effect is the smaller cache worked out in the hardware section above.

Sliding-window attention (SWA)

In v0.1, each layer attends only to the previous 4,096 hidden states rather than to the whole sequence. Because layers are stacked, information still travels further back than one window: a token at layer k attends to tokens within the window at layer k−1, which themselves attended to earlier tokens. The announcement gives the motivation as a compute cost that is linear in sequence length, and reports “a 2x speed improvement for sequence length of 16k with a window of 4k” from the matching changes to FlashAttention and xFormers.

Sliding-window attention is not in the version most people run today. The v0.2 model card lists “No Sliding-Window Attention” among its changes, and the v0.2 and v0.3 configuration files set sliding_window to null with a 32,768-token position limit. If you pull mistral from Ollama you get v0.3, which uses ordinary full attention over a 32K context.

Reported benchmarks

These are Mistral AI's own figures from Table 2 of the paper, for the v0.1 base model. Mistral AI re-evaluated every model in the table with its own pipeline; this site has not run them. The paper also reports an MT-Bench score of 6.84 for Mistral 7B Instruct against 6.65 for Llama 2 13B Chat.

The headline claim in 2023 was that a 7B model beat Llama 2 13B, a model nearly twice its size, on every benchmark in the table. That comparison is against 2023 models. Microsoft's later evaluation on the Phi-3 Mini model card scores Mistral 7B v0.1 at 61.7 on MMLU against 70.9 for the 3.8B Phi-3 Mini, so smaller models have since passed it on reasoning benchmarks.

Run Mistral 7B locally

Ollama is the shortest route. Every tag below was checked against the tag list on 29 September 2026. mistral, mistral:latest, mistral:7b and mistral:v0.3 are the same 4.4 GB build.

# Install Ollama (Linux; macOS and Windows use the installer from ollama.com)
curl -fsSL https://ollama.com/install.sh | sh

# Mistral 7B Instruct v0.3, Q4_K_M, 4.4 GB
ollama pull mistral
ollama run mistral

# Higher precision for a 12 GB card, 7.7 GB
ollama pull mistral:7b-instruct-v0.3-q8_0

# The base text-completion model rather than the chat model, 4.1 GB
ollama pull mistral:text

# The original September 2023 release, 4.1 GB
ollama pull mistral:v0.1

Raising the context above 4K

Ollama documents one environment variable for this. Check the memory column in the hardware table first, because the cache grows with the context you allocate.

# Serve with a 32K context (the maximum for v0.2 and v0.3)
OLLAMA_CONTEXT_LENGTH=32768 ollama serve

# Confirm the allocated context and the GPU/CPU split
ollama ps

v0.3 supports function calling; the model card documents it for mistral_inference and for Transformers 4.42 or newer, and Ollama's library page shows the raw-mode prompt format. The Ollama function calling guide covers tool use across models.

What replaced Mistral 7B

Mistral AI has released several small open-weight models since. Licences differ between them, which matters if Apache 2.0 was the reason you chose Mistral 7B.

On a 12 GB card, Mistral Nemo is the direct upgrade: same licence, four times the context, and a 7.1 GB download. On a 24 GB card, Mistral Small 3.2 at 15 GB is the current Apache-licensed choice. Mistral 7B still makes sense on a 6 GB or 8 GB card, on CPU-only machines, and as a base for the many community fine-tunes built on it, such as OpenHermes 2.5 (below). For how it sits against Meta's models of the same period, see Llama vs Mistral vs CodeLlama.

OpenHermes 2.5: the best-known Mistral 7B fine-tune

OpenHermes 2.5 is a community fine-tune of Mistral 7B by Teknium, released under Apache 2.0. Its model card says it was trained on 1,000,000 entries of mostly GPT-4-generated data plus filtered open datasets, with extra code data; the card reports HumanEval at 50.7% pass@1 (testing by the Glaive team, a different harness from the paper's). It uses the ChatML prompt format (<|im_start|> / <|im_end|>) rather than [INST], which Ollama handles for you. Memory needs are the same as Mistral 7B at the same quantisation.

# OpenHermes 2.5 Mistral 7B, q4_0, 4.1 GB
ollama run openhermes

# The same model at Q4_K_M, 4.4 GB
ollama run openhermes:7b-mistral-v2.5-q4_K_M

It dates from late 2023, so the newer Mistral releases in the table above are the better choice for most new projects; OpenHermes is still worth a try if you prefer ChatML or its writing style.

Frequently asked questions

Sources