Phi-3 Small 7B: sizes, requirements and why it is not on Ollama

Phi-3 Small hardware requirements

A 24 GB NVIDIA card for the original weights, or an 8 GB card for the int4 ONNX build. The arithmetic is parameters × bits per weight: 7.39 billion × 16 bits ÷ 8 is 14.8 GB, which is the size of the safetensors files on Hugging Face. The 5.0 GB int4 build works out to 5.4 bits per weight.

There is a catch with the original weights. The model card says Phi-3 Small uses flash attention 2 and Triton blocksparse attention by default, “which requires certain types of GPU hardware to run”, and the only GPU types Microsoft lists are the NVIDIA A100, A6000 and H100. Microsoft's sample code asserts that CUDA is available. For anything else, including CPUs and Windows machines with AMD or Intel graphics, the card points to the ONNX builds.

Sizes are the folder totals reported by the Hugging Face API for microsoft/Phi-3-small-8k-instruct-onnx-cuda. The “what fits” column is arithmetic, not a measurement. To size other models, use the VRAM calculator.

Phi-3 model sizes on Ollama, and RAM for phi3:mini

phi3:mini is 2.2 GB and needs about 5 GB of free memory at a 4K context, so it runs on a machine with 8 GB of RAM or a 6 GB graphics card. phi3:medium is 7.9 GB. Ollama's library describes Phi-3 as Mini and Medium only; Small is absent.

The 5 GB figure for phi3:mini is the 2.2 GB file, plus 1.6 GB of KV cache for 4,096 tokens, plus a 1 GB allowance for buffers. Phi-3 Mini does not use grouped-query attention, so its cache is large for a model this size: 32 layers × 3072 × 2 × 2 bytes is 393 KB per token. That matters because Ollama lists phi3:mini with a 128K context. Filling all of it would take 51.5 GB of cache, so set the context length to what you need.

All sizes are from Ollama's phi3 tag list. The individual pages for Phi-3 Mini 3.8B and Phi-3 Medium 14B go further, and the Ollama system requirements guide covers memory by model size.

Specifications

From the model card and the config.json. The family is described in the Phi-3 Technical Report (arXiv:2404.14219, April 2024), which covers the 3.8B, 7B and 14B models together.

Two of these set Small apart from its siblings. It uses the tiktoken tokenizer with a 100,352-token vocabulary, where Mini and Medium use a 32,064-token vocabulary. And it alternates dense and blocksparse attention layers, where the other two are standard transformers. Its config declares its own model type, phi3small. That is the reason the usual GGUF route did not pick it up: a llama.cpp request for conversion support, issue 8241, was closed as not planned in July 2024.

How to run Phi-3 Small

With Transformers on a supported NVIDIA GPU. The model card asks for tiktoken and triton and for trust_remote_code=True, because the model code ships in the repository. This is Microsoft's sample, shortened:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model_id = "microsoft/Phi-3-small-8k-instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    trust_remote_code=True,
)
assert torch.cuda.is_available(), "This model needs a GPU to run ..."
device = torch.cuda.current_device()
model = model.to(device)
tokenizer = AutoTokenizer.from_pretrained(model_id)

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, device=device)
messages = [{"role": "user", "content": "What about solving an 2x + 3 = 7 equation?"}]
output = pipe(messages, max_new_tokens=500, return_full_text=False, do_sample=False)
print(output[0]["generated_text"])

On other hardware, use the ONNX builds with ONNX Runtime GenAI. Microsoft publishes int4 builds for DirectML, CUDA, and CPU and mobile, plus an fp16 CUDA build. If what you want is a small Microsoft model behind ollama run, that is not this model; pull one of the models in the table below.

Benchmarks Microsoft reported

Copied from the Phi-3-small-8k-instruct model card. Microsoft ran every column through the same internal pipeline and notes that the numbers “might differ from other published numbers due to slightly different choices in the evaluation”. This site has not run them. The comparison models are the ones Microsoft chose in May 2024.

The TriviaQA row is the honest weak spot: 58.1, below every other model in the table. Microsoft's own category summary puts factual knowledge at 38.6. Phi-3 Small reasons well for its size and knows less than larger models, which is what you would expect from a model trained on a filtered, reasoning-heavy dataset.

What replaced Phi-3 Small

Microsoft did not release a 7B model in the Phi-3.5 or Phi-4 generations. The line continued at 3.8B and 14B, and both of those run on Ollama. Sizes are from the Ollama tag lists linked under Sources.

For 7B to 8B models from other vendors that do run on Ollama, see Llama 3.1 8B, Qwen 2.5 7B, Mistral 7B and Gemma 7B, or go by card with the best LLM for 8 GB of VRAM.

Frequently asked questions

Sources

• Phi-3-small-8k-instruct model card — Microsoft: specifications, hardware notes, benchmark table, licence
• Phi-3 Technical Report — arXiv:2404.14219, 22 April 2024
• Phi-3 Small ONNX builds — int4 and fp16 file sizes
• Phi-3 Cookbook — Microsoft's worked examples
• Ollama phi3 tags, phi4 tags and phi4-mini tags — every size quoted for Mini, Medium and Phi-4
• Phi-4 model card — release date, context length, licence