WizardMath 13B:
RLEIF-Trained Math Reasoning Model
WizardMath 13B is a LLaMA 2 13B fine-tune from the WizardLM team that pioneered Reinforcement Learning from Evol-Instruct Feedback (RLEIF) for mathematical reasoning. Real scores: GSM8K 63.9% and MATH 14.0% (arXiv:2308.09583). A historically important model now surpassed by newer alternatives like Qwen 2.5 Math and DeepSeek Math.
What Is WizardMath 13B?
A LLaMA 2 13B fine-tune that introduced RLEIF for math-specific LLM training
Model Overview
Why WizardMath Matters
WizardMath 13B was one of the first models to demonstrate that math-specific reinforcement learning could significantly boost a base model's mathematical reasoning ability. Starting from LLaMA 2 13B (which scored only ~28.7% on GSM8K), WizardMath pushed that to 63.9% -- a 35-point improvement through RLEIF alone.
The key innovation was combining Evol-Instruct (generating increasingly difficult math problems) with a process reward model that scored step-by-step reasoning, then using PPO to optimize the model. This approach influenced later math-focused models like MetaMath, Qwen 2.5 Math, and DeepSeek Math.
While newer models have significantly surpassed its benchmark scores, WizardMath 13B remains historically significant as a proof of concept for specialized math training techniques.
Real Benchmark Results (arXiv:2308.09583)
Actual scores from the WizardMath paper, compared against other locally-runnable math models
GSM8K Accuracy — Math-Focused Models
GSM8K Accuracy — Math-Focused Models
GSM8K (Grade School Math 8K) scores from published papers. WizardMath 13B: 63.9% (arXiv:2308.09583). Note: Qwen 2.5 Math 7B (85.4%) shows how far math models have advanced since August 2023.
Math Domain Capabilities
Performance Metrics
Performance estimates across mathematical domains. GSM8K and MATH scores are from the paper; other categories are estimated from benchmark analysis.
Model Comparison — Local Math Models
| Model | Size | RAM Required | Speed | Quality | Cost/Month |
|---|---|---|---|---|---|
| WizardMath 13B | ~8GB (Q4) | 10GB+ | ~25 tok/s | 64% | Free |
| MetaMath 13B | ~8GB (Q4) | 10GB+ | ~25 tok/s | 72% | Free |
| Llama 2 13B (base) | ~8GB (Q4) | 10GB+ | ~28 tok/s | 29% | Free |
| Mistral 7B | ~4.4GB (Q4) | 6GB+ | ~40 tok/s | 52% | Free |
| Qwen 2.5 Math 7B | ~4.7GB (Q4) | 6GB+ | ~38 tok/s | 85% | Free |
RLEIF: The Key Innovation
Reinforcement Learning from Evol-Instruct Feedback — the training method that made WizardMath possible
Evol-Instruct Generation
Automatically generates math problems of increasing difficulty through evolutionary prompting. Starts from seed problems and evolves them into harder variants, creating a diverse training set spanning arithmetic to algebra to word problems.
Solution Generation
The model generates candidate solutions for each evolved problem. Multiple solution attempts are sampled to create a pool of reasoning chains with varying quality levels for the reward model to evaluate.
Process Reward Scoring
A process reward model (PRM) evaluates each step of the reasoning chain, not just the final answer. This step-level feedback identifies where reasoning goes wrong and provides fine-grained training signal.
PPO Training
Proximal Policy Optimization uses the reward signal from the PRM to update model weights. The model learns to produce step-by-step reasoning chains that the reward model scores highly, reinforcing correct mathematical reasoning patterns.
Why RLEIF Was Groundbreaking
Before RLEIF
- - LLaMA 2 13B base: 28.7% GSM8K
- - Standard SFT fine-tuning showed limited math improvement
- - Most math training used static datasets without RL
- - Outcome-based reward models missed reasoning errors
After RLEIF
- - WizardMath 13B: 63.9% GSM8K (+35.2 points)
- - Process reward scoring caught step-level errors
- - Evol-Instruct created diverse difficulty levels
- - Influenced MetaMath, Qwen Math, DeepSeek Math designs
VRAM & Hardware Requirements
Real VRAM usage by quantization level for WizardMath 13B
VRAM by Quantization Level (GB)
Memory Usage Over Time
VRAM requirements for WizardMath 13B at different quantization levels. Q4_K_M (~8GB) is the most popular choice for consumer GPUs. CPU-only inference is possible but significantly slower.
System Requirements
| Quantization | VRAM | File Size | Quality Loss | Best For |
|---|---|---|---|---|
| Q4_K_M | ~8GB | ~7.4GB | Minimal | RTX 3070/4060, most consumer GPUs |
| Q5_K_M | ~10GB | ~9GB | Very low | RTX 3080/4070, good quality-speed balance |
| Q8_0 | ~14GB | ~13GB | Negligible | RTX 3090/4080/4090 |
| FP16 | ~26GB | ~26GB | None | RTX 4090 / A100 — full precision |
Ollama Setup Guide
Get WizardMath 13B running locally with Ollama in minutes
Install Ollama
Download and install Ollama (or visit ollama.com for macOS/Windows installers)
Pull WizardMath 13B
Downloads the WizardMath model (~7.4GB for Q4_K_M quantization)
Test with a math problem
Verify the model works with a simple algebra problem
Test GSM8K-style word problem
Test the type of problem WizardMath was specifically trained on
Example Session
Note: WizardMath performs best on grade-school and basic algebra problems (GSM8K-style). For competition-level math (MATH benchmark), it only scores 14.0%. For advanced math, consider Qwen 2.5 Math or DeepSeek Math instead.
Local Math Model Alternatives (2026)
Math-focused models you can run locally, ranked by GSM8K accuracy
| Model | GSM8K | MATH | Params | VRAM (Q4) | Ollama Available |
|---|---|---|---|---|---|
| Qwen 2.5 Math 7B | 85.4% | 52.7% | 7B | ~4.7GB | Yes |
| DeepSeek Math 7B | 82.8% | 44.4% | 7B | ~4.5GB | Yes |
| MetaMath 13B | 72.3% | 22.4% | 13B | ~8GB | Yes |
| WizardMath 13B | 63.9% | 14.0% | 13B | ~8GB | Yes |
| Mistral 7B (general) | 52.2% | 13.1% | 7B | ~4.4GB | Yes |
Recommendation (March 2026): If you need a local math model today, Qwen 2.5 Math 7B is the strongest choice — it scores 85.4% on GSM8K (vs WizardMath's 63.9%) while requiring less VRAM (~4.7GB vs ~8GB). WizardMath 13B is best studied for its training methodology (RLEIF) rather than deployed for production math tasks.
Honest 2026 Assessment
Where WizardMath 13B stands today — strengths, limitations, and historical significance
Historical Significance
- +Pioneered RLEIF: First major demonstration of RL + process reward models for math training
- ++35 point GSM8K improvement: Proved math-specific training could transform a base model
- +Influenced future models: RLEIF ideas adopted by MetaMath, Qwen Math, DeepSeek Math
- +Open weights: Available for study and local deployment under LLaMA 2 license
- +Step-by-step reasoning: Produces detailed solution chains for educational use
2026 Limitations
- -Surpassed by newer models: Qwen 2.5 Math 7B scores 85.4% GSM8K with less VRAM
- -Weak on competition math: Only 14.0% on MATH benchmark (competition-level problems)
- -Limited context: 4,096 tokens vs 32K+ in modern models
- -LLaMA 2 base: Older base model with known limitations in reasoning
- -General knowledge tradeoff: Math specialization reduced MMLU to ~52%
Who Should Use WizardMath 13B in 2026?
Researchers
Studying RLEIF methodology, process reward models, or math-specific LLM training techniques
Students
Learning about LLM fine-tuning approaches and comparing training methods across model families
Hobbyists
Experimenting with local math models on older hardware (already have 8GB+ VRAM GPUs)
For production math tasks in 2026: Use Qwen 2.5 Math 7B or DeepSeek Math 7B instead. They score 20+ points higher on GSM8K while requiring less VRAM.
Authoritative Sources & Research
Primary Sources
Ready to Go Beyond Tutorials?
25 structured courses with hands-on chapters - build RAG chatbots, AI agents, and ML pipelines on your own hardware.
Go from reading about AI to building with AI
25 structured courses. Hands-on projects. Runs on your machine. Start free.
Written by
Barsha Mohapatra
Founder and Editor, Local AI Master · Founder, Little AI Master
Founder of Little AI Master and the editor behind every course and guide on Local AI Master. Years of tutoring students taught her where AI education actually fails.
WizardMath 13B RLEIF Training Pipeline
The RLEIF training pipeline: Evol-Instruct generates math problems, the model generates solutions, a process reward model scores step-by-step reasoning, and PPO optimizes the model weights
Related Guides
Continue your local AI journey with these comprehensive guides
Continue Learning
Explore more math-focused local AI models and related resources:
- PILLARLocal AI Models Directory: Every Model Compared
- Alpaca 7B: Stanford\
- Amazon Chronos: Time Series Forecasting Models (Complete Guide)
- Anima 2.9B on 8GB: The Anime Model Taking Over Civitai
- Aquila 7B by BAAI: Chinese-English Bilingual (FlagAI)
- Baichuan2-13B: Chinese LLM | 59% CMMLU, Bilingual, Free License 2026
- Bark by Suno AI: Open-Source Text-to-Audio Generation Guide
- ChatGLM3-6B: Tsinghua Chinese AI | Code Interpreter, 6GB RAM 2026
- Claude 3 Opus Review: Benchmarks, Pricing & API Guide 2026
- Claude 3 Sonnet Review: Benchmarks, API Pricing & Alternatives 2026
Grab the AI Starter Kit — career roadmap, cheat sheet, setup guide
No spam. Unsubscribe with one click.
Found your model? Now build something with it.
25 hands-on courses — RAG, agents, fine-tuning — all running locally. First chapter free, no card.