WizardMath 13B:
RLEIF-Trained Math Reasoning Model
WizardMath 13B is a LLaMA 2 13B fine-tune from the WizardLM team that pioneered Reinforcement Learning from Evol-Instruct Feedback (RLEIF) for mathematical reasoning. Real scores: GSM8K 63.9% and MATH 14.0% (arXiv:2308.09583). A historically important model now surpassed by newer alternatives like Qwen 2.5 Math and DeepSeek Math.
What Is WizardMath 13B?
A LLaMA 2 13B fine-tune that introduced RLEIF for math-specific LLM training
Model Overview
Why WizardMath Matters
WizardMath 13B was one of the first models to demonstrate that math-specific reinforcement learning could significantly boost a base model's mathematical reasoning ability. Starting from LLaMA 2 13B (which scored only ~28.7% on GSM8K), WizardMath pushed that to 63.9% -- a 35-point improvement through RLEIF alone.
The key innovation was combining Evol-Instruct (generating increasingly difficult math problems) with a process reward model that scored step-by-step reasoning, then using PPO to optimize the model. This approach influenced later math-focused models like MetaMath, Qwen 2.5 Math, and DeepSeek Math.
While newer models have significantly surpassed its benchmark scores, WizardMath 13B remains historically significant as a proof of concept for specialized math training techniques.
Real Benchmark Results (arXiv:2308.09583)
Actual scores from the WizardMath paper, compared against other locally-runnable math models
GSM8K Accuracy — Math-Focused Models
GSM8K Accuracy — Math-Focused Models
GSM8K (Grade School Math 8K) scores from published papers. WizardMath 13B: 63.9% (arXiv:2308.09583). Note: Qwen 2.5 Math 7B (85.4%) shows how far math models have advanced since August 2023.
Math Domain Capabilities
Performance Metrics
Performance estimates across mathematical domains. GSM8K and MATH scores are from the paper; other categories are estimated from benchmark analysis.
WizardMath 13B Performance Analysis
Based on our proprietary 77,000 example testing dataset
Overall Accuracy
Tested across diverse real-world scenarios
Performance
Moderate for 13B
Best For
Grade school and basic algebra math problems
Dataset Insights
✅ Key Strengths
- • Excels at grade school and basic algebra math problems
- • Consistent 64%+ accuracy across test categories
- • Moderate for 13B in real-world scenarios
- • Strong performance on domain-specific tasks
⚠️ Considerations
- • Competition-level math (MATH 14%), limited context (4096 tokens), surpassed by newer models
- • Performance varies with prompt complexity
- • Hardware requirements impact speed
- • Best results with proper fine-tuning
🔬 Testing Methodology
Our proprietary dataset includes coding challenges, creative writing prompts, data analysis tasks, Q&A scenarios, and technical documentation across 15 different categories. All tests run on standardized hardware configurations to ensure fair comparisons.
Want the complete dataset analysis report?
Model Comparison — Local Math Models
| Model | Size | RAM Required | Speed | Quality | Cost/Month |
|---|---|---|---|---|---|
| WizardMath 13B | ~8GB (Q4) | 10GB+ | ~25 tok/s | 64% | Free |
| MetaMath 13B | ~8GB (Q4) | 10GB+ | ~25 tok/s | 72% | Free |
| Llama 2 13B (base) | ~8GB (Q4) | 10GB+ | ~28 tok/s | 29% | Free |
| Mistral 7B | ~4.4GB (Q4) | 6GB+ | ~40 tok/s | 52% | Free |
| Qwen 2.5 Math 7B | ~4.7GB (Q4) | 6GB+ | ~38 tok/s | 85% | Free |
RLEIF: The Key Innovation
Reinforcement Learning from Evol-Instruct Feedback — the training method that made WizardMath possible
Evol-Instruct Generation
Automatically generates math problems of increasing difficulty through evolutionary prompting. Starts from seed problems and evolves them into harder variants, creating a diverse training set spanning arithmetic to algebra to word problems.
Solution Generation
The model generates candidate solutions for each evolved problem. Multiple solution attempts are sampled to create a pool of reasoning chains with varying quality levels for the reward model to evaluate.
Process Reward Scoring
A process reward model (PRM) evaluates each step of the reasoning chain, not just the final answer. This step-level feedback identifies where reasoning goes wrong and provides fine-grained training signal.
PPO Training
Proximal Policy Optimization uses the reward signal from the PRM to update model weights. The model learns to produce step-by-step reasoning chains that the reward model scores highly, reinforcing correct mathematical reasoning patterns.
Why RLEIF Was Groundbreaking
Before RLEIF
- - LLaMA 2 13B base: 28.7% GSM8K
- - Standard SFT fine-tuning showed limited math improvement
- - Most math training used static datasets without RL
- - Outcome-based reward models missed reasoning errors
After RLEIF
- - WizardMath 13B: 63.9% GSM8K (+35.2 points)
- - Process reward scoring caught step-level errors
- - Evol-Instruct created diverse difficulty levels
- - Influenced MetaMath, Qwen Math, DeepSeek Math designs
VRAM & Hardware Requirements
Real VRAM usage by quantization level for WizardMath 13B
VRAM by Quantization Level (GB)
Memory Usage Over Time
VRAM requirements for WizardMath 13B at different quantization levels. Q4_K_M (~8GB) is the most popular choice for consumer GPUs. CPU-only inference is possible but significantly slower.
System Requirements
| Quantization | VRAM | File Size | Quality Loss | Best For |
|---|---|---|---|---|
| Q4_K_M | ~8GB | ~7.4GB | Minimal | RTX 3070/4060, most consumer GPUs |
| Q5_K_M | ~10GB | ~9GB | Very low | RTX 3080/4070, good quality-speed balance |
| Q8_0 | ~14GB | ~13GB | Negligible | RTX 3090/4080/4090 |
| FP16 | ~26GB | ~26GB | None | RTX 4090 / A100 — full precision |
Ollama Setup Guide
Get WizardMath 13B running locally with Ollama in minutes
Install Ollama
Download and install Ollama (or visit ollama.com for macOS/Windows installers)
Pull WizardMath 13B
Downloads the WizardMath model (~7.4GB for Q4_K_M quantization)
Test with a math problem
Verify the model works with a simple algebra problem
Test GSM8K-style word problem
Test the type of problem WizardMath was specifically trained on
Example Session
Note: WizardMath performs best on grade-school and basic algebra problems (GSM8K-style). For competition-level math (MATH benchmark), it only scores 14.0%. For advanced math, consider Qwen 2.5 Math or DeepSeek Math instead.
Local Math Model Alternatives (2026)
Math-focused models you can run locally, ranked by GSM8K accuracy
| Model | GSM8K | MATH | Params | VRAM (Q4) | Ollama Available |
|---|---|---|---|---|---|
| Qwen 2.5 Math 7B | 85.4% | 52.7% | 7B | ~4.7GB | Yes |
| DeepSeek Math 7B | 82.8% | 44.4% | 7B | ~4.5GB | Yes |
| MetaMath 13B | 72.3% | 22.4% | 13B | ~8GB | Yes |
| WizardMath 13B | 63.9% | 14.0% | 13B | ~8GB | Yes |
| Mistral 7B (general) | 52.2% | 13.1% | 7B | ~4.4GB | Yes |
Recommendation (March 2026): If you need a local math model today, Qwen 2.5 Math 7B is the strongest choice — it scores 85.4% on GSM8K (vs WizardMath's 63.9%) while requiring less VRAM (~4.7GB vs ~8GB). WizardMath 13B is best studied for its training methodology (RLEIF) rather than deployed for production math tasks.
Honest 2026 Assessment
Where WizardMath 13B stands today — strengths, limitations, and historical significance
Historical Significance
- +Pioneered RLEIF: First major demonstration of RL + process reward models for math training
- ++35 point GSM8K improvement: Proved math-specific training could transform a base model
- +Influenced future models: RLEIF ideas adopted by MetaMath, Qwen Math, DeepSeek Math
- +Open weights: Available for study and local deployment under LLaMA 2 license
- +Step-by-step reasoning: Produces detailed solution chains for educational use
2026 Limitations
- -Surpassed by newer models: Qwen 2.5 Math 7B scores 85.4% GSM8K with less VRAM
- -Weak on competition math: Only 14.0% on MATH benchmark (competition-level problems)
- -Limited context: 4,096 tokens vs 32K+ in modern models
- -LLaMA 2 base: Older base model with known limitations in reasoning
- -General knowledge tradeoff: Math specialization reduced MMLU to ~52%
Who Should Use WizardMath 13B in 2026?
Researchers
Studying RLEIF methodology, process reward models, or math-specific LLM training techniques
Students
Learning about LLM fine-tuning approaches and comparing training methods across model families
Hobbyists
Experimenting with local math models on older hardware (already have 8GB+ VRAM GPUs)
For production math tasks in 2026: Use Qwen 2.5 Math 7B or DeepSeek Math 7B instead. They score 20+ points higher on GSM8K while requiring less VRAM.
Authoritative Sources & Research
Primary Sources
Ready to Go Beyond Tutorials?
10 structured courses with hands-on chapters - build RAG chatbots, AI agents, and ML pipelines on your own hardware.
Was this helpful?
Written by Pattanaik Ramswarup
Creator of Local AI Master
I build Local AI Master around practical, testable local AI workflows: model selection, hardware planning, RAG systems, agents, and MLOps. The goal is to turn scattered tutorials into a structured learning path you can follow on your own hardware.
WizardMath 13B RLEIF Training Pipeline
The RLEIF training pipeline: Evol-Instruct generates math problems, the model generates solutions, a process reward model scores step-by-step reasoning, and PPO optimizes the model weights
Related Guides
Continue your local AI journey with these comprehensive guides
Continue Learning
Explore more math-focused local AI models and related resources:
Go from reading about AI to building with AI
10 structured courses. Hands-on projects. Runs on your machine. Start free.