Mixture of Experts (MoE) Explained: DeepSeek V3 Active vs Total Parameters

Short answer: DeepSeek V3 is a Mixture-of-Experts model with 671B total parameters and 37B active per token (DeepSeek-V3 technical report, December 2024). Each MoE layer holds 1 shared expert and 256 routed experts; every token goes to the shared expert plus the 8 routed experts its router picks. You pay memory for all 671B and compute for 37B. That split, total versus active, is the whole idea of MoE, and it decides what you can run at home.

What is Mixture of Experts?

Mixture of Experts (MoE) is an architecture that grows a model's capacity without growing the compute per token in proportion, by using sparse activation: only some parts of the model process each token.

The Core Idea

Input Token → Router → Select Top-K Experts → Process → Combine Outputs
                ↓
       [Expert 1] [Expert 2] [Expert 3] ... [Expert N]
          ✓          ✓          ✗              ✗
       (active)   (active)  (inactive)    (inactive)

Instead of one large feed-forward block per layer, MoE uses:

• Multiple expert networks (8 in Mixtral, 128 in Qwen3 MoE and gpt-oss-120b, 256 routed in DeepSeek V3)
• A router that scores the experts for each token
• Sparse activation: only the top-K experts run (K = 2 for Mixtral, 4 for gpt-oss, 8 for Qwen3 MoE and DeepSeek V3)

The attention layers are shared by all tokens; the experts replace the feed-forward (MLP) block. That is why "active parameters" is larger than "K experts x expert size": attention, embeddings and any shared experts are always active.

Dense vs MoE: The Key Difference

Dense Model (Traditional)

• All parameters active for every token
• 70B model = 70B parameters of compute per token
• Quality scales with size, but so does cost

MoE Model (Sparse)

• Only some experts active per token
• DeepSeek V3: 671B total, 37B active = roughly 37B parameters of compute per token
• Capacity of the large model, per-token cost close to the active size

DeepSeek V3: Active vs Total Parameters

DeepSeek V3 is the model most people mean when they search for "MoE" today, so here are its numbers from primary sources:

DeepSeek R1 uses the same V3 base architecture, so the same 671B/37B split applies to it.

DeepSeekMoE: fine-grained and shared experts

The design comes from the DeepSeekMoE paper (arXiv 2401.06066, January 2024), which names two strategies:

1. Fine-grained expert segmentation: split experts into many smaller ones and activate more of them, so combinations of experts can specialise more precisely.
2. Shared expert isolation: reserve a subset of experts that are always activated for every token, to capture common knowledge so the routed experts do not each have to relearn it.

DeepSeek V3's "256 routed + 1 shared, 8 active" layout is those two ideas at scale.

Real-World Example: Mixtral 8x7B

Result: Mixtral delivered big-model quality at a per-token cost close to a 13B model, but you still need room for all 26GB of weights.

Mixtral 8x22B

Mistral's larger MoE (announced April 17, 2024) uses 39B active parameters out of 141B total, has a 64K-token context window, is natively capable of function calling, and is fluent in English, French, Italian, German and Spanish, all under Apache 2.0 (Mistral announcement). In Ollama it is mixtral:8x22b, an 80GB download, so it is a 96GB+ unified-memory or multi-GPU model, not a single-card one.

ollama pull mixtral:8x7b    # 26GB
ollama pull mixtral:8x22b   # 80GB

Both are 2023-2024 models; on the same memory budget, Qwen3 30B-A3B and gpt-oss are the more current MoE picks (see the table below).

Major MoE Models in 2026

Parameter counts come from each model's announcement or Hugging Face config; download sizes are what Ollama lists for the tag (checked November 2, 2026). Plan for a few GB more than the download for context (KV cache).

Sources: OpenAI gpt-oss model cards (117B/5.1B and 21B/3.6B); Qwen3-30B-A3B model card (30.5B total, 3.3B activated, 128 experts, 8 activated); Meta's Llama 4 model card (Scout 109B/17B with 16 experts, Maverick 400B/17B); Mistral announcements for Mixtral; DeepSeek-V3 technical report.

To size any of these against your own GPU and RAM, use the MoE VRAM calculator.

How Expert Routing Works

The Router Network

# Simplified router logic
def route(token_embedding, router_weights, k=2):
    # Router scores each expert
    scores = softmax(token_embedding @ router_weights)

    # Select top-K experts
    top_k_indices = top_k(scores, k=k)
    top_k_weights = scores[top_k_indices]

    # Normalize weights
    top_k_weights = top_k_weights / sum(top_k_weights)

    return top_k_indices, top_k_weights

Combining Expert Outputs

def forward(token, experts, router, shared_experts=()):
    indices, weights = router(token)

    output = 0
    for idx, weight in zip(indices, weights):
        expert_output = experts[idx](token)
        output += weight * expert_output

    # DeepSeekMoE-style models also add always-on shared experts
    for shared in shared_experts:
        output += shared(token)

    return output

Advantages of MoE

1. Better Quality per FLOP

• DeepSeek V3: 671B of capacity with 37B of compute per token
• More total parameters to store knowledge without paying for them on every token
• Routing lets combinations of experts handle different kinds of tokens

2. Faster Inference

• Token generation on local hardware is limited by memory bandwidth: every token has to read the weights that are used
• An MoE model only reads its active weights per token, so a 3.3B-active model behaves like a small model at generation time even though it stores 30B
• Mistral's own figure for Mixtral 8x7B was 6x faster inference than Llama 2 70B (Mistral, December 2023)

3. Scaling Efficiency

• Add experts without a proportional compute increase
• The route labs have used to reach hundreds of billions of parameters
• Better hardware utilization in datacenters with expert parallelism

Disadvantages of MoE

1. Higher Total VRAM

• All experts must fit in memory (or be split between VRAM and system RAM)
• A 671B model needs storage for 671B parameters
• Quantization helps, but the total stays far larger than a dense model with the same active size

2. Load Balancing Challenges

• Some experts may be overused, others underused
• Usually needs auxiliary losses to balance (DeepSeek V3 uses an auxiliary-loss-free method instead)
• Poor balance reduces effective capacity

3. Training Complexity

• More hyperparameters to tune
• Expert collapse possible
• Harder to debug and optimize

VRAM Considerations for Local Use

What Fits in VRAM

Download sizes are Ollama's defaults (mostly 4-bit). "Fits" means the whole model sits in GPU or unified memory with a few GB left for context.

Llama 4 Scout is not a 12GB model. Meta's model card says Scout "can fit within a single H100 GPU with on-the-fly int4 quantization" (that is an 80GB card), and Ollama's llama4:scout download is 67GB. Plan on 64GB+ of fast memory.

DeepSeek V3 does not run on a 24GB GPU on its own. Ollama's deepseek-v3:671b is 404GB. Unsloth's dynamic GGUFs shrink it as far as anyone has: 131GB for the 1.58-bit DeepSeek R1 quant and 186GB for the 1.78-bit DeepSeek-V3-0324 quant, with Unsloth suggesting at least 180GB of combined VRAM + RAM for its recommended 2.42/2.71-bit versions (Unsloth model cards on Hugging Face). A 24GB GPU helps by holding some layers, but the bulk lives in system RAM. See our DeepSeek V3 vs V3.1 vs R1 comparison for which of the three to run.

Why MoE Models Need Full VRAM

Even though only a few experts are active, all expert weights must be loaded because:

• Different tokens route to different experts
• You can't predict which experts the next token will need
• Reading experts from disk per token would be far too slow

The workaround is to put experts in system RAM instead of VRAM. llama.cpp has --cpu-moe (keep all MoE expert weights on the CPU) and --n-cpu-moe N (keep the experts of the first N layers on the CPU), plus the general --override-tensor flag. Attention and shared weights stay on the GPU, which is what makes large MoE models usable on a 24GB card plus lots of RAM.

Speed: bandwidth ceilings, not benchmarks

Generation speed has a simple upper bound: memory bandwidth divided by the bytes read per token. For a dense model that is the whole file; for an MoE model it is roughly the active share. On an RTX 4090 (1,008 GB/s, NVIDIA spec) a 19GB dense model would have a ~53 tok/s ceiling, while Qwen3 30B-A3B reads only its ~3.3B active parameters per token, which is why it feels like a small model. We do not publish tok/s figures for MoE models: how fast they run depends on how the runtime splits experts between GPU and RAM, so measure on your own machine.

MoE in Practice

Running Mixtral Locally

# 26GB download; needs a 32GB GPU or partial offload
ollama pull mixtral:8x7b
ollama run mixtral:8x7b

Running a Small MoE Model

# Fits a 16GB card
ollama run gpt-oss:20b

# Fits a 24GB card
ollama run qwen3:30b-a3b

Running DeepSeek V3

# 404GB download: a server or 512GB unified-memory machine, not a desktop GPU
ollama run deepseek-v3:671b

Checking Expert Activation

Most runtimes do not expose per-expert usage. If you load a model with Hugging Face Transformers, many MoE implementations can return the router logits (output_router_logits=True in the model config), which you can tally to see which experts fire.

The Future of MoE

MoE is the dominant architecture for large models because:

1. Frontier labs use it: DeepSeek, Meta (Llama 4), Qwen and OpenAI (gpt-oss) all ship open MoE models; GPT-4 is widely rumored to be one
2. Efficiency scales: Capacity can grow without the same growth in compute per token
3. Runtimes caught up: llama.cpp, Ollama, MLX and vLLM all run MoE models, and llama.cpp can split experts between VRAM and RAM
4. Research advances: Fine-grained and shared experts, better routing and load balancing

Key Takeaways

1. MoE = many experts, sparse activation
2. Active params << Total params (DeepSeek V3: 37B active of 671B)
3. Per-token cost of the active size, knowledge capacity of the total size
4. Memory is paid for the total; compute and bandwidth only for the active part
5. Dominant architecture for large open models in 2025-2026

Next Steps

1. Run DeepSeek R1 (MoE at 671B, or its dense distills) locally
2. Try Llama 4 (MoE architecture)
3. Size an MoE model for your hardware
4. Compare free local models, MoE and dense
5. Choose hardware for MoE models

---

MoE is the most significant architectural shift in open LLMs since the transformer itself. Knowing the difference between total and active parameters tells you both what a model needs in memory and how fast it can run.