Nemotron 3.5 Lightning: A 30B Agent Model That Runs at Reading Speed on 32GB of RAM

Every build and its real size

These are file sizes read off the Hugging Face repository listings, not estimates. Two repositories matter: ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF, which NVIDIA links from its own card as the official GGUF, and unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF, which ships a far wider quant ladder.

Two things worth pausing on. First: unsloth's IQ1_M, IQ2_XXS and IQ2_M all weigh exactly 19.4 GB on the repository listing, so dropping to a more aggressive quant buys you nothing below that line — the floor for this architecture is around 19 GB and going lower just costs quality. Second: the file sizes are much higher than a naive “30B x 4 bits” calculation because the Mamba-2 and attention layers are not quantised as hard as the expert weights.

Per-tier fit table

The number that matters is total addressable memory, not GPU class. A 12 GB card with 32 GB of system RAM runs this better than a 16 GB card with 16 GB of RAM, because the model spills to RAM either way and only 3B parameters are read per token. Pick your row by the memory you can actually give the model.

If your box is on the low side of this table, the general CPU-inference rules apply — running LLMs on CPU only and how much RAM local AI actually needs cover the tradeoffs in more depth.

Setup: Ollama and llama.cpp

Ollama is one command and works out of the box with tool calling and thinking enabled. NVIDIA lists Ollama 0.32.9 as the validated version on its NVFP4 card.

Ollama

ollama run nemotron-3.5-lightning

# Explicit tags (all verified on the Ollama library listing):
#   nemotron-3.5-lightning:30b-a3b-q4_K_M   25 GB
#   nemotron-3.5-lightning:30b-a3b-q8_0     35 GB
#   nemotron-3.5-lightning:30b-a3b-bf16     66 GB
#   nemotron-3.5-lightning:30b-a3b-mlx      23 GB  (Apple Silicon)
#   nemotron-3.5-lightning:30b-a3b-mxfp8    34 GB  (Apple Silicon)

One behaviour to know before you blame the model: per NVIDIA's card, Ollama picks context from available VRAM — under 24 GB it uses 4K, 24-48 GB gets 32K, and 48 GB+ gets 256K. If your agent keeps forgetting things on a 16 GB card, that 4K default is why. Raise it with /set parameter num_ctx 32768, accepting that it will push more of the model onto CPU.

llama.cpp

# Simplest start (pulls the official GGUF):
llama-server -hf ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF --port 8000

# NVIDIA's fuller recipe, with the recommended sampling:
llama-server \
  -hf ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF:Q4_0 \
  --temp 1.0 --top-p 0.95 \
  -np 1 -c 40960 --port 8000 \
  -ngl 99 -fa on --jinja --no-webui --fit off

A small trap: NVIDIA's published recipe writes that last flag as :Q4_K_M, but the official ggml-org repository currently only contains Q4_0, Q8_0 and BF16 — there is no Q4_K_M file there to fetch. Use :Q4_0 as above, or point -hf at the unsloth repo if you want the Q4_K_M ladder.

Sampling matters more than usual here: NVIDIA recommends temperature 1.0 and top_p 0.95, which feels high if you are used to 0.6-0.7 defaults. Reasoning is on by default and can be switched off per-request with the enable_thinking chat-template argument — useful when you want a direct answer from a tool-calling loop rather than a page of deliberation.

The official GGUF repo also ships separate mtp- files (1.16 GB at Q4_0, 2.18 GB at Q8_0). Those are the Multi-Token Prediction draft weights for speculative decoding, not alternative model builds — don't download them expecting a small Nemotron.

How fast, honestly

We have not benchmarked this model on our own hardware yet, so we are not going to publish a tokens-per-second table we cannot stand behind. What we can give you is the arithmetic, which you can check yourself.

Generation speed on a memory-bound box is roughly memory bandwidth divided by bytes read per token. With 3B of 30B parameters active, a token touches on the order of a tenth of the file — call it ~2.5 GB for the 25 GB Q4_K_M build, before router and shared-layer overhead. A dual-channel DDR5-5600 desktop moves roughly 90 GB/s in theory and materially less in practice. Divide those and you land in the tens of tokens per second as a ceiling, and comfortably above reading pace (a fast reader is around 7-10 tokens per second) once you halve it for real-world efficiency. That is the whole reason this model is interesting on a CPU box, and it is why a dense 30B — which would read the entire 25 GB every token — is not.

Two caveats that arithmetic hides. Prompt processing is compute-bound, not bandwidth-bound, so a long context takes a while to ingest on CPU even though generation is quick afterwards. And the hybrid Mamba-2 layers carry recurrent state alongside the usual KV cache, so memory use grows differently with context than you may be used to; NVIDIA's own datacentre recipes set the Mamba SSM cache to FP16 specifically to keep that under control at high concurrency.

The OpenMDW-1.1 licence, in plain terms

OpenMDW-1.1 is about as permissive as model licences get — commercial use is fine, and it explicitly places no restrictions on what you do with the outputs. It is an unfamiliar name because it is new, not because it is restrictive. Reading the actual text, it is closest in spirit to MIT with a patent-retaliation clause bolted on.

• ✓Permission to “deal in the Model Materials without restriction”, free of charge, under copyright, patent, database and trade-secret rights. No revenue cap, no user threshold, no acceptable-use appendix.
• ✓Outputs are yours. The licence states it “does not impose any restrictions or obligations with respect to any use, modification, or sharing of any outputs”. That is unusually clear compared with most open-weight licences.
• !If you redistribute the weights — or any portion, including your own fine-tune — you must ship a copy of the agreement and keep the origin notices.
• !Patent retaliation, and it is broader than usual. The clause is not limited to suing NVIDIA: file or join a lawsuit against any person or entity claiming the Model Materials infringe a patent or copyright and all your rights terminate — unless that suit was a response to one brought against you first.
• !You carry the rights-clearance risk. The agreement pushes responsibility for clearing third-party rights in the model onto you, and disclaims warranties broadly. Normal for open weights, but read it if you are shipping to enterprise customers.

NVIDIA's card also states plainly: “This model is ready for commercial use.” If licence terms are the thing gating your deployment, this one is not the blocker — and compared with the research-only terms attached to a lot of speech and vision models, it is a genuine advantage.

Benchmarks vs its class

All of these are NVIDIA's own numbers, run on NVIDIA's harness (NeMo Gym / NeMo Evaluator), and NVIDIA notes they may differ from what other vendors self-report. Taken at face value they tell a consistent story: Lightning is not the strongest model in its size class, and NVIDIA is not pretending otherwise — the comparison column it loses to most often is Qwen 3.6 35B-A3B.

The shape here is worth reading properly. On raw knowledge, reasoning and software engineering, Lightning trails Qwen 3.6 by a wide margin. Where it holds up is the agent-shaped column: 85.37 on PinchBench for tool use, 36.97 on BrowseComp — beating both Gemma 4 26B-A4B (26.30) and Nemotron's own bigger Super sibling (22.77) — and it out-scores Qwen on instruction following. If your workload is “call the right tool, follow the format, don't wander off”, that profile is the one you want. If your workload is “fix this bug across four files”, look at Qwen3.6-27B or GPT-OSS instead.

Honest limitations

• ✗Nothing under ~19 GB. There is no 8 GB build and there is not going to be a useful one — the floor is set by total parameters. If you have a 3060 and 16 GB of RAM, this model is not for you.
• ✗1M context is marketing for local users. NVIDIA validates it on GB200, B200 or 8x H100 only. A single H100 is listed at 256K and called memory-bound. Locally, plan on 32-64K.
• ✗Six supported languages, not fifty. The card lists English, Spanish, French, German, Italian and Japanese. It saw more in pre-training, but those six are what NVIDIA stands behind.
• ✗The Mamba hybrid is still new tooling. Runtime support is good — llama.cpp, Ollama, LM Studio, vLLM, SGLang all list it — but it is weeks old. Expect rough edges in anything more exotic, and pin your llama.cpp version.
• ✗Reasoning is on by default, which means an agent loop that expects short answers will get long ones until you disable thinking. Budget tokens accordingly.
• !The BF16 repo is not the one you want. NVIDIA is explicit that BF16 exists for post-training and for building your own quants; the NVFP4 release is the deployment path, and GGUF is the local path.

Frequently asked questions