Microsoft's Mage-VL Runs Video Understanding on 8GB VRAM — Here's What It Can and Can't Watch

What it actually is

Two components in one checkpoint: Mage-ViT, a vision encoder trained entirely from scratch, and a Qwen3-4B-Instruct-2507 decoder joined to it by a two-layer MLP projector. The Qwen3 backbone is the only pretrained piece. Microsoft did not initialise the vision tower from a web-scale image-text model — the card reports it was trained on roughly 100M unlabeled images and video frames and still matches encoders trained on billions of pairs.

The interesting part is how it consumes video. A conventional video LLM decodes the file, samples N frames at even intervals, and pushes a dense patch grid through the encoder for each one. Mage-VL instead reads the structure the encoder already produced: anchor (I) frames keep all their patches, predicted (P) frames keep only the patches where the codec spent bits. If nothing moved, nothing gets tokenised. A shared 3D rotary position encoding keeps the spatio-temporal layout intact across the sparse set.

On top of that sits a two-speed design the card calls System 1 / System 2: a lightweight cognition gate watches each rolling codec window and predicts whether anything worth talking about just happened. Only when it opens does the full decoder generate. That gate is trained separately in the final stage with the encoder and language model frozen, and ships as streammind_gate.safetensors — a 1.07 GB sidecar.

Fit by VRAM tier

The lever that decides your VRAM is not the model — it is the context length and the frame budget. A 4.74B model is small. What blows up memory is the number of codec canvases you feed it. The GGUF maintainer measured the same clip at 7.56 GiB with a 16,384-token context and about 10.4 GiB at 32,768 — a 2.8 GiB swing from one setting. Read the table with that in mind.

The GGUF and MLX numbers are the respective maintainers' own measurements — the GGUF figures on a Radeon AI PRO R9700 under Vulkan, the MLX figures on an M4 and an M5 Max. None of them are ours: we do not own that Radeon card, and we have not re-run these clips, so treat every figure as the maintainer's claim rather than an independent result. If you are picking Apple hardware for this kind of work, MLX vs CUDA for local AI covers why the unified-memory path behaves so differently from a discrete card.

Setup: the three real paths

There is no one-line install. Pick the path that matches your hardware, and accept that two of the three involve some assembly.

1. Transformers — the official path

This is the reference implementation, and the only one that gives you every capability. It needs Transformers 5.7 or newer, and ffmpeg/ffprobe on your PATH for anything codec-related.

pip install "transformers>=5.7" accelerate pillow torch torchvision \
  opencv-python codec-video-prep

# image
python inference.py --mode offline --image examples/dog.jpg \
  --question "Describe this image in detail."

# video via the traditional codec path (H.264/HEVC)
python inference.py --mode offline --video clip.mp4 \
  --video-backend codec --codec-engine traditional --num-frames 32 \
  --question "Describe this video."

inference.py ships in the model repository. Swap --codec-engine traditional for neural to use the bundled DCVC-RT engine, or --video-backend frames to fall back to plain frame sampling and skip the codec machinery entirely.

2. GGUF — the 8GB path, and the fiddly one

One community repository, JohnTdi/Mage-VL-GGUF, published 11 August 2026. It does not work with the llama.cpp you already have. The project ships pinned Vulkan and CUDA Docker images that clone llama.cpp, apply a runtime patch implementing Mage's temporal patching, 2x2 spatial merger and 3D RoPE, and start llama-server with both GGUF files. Requirements are Linux, Docker Engine, and Docker Compose 2.30 or newer — there is no native macOS or Windows route here.

git clone https://github.com/JohnTDI-cpu/mage-vl-gguf.git
cd mage-vl-gguf
./install.sh   # detects Vulkan/CUDA, picks an 8/16/24-32 GB profile

# then, once it is up:
curl --fail http://localhost:8080/v1/video/analyze \
  -F video=@./clip.mp4 \
  -F 'prompt=Describe the important events in temporal order.' \
  -F max_tokens=256

You can override detection with MAGE_PROFILE=8|16|24|32. The 16 GB profile's exact defaults are documented (16,384 context, 256 sampled frames, 150,000 pixels per canvas, F16 KV cache); the 8 GB profile's are not spelled out, so if you are on an 8GB card, expect to tune MAGE_SAMPLED_FRAMES and MAGE_MAX_PIXELS down yourself.

Quantization cost, per the maintainer's own evaluation: WikiText-2 perplexity moves from 7.6592 at BF16 to 7.6710 at Q8_0 (+0.15%) and 7.8739 at Q4_K_M (+2.80%). On Video-MME the BF16 reference is 64.00, Q8 + vision Q8 scored 63.33, and Q4 + vision Q8 scored 61.96. Q8 is close to free; Q4 costs you about two points of video accuracy.

3. MLX — the cleanest path, if you are on a Mac

The mage_vl architecture was merged into mlx-vlm on 29 July 2026, so no fork or branch install is needed and no PyTorch is required at all.

pip install "mlx-vlm>=0.6.9"

python - <<'PY'
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("mlx-community/Mage-VL-8bit")
prompt = apply_chat_template(processor, model.config,
                             "Describe this image in detail.", num_images=1)
print(generate(model, processor, prompt, image=["dog.jpg"], max_tokens=64).text)
PY

The catch: the mlx-community conversions deliberately exclude the DCVC neural-codec runtime and the StreamMind gate weights. So on the MLX path you get image and frame-sampled video, but not the codec-native token reduction that is the model's whole point, and not the proactive streaming gate. A separate community port (sr29/Mage-VL-mlx-4bit) does implement the gate, pulling the weights from the upstream repo, but still notes the codec backend needs the external codec engine.

Benchmarks vs Qwen3-VL-4B

This is the cleanest comparison in open vision-language models right now: same 4B Qwen3 backbone, only the vision encoder differs. All figures below are from Microsoft's model card — vendor-reported, not our measurements.

The pattern is consistent and useful: Mage-VL wins anything that depends on when and wherethings happened, and loses on reading small text off a still frame. If your job is "find the moment the delivery van pulled up", this is the better model. If it is "transcribe this scanned invoice", it is not — go to a dedicated OCR model instead.

On OVO-Bench, which tests online video understanding, the card puts Mage-VL at 64.00 overall against Qwen3-VL-4B's 63.00 and a human reference of 92.77 — with Mage-VL taking a large lead on real-time perception (79.84 vs 72.8) while losing on backward tracing (48.15 vs 53.1). For still images specifically, our best local vision-language models comparison is the better guide — and MiniCPM-V remains the lighter option if you only need to look at pictures.

Honest limitations

The model is ready; the tooling is weeks old. That gap is the entire risk of adopting this today.

• !The ecosystem is one repo deep. One community GGUF conversion, published 11 August 2026 — seventeen days after the model — and sitting in the low hundreds of downloads when we checked. No Ollama entry, no LM Studio or Jan listing. If that maintainer stops, the 8GB path stops with them.
• !The GGUF route is Linux + Docker only. It rebuilds a patched llama.cpp from pinned source. That is a real install, not a download.
• !The MLX route drops the headline feature. The mlx-community conversions ship without the neural codec runtime and without the streaming gate. You get a competent 4B video model on a Mac; you do not get codec-native sparsity.
• !Codec preprocessing is the bottleneck, not inference. Decode ran at 80.66 tok/s in the maintainer's test, but the DCVC-RT plus canvas-construction stage only managed 1.55 fps at 1080p. Video prep, not generation, is what you wait on.
• !OCR on video frames is weak. The GGUF project found a single general prompt can miss a readable on-screen caption even at maximum visual detail, and built a separate OCR-focused inference pass to work around it. Treat any text-on-screen requirement as needing a second model.
• !Audio is ignored. The GGUF gateway states audio tracks are discarded. If your question is "what did they say", you need a Whisper pipeline alongside it.

Security cameras and screen recordings

This is the use case that makes a 4.74B video model worth the setup pain — and the honest answer is one camera at reduced resolution, not a whole property. The streaming gate exists precisely so you are not running a 4B model against every frame of an empty driveway. It scores each rolling window, stays quiet on routine content, and only wakes the decoder when something completes.

The GGUF runtime wires this to RTSP/RTMP, HTTP/HLS and local sources, encoding each transport segment once and sharing the embedding between the gate and the triggered response. The maintainer reports throughput on a Radeon AI PRO R9700 of 14.30 fps at 854x480, 6.34 fps at 1280x720, and 1.55 fps at 1920x1080 — which is why the shipped 16 GiB profile accepts a 1080p or 4K source but downsizes it to at most 480 pixels high and analyses at 8 fps. That is a reasonable trade for "did anyone approach the door". It is not enough to read a licence plate.

If what you actually want is object-and-person detection across several cameras with sensible notifications, Frigate with an Ollama model is still the more practical architecture — cheap detection first, a language model only on the clips that matter. Mage-VL is the better fit for the second half of that pipeline: give it the 30-second clip Frigate already flagged and ask what happened in it. Our broader local AI video analysis guide covers the tooling around that pattern, and local AI vision tasks maps which job needs which class of model.

Frequently asked questions