MLC-LLM Setup Guide (2026): Cross-Platform LLM Inference on Any Device

MLC-LLM is the universal compiler for LLM inference. Built on Apache TVM, it produces device-specific binary libraries that run on every major backend — CUDA, ROCm, Metal, Vulkan, OpenCL, WebGPU, and CPU. It is the only mainstream path to run LLMs natively in the browser, on Android, on iOS, and on Intel Arc, all from one source.

This guide covers everything: how the compilation pipeline works, installing MLCEngine, compiling models for each backend, the WebLLM browser runtime, the iOS and Android apps, MLCEngine OpenAI-compatible serving, quantization choices, and benchmarks across NVIDIA / AMD / Apple / mobile.

Table of Contents

1. What MLC-LLM Is and Why TVM Matters
2. Hardware Coverage
3. Installation: pip, Docker, Source
4. The Pipeline: Convert → Compile → Run
5. Your First Model: Llama 3.2 3B on CUDA
6. Quantization Formats (q4f16_1, q4f32_1, q3f16_1)
7. MLCEngine OpenAI-Compatible Server
8. Apple Silicon (Metal)
9. AMD GPUs (ROCm + Vulkan)
10. Intel Arc (Vulkan)
11. Android Deployment
12. iOS Deployment
13. WebLLM: Running in the Browser
14. Long Context, FlashAttention, KV Cache
15. Tool Calling and Structured Output
16. Benchmarks Across Platforms
17. Tuning Recipes
18. Common Errors
19. FAQ

---

What MLC-LLM Is and Why TVM Matters

MLC-LLM stands for "Machine Learning Compilation for Large Language Models." It is a project from CMU and the OctoML / Apache TVM community.

The compiler approach:

Hugging Face model
       │
       ▼
[mlc_llm convert_weight]   # weight conversion
       │
       ▼
MLC checkpoint format
       │
       ▼
[mlc_llm compile]          # TVM compiles to device-specific code
       │
       ▼
.so / .dylib / .dll / .wasm / .so (Android) / .a (iOS)
       │
       ▼
[mlc_llm serve / chat / library APIs]

TVM autotunes kernels for the target device. The output is a binary that performs comparably to hand-written CUDA on NVIDIA, hand-written Metal on Apple, hand-written Vulkan on Intel — without you needing to write any of those.

The trade-off, similar to TensorRT-LLM: compilation takes time (10-30 min per target). The win: one Python source produces optimized code for every backend.

---

Hardware Coverage

The diversity is unique. No other major LLM inference engine covers WebGPU + iOS + Android + Intel Arc + AMD ROCm + NVIDIA from one codebase.

---

Installation: pip, Docker, Source

pip (recommended)

python3.11 -m venv ~/venvs/mlc
source ~/venvs/mlc/bin/activate

# CUDA 12.4
pip install --pre -U -f https://mlc.ai/wheels mlc-llm-cu124 mlc-ai-cu124

# Metal (Mac)
pip install --pre -U -f https://mlc.ai/wheels mlc-llm mlc-ai

# ROCm 6.2
pip install --pre -U -f https://mlc.ai/wheels mlc-llm-rocm62 mlc-ai-rocm62

# Vulkan (Intel Arc, multi-vendor)
pip install --pre -U -f https://mlc.ai/wheels mlc-llm-vulkan mlc-ai-vulkan

# Verify
python -c "import mlc_llm; print(mlc_llm.__version__)"

Docker

docker run --gpus all -it --rm \
    -p 8000:8000 \
    -v $(pwd)/models:/models \
    mlcaidev/mlc-llm:latest \
    mlc_llm serve /models/Llama-3.2-3B-q4f16_1

From source

git clone --recursive https://github.com/mlc-ai/mlc-llm.git
cd mlc-llm
mkdir build && cd build
python ../cmake/gen_cmake_config.py    # answer y/n for each backend
cmake .. && cmake --build . --parallel

Source builds take 30-60 minutes; only needed for custom kernels or unsupported targets.

---

The Pipeline: Convert → Compile → Run

# 1. Convert HF weights to MLC format
mlc_llm convert_weight ./Llama-3.2-3B-Instruct \
    --quantization q4f16_1 \
    --output ./mlc/Llama-3.2-3B-Instruct-q4f16_1

# 2. Generate model config
mlc_llm gen_config ./Llama-3.2-3B-Instruct \
    --quantization q4f16_1 \
    --conv-template llama-3 \
    --output ./mlc/Llama-3.2-3B-Instruct-q4f16_1

# 3. Compile to target device
mlc_llm compile ./mlc/Llama-3.2-3B-Instruct-q4f16_1/mlc-chat-config.json \
    --device cuda \
    --output ./mlc/Llama-3.2-3B-Instruct-q4f16_1-cuda.so

# 4. Run via Python or serve
mlc_llm chat ./mlc/Llama-3.2-3B-Instruct-q4f16_1 \
    --model-lib ./mlc/Llama-3.2-3B-Instruct-q4f16_1-cuda.so

The mlc-chat-config.json lives in the model directory and tells the runtime which library to load.

---

Your First Model: Llama 3.2 3B on CUDA

# Download from MLC Hugging Face hub (pre-quantized + pre-compiled for cuda)
git lfs install
git clone https://huggingface.co/mlc-ai/Llama-3.2-3B-Instruct-q4f16_1-MLC

# Run
mlc_llm serve HF://mlc-ai/Llama-3.2-3B-Instruct-q4f16_1-MLC \
    --host 0.0.0.0 --port 8000

# Test (OpenAI compatible)
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "llama-3.2-3b",
        "messages": [{"role":"user","content":"Hello"}]
    }'

The HF:// prefix tells MLCEngine to download from Hugging Face directly. For local models pass a filesystem path.

---

Quantization Formats

In practice: q4f16_1 for 95% of users; q4f32_1 if you have VRAM headroom and quality matters; q3f16_1 only when you have to.

---

MLCEngine OpenAI-Compatible Server

mlc_llm serve HF://mlc-ai/Llama-3.1-8B-Instruct-q4f16_1-MLC \
    --host 0.0.0.0 --port 8000 \
    --device cuda \
    --mode local \
    --max-batch-size 8

--mode options:

• local — single user, lowest latency
• interactive — small batch, balanced
• server — high throughput, multi-user (similar to vLLM continuous batching)

Endpoints: /v1/chat/completions, /v1/completions, /v1/embeddings, /health.

Tool calling and JSON mode work via the OpenAI tools and response_format fields. xgrammar is the constrained-generation backend.

---

Apple Silicon (Metal)

Mac-native build is the simplest:

pip install --pre -U -f https://mlc.ai/wheels mlc-llm mlc-ai

mlc_llm serve HF://mlc-ai/Llama-3.1-8B-Instruct-q4f16_1-MLC --device metal

On an M4 Max, the engines that run Llama 3.1 8B at 4-bit all share one hard limit: memory bandwidth divided by the model size. Apple lists 410 GB/s or 546 GB/s depending on the M4 Max variant, and a 4-bit 8B checkpoint is about 4.9 GB (the size Ollama lists for llama3.1:8b), so the ceiling is roughly 84-111 tok/s.

No independent benchmark compares all four on the same Mac with the same model, so treat claims that one is faster as model- and version-specific; the ceiling above applies to all of them. For a measured reference point, Ollama 0.40.13 ran llama3.1:8b at a median 27.2 tok/s on our Apple M3 Pro (18 GB), against a 31 tok/s ceiling on that chip's 150 GB/s. For deep coverage of Mac local AI, see MLX vs CUDA and Apple M4 for AI Guide.

---

AMD GPUs (ROCm + Vulkan)

ROCm (RX 7000+, MI series)

pip install --pre -U -f https://mlc.ai/wheels mlc-llm-rocm62 mlc-ai-rocm62

mlc_llm serve HF://mlc-ai/Llama-3.1-8B-Instruct-q4f16_1-MLC --device rocm

Vulkan (RX 6000-series, RX 5000-series, integrated)

pip install --pre -U -f https://mlc.ai/wheels mlc-llm-vulkan mlc-ai-vulkan

mlc_llm serve HF://mlc-ai/Llama-3.1-8B-Instruct-q4f16_1-MLC --device vulkan

Vulkan is 60-80% the speed of native ROCm. For RX 6000-series and older where ROCm is unofficial, Vulkan is often the better choice. See AMD ROCm Setup for the broader AMD picture.

---

Intel Arc (Vulkan)

pip install --pre -U -f https://mlc.ai/wheels mlc-llm-vulkan mlc-ai-vulkan

# Verify Vulkan sees the Arc
vulkaninfo --summary | grep "Intel"

mlc_llm serve HF://mlc-ai/Llama-3.1-8B-Instruct-q4f16_1-MLC --device vulkan

On the Arc A770 16GB (560 GB/s, Intel spec), Llama 3.1 8B at 4-bit has a ceiling of about 114 tok/s; Vulkan typically lands well below it. See Intel Arc A770 Local AI for the dedicated Arc guide.

---

Android Deployment

The MLC-LLM Android app source is in android/MLCChat in the repo.

cd mlc-llm/android/MLCChat
# Build with Android Studio (Arctic Fox or newer)

Bundled models in the demo app: Llama 3.2 1B / 3B q4f16_1, Qwen 2.5 1.5B q4f16_1, Phi-3.5 Mini q4f16_1, Gemma 2 2B q4f16_1.

For your own app, use mlc4j (Java/Kotlin bindings) and ship a .so library + tokenizer + config. Per-platform compile:

mlc_llm compile model-config.json \
    --device android \
    --opt 'cublas_workspace_size=16' \
    --output libmlc-llm-llama-3.2-3b-android.so

Performance on Snapdragon 8 Gen 3 (Galaxy S24): Qualcomm does not publish a memory-bandwidth figure for the chip, so no speed ceiling is quoted here. A 3B q4f16_1 model is the practical size for sustained use; time it on your target device.

---

iOS Deployment

cd mlc-llm/ios/MLCChat
# Build with Xcode 15+

iOS uses Metal as the backend (same as Mac). Bundled models: Llama 3.2 1B / 3B q4f16_1, Qwen 2.5 1.5B / 3B q4f16_1, Phi-3.5 Mini.

For App Store distribution, the limit is your bundle size (typically <500 MB) — ship a 1B q4f16_1 model (~600 MB) via on-demand download instead.

Performance on iPhone 15 Pro: Apple does not publish the A17 Pro's memory bandwidth, so no ceiling is quoted. Expect speed to drop once the phone heats up under continuous load.

---

WebLLM: Running in the Browser

WebLLM is the JavaScript / TypeScript wrapper around MLC-LLM compiled to WebGPU + WebAssembly.

npm install @mlc-ai/web-llm

import { CreateMLCEngine } from "@mlc-ai/web-llm";

const engine = await CreateMLCEngine(
  "Llama-3.1-8B-Instruct-q4f16_1-MLC",
  { initProgressCallback: p => console.log(p) }
);

const reply = await engine.chat.completions.create({
  messages: [{ role: "user", content: "Hello" }],
  stream: true,
});

for await (const chunk of reply) {
  process.stdout.write(chunk.choices[0].delta.content || "");
}

The model downloads on first run (1-5 GB depending on size) and caches in IndexedDB. Subsequent loads are instant.

Browser support:

• Chrome 113+ ✅
• Safari 18+ ✅ (macOS 15, iOS 18)
• Edge 113+ ✅
• Firefox: experimental (about:config: dom.webgpu.enabled=true)

Speed ceilings (Llama 3.1 8B at 4-bit, about 4.9 GB):

Ceiling = vendor memory bandwidth ÷ model size. It is an upper bound, not a measurement; WebGPU in a browser lands further below it than native engines do.

For deeper coverage see WebLLM Browser AI Guide.

---

Long Context, FlashAttention, KV Cache

MLC-LLM supports up to the model's native context length plus RoPE scaling for extension. FlashAttention is implemented via TVM (not the official Tri Dao kernel) and enabled automatically.

KV cache quantization is supported via --quantization q4f16_kv etc. (limited to specific backends as of mid-2026).

For 32K-context Llama 3.1 8B on a 12 GB RTX 3060:

mlc_llm compile model-config.json \
    --device cuda \
    --max-seq-len 32768 \
    --output llama-3.1-8b-32k-cuda.so

max-seq-len controls the engine's reserved KV-cache memory. Lower = more concurrent requests fit; higher = longer single requests fit.

---

Tool Calling and Structured Output

MLCEngine supports OpenAI-compatible tool calling for chat-tuned models:

{
  "model": "llama-3.1-8b",
  "messages": [...],
  "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {...}}}],
  "tool_choice": "auto"
}

For JSON-schema constrained generation, pass response_format:

{
  "response_format": {
    "type": "json_schema",
    "json_schema": {"name": "person", "schema": {...}}
  }
}

xgrammar is the backend; throughput overhead is 5-15%, similar to vLLM.

---

Benchmarks Across Platforms

Llama 3.1 8B at 4-bit (about 4.9 GB), batch size 1. Single-stream decoding reads every weight once per token, so each device has a hard speed ceiling of memory bandwidth ÷ model size. Engines on the same device share that ceiling; their overhead decides how close they get.

Ceilings are upper bounds, not measurements. Bandwidth sources: NVIDIA Ada whitepaper (RTX 4090), Apple MacBook Pro tech specs (M4 Max), AMD product page (RX 7900 XTX), Intel product specs (Arc A770 16GB).

On CUDA, specialized engines such as vLLM and ExLlamaV2 are the usual choice for raw NVIDIA throughput, and llama.cpp is the default on Metal. MLC's case is portability: one toolchain across every platform here, and the only option for some (browser, iOS, Android).

---

Tuning Recipes

NVIDIA RTX 4090

mlc_llm compile model-config.json \
    --device cuda --opt 'cublas_workspace_size=16' \
    --max-seq-len 16384 --max-batch-size 8 \
    --output llama-3.1-8b-cuda.so

Apple M4 Max

mlc_llm compile model-config.json \
    --device metal --max-seq-len 16384 \
    --output llama-3.1-8b-metal.dylib

Android

mlc_llm compile model-config.json \
    --device android \
    --max-seq-len 4096 \
    --quantization q4f16_1 \
    --output libmlc-llm-llama-3.2-3b-android.so

WebGPU (browser)

mlc_llm compile model-config.json \
    --device webgpu \
    --max-seq-len 4096 \
    --output llama-3.1-8b-webgpu.wasm

---

Common Errors

---

FAQ

See answers to common MLC-LLM questions below.

---

Sources: MLC-LLM GitHub | MLC-LLM docs | WebLLM | Apache TVM | Speeds are bandwidth ceilings from vendor memory-bandwidth specs (NVIDIA, AMD, Intel, Apple), not measurements; the one measured figure is our Apple M3 Pro run with Ollama 0.40.13.

Related guides:

• WebLLM Browser AI Guide
• AMD ROCm Setup for Local LLMs
• MLX vs CUDA for Local AI
• Gemini Nano Android Guide
• Intel Arc A770 Local AI
• vLLM Complete Setup Guide
• TensorRT-LLM Setup Guide