WhisperX Guide (2026): Word-Level Timestamps + Speaker Diarization for Local STT

WhisperX is the right tool when plain transcription is not enough. Built on Faster-Whisper, it adds forced phoneme alignment for precise word-level timestamps and pyannote-based speaker diarization. For podcasts, interviews, meeting transcripts, video subtitles, and any workflow where you need to know exactly who said what when, WhisperX is the production-grade open-source answer.

If you only need plain transcripts without speaker labels, start with our guide to run Whisper locally instead. This guide covers everything: installation, getting the Hugging Face token for pyannote, basic transcription with diarization, output formats (SRT/VTT/JSON), batching for long-form audio, multi-language support, integration with video pipelines, accuracy tuning, and the cases where WhisperX is overkill vs. where it's essential.

Table of Contents

1. What WhisperX Is
2. Word-Level Timestamps via Forced Alignment
3. Speaker Diarization via Pyannote
4. Hardware Requirements
5. Installation
6. Hugging Face Token Setup
7. Your First Transcription with Speakers
8. CLI Usage
9. Python API
10. Output Formats: SRT, VTT, JSON
11. Multi-Language Support
12. Setting Speaker Counts
13. Batched Long-Form Inference
14. Integration with Video Pipelines
15. Accuracy Tuning
16. Performance Benchmarks
17. WhisperX vs Commercial Services
18. Tuning Recipes
19. Troubleshooting
20. FAQ

---

What WhisperX Is

WhisperX (m-bain/whisperX) is a Python library + CLI that wraps Faster-Whisper and adds:

• Forced phoneme alignment via wav2vec2 → accurate word-level timestamps (Whisper's own word timings are inferred and can drift)
• Speaker diarization via pyannote.audio (speaker-diarization-community-1 in current releases) → "who said what when"
• Batched inference with VAD-based segmentation → higher throughput on long audio
• Output formatters for SRT, VTT, TSV, JSON

Project: github.com/m-bain/whisperX. License: BSD 2-Clause. Latest PyPI release as of November 2, 2026: 3.8.6 (May 25, 2026).

WhisperX inherits its accuracy from the underlying Whisper model. If raw transcription speed matters more to you than diarization, it's worth comparing the engine itself against NVIDIA's faster ASR model in our Parakeet vs Whisper comparison.

---

Word-Level Timestamps via Forced Alignment

Whisper natively emits timestamps but only at the segment level (typically 5-30 seconds per segment). For word-level timestamps, Whisper interpolates — and the result drifts by hundreds of milliseconds.

WhisperX runs the transcript through a wav2vec2 phoneme alignment model that maps each word to its precise audio position. Result: per-word start and end times tight enough for karaoke-style word highlighting in subtitles; the WhisperX paper (INTERSPEECH 2023) reports state-of-the-art word segmentation results.

The alignment step is a second model pass (the README notes that installing Numba via pip install "whisperx[fast]" speeds up alignment on long files) but is essential for video subtitles, dubbed content alignment, and any time-sensitive transcript use.

---

Speaker Diarization via Pyannote

Pyannote is the most widely used open-source speaker diarization toolkit; current WhisperX uses its speaker-diarization-community-1 pipeline. WhisperX:

1. Runs pyannote on the audio → speaker turn boundaries
2. Aligns Whisper transcript words with diarization output
3. Assigns each word to a speaker label (SPEAKER_00, SPEAKER_01, ...)

Published diarization error rates for community-1 (pyannote model card, benchmark last updated September 2025; lower is better): VoxConverse 11.2%, AMI headset meetings 17.0%, DIHARD 3 20.2%, CALLHOME telephone 26.7%. Clean two-speaker audio sits at the easy end; crowded meetings with crosstalk sit at the hard end.

You can map labels to real names post-hoc: SPEAKER_00 → Alice, SPEAKER_01 → Bob based on the first known voice sample.

---

Hardware Requirements

The only speed figure the WhisperX project publishes is 70x realtime for batched transcription with large-v2 on a GPU (README, checked November 2, 2026; the GPU is not named), and the WhisperX paper reports a twelve-fold speedup from its VAD Cut & Merge batching. We have not timed WhisperX ourselves, so the table below shows where each stage runs and what limits it rather than real-time factors:

VRAM: the README says the backend needs under 8 GB of GPU memory for large-v2 with beam_size=5; 8 GB+ is comfortable. CPU diarization is impractical except for small workloads.

---

Installation

python3.11 -m venv ~/venvs/whisperx
source ~/venvs/whisperx/bin/activate

# GPU: install the CUDA toolkit 12.8 first (the version the README names), then:
pip install whisperx
pip install "whisperx[fast]"   # optional: Numba-accelerated alignment

For CPU only:

pip install whisperx
# torch CPU is installed by default

For Docker: the project does not publish an official container image. Build your own from an NVIDIA CUDA runtime base image with pip install whisperx, run it with --gpus all, and mount your audio folder; review any community image before running it on private recordings.

---

Hugging Face Token Setup

Diarization needs a Hugging Face token because pyannote models are gated.

1. Visit huggingface.co/pyannote/speaker-diarization-community-1 → accept the user conditions
2. (Older WhisperX releases on pyannote.audio 3.x only: also accept huggingface.co/pyannote/speaker-diarization-3.1 and huggingface.co/pyannote/segmentation-3.0)
3. Go to huggingface.co/settings/tokens → create a Read token
4. Export: export HF_TOKEN=hf_xxxxxxxxxxxxxxx

Pass to WhisperX via --hf_token (CLI) or the token argument of DiarizationPipeline (Python).

For transcription-only (no diarization), no token needed.

---

Your First Transcription with Speakers

whisperx audio.mp3 \
    --model large-v3 \
    --diarize \
    --hf_token $HF_TOKEN \
    --min_speakers 2 \
    --max_speakers 4 \
    --output_format all

Outputs in current directory:

• audio.srt — subtitles with speaker labels
• audio.vtt — WebVTT for HTML5 video
• audio.json — full structured output (words, timestamps, speakers)
• audio.tsv — tab-separated for spreadsheet import
• audio.txt — plain text

---

CLI Usage

# Basic transcription only (fast)
whisperx audio.mp3 --model large-v3

# Word timestamps (alignment)
whisperx audio.mp3 --model large-v3 --align

# Full pipeline: transcription + alignment + diarization
whisperx audio.mp3 \
    --model large-v3 \
    --diarize \
    --hf_token $HF_TOKEN \
    --min_speakers 2 --max_speakers 4 \
    --language en \
    --batch_size 16 \
    --compute_type float16 \
    --output_format srt \
    --highlight_words True

Useful flags:

• --language en — skip language detection (faster, more accurate)
• --batch_size 16 — VAD-batched inference (RTX 4090: bump to 32)
• --compute_type float16 — int8_float16 for tighter VRAM
• --highlight_words True — per-word highlights in SRT output
• --print_progress True — progress bar

---

Python API

import whisperx

device = "cuda"
audio = whisperx.load_audio("audio.mp3")

# 1. Transcribe with Whisper
model = whisperx.load_model("large-v3", device, compute_type="float16")
result = model.transcribe(audio, batch_size=16)

# 2. Align (word-level timestamps)
align_model, metadata = whisperx.load_align_model(
    language_code=result["language"], device=device
)
result = whisperx.align(result["segments"], align_model, metadata, audio, device)

# 3. Diarize
from whisperx.diarize import DiarizationPipeline
diarize_model = DiarizationPipeline(token=HF_TOKEN, device=device)
diarize_segments = diarize_model(audio, min_speakers=2, max_speakers=4)
result = whisperx.assign_word_speakers(diarize_segments, result)

# Result has: result["segments"] with .words[] containing .word .start .end .speaker

---

Output Formats: SRT, VTT, JSON

SRT (subtitles)

1
00:00:00,520 --> 00:00:03,840
[SPEAKER_00] Welcome to the show, today we are talking about local AI.

2
00:00:04,120 --> 00:00:07,200
[SPEAKER_01] Thanks for having me. It is great to be here.

VTT (HTML5 video)

WEBVTT

00:00.520 --> 00:03.840
<v SPEAKER_00>Welcome to the show, today we are talking about local AI.</v>

00:04.120 --> 00:07.200
<v SPEAKER_01>Thanks for having me. It is great to be here.</v>

JSON (full structured)

{
  "segments": [{
    "start": 0.52, "end": 3.84,
    "text": "Welcome to the show...",
    "speaker": "SPEAKER_00",
    "words": [
      { "word": "Welcome", "start": 0.52, "end": 0.91, "speaker": "SPEAKER_00" },
      { "word": "to", "start": 0.94, "end": 1.06, "speaker": "SPEAKER_00" },
      { "word": "the", "start": 1.08, "end": 1.18, "speaker": "SPEAKER_00" }
    ]
  }],
  "language": "en"
}

The JSON drives downstream applications: TikTok-style word highlights, video editor sync, searchable archives.

---

Multi-Language Support

WhisperX inherits Whisper's 99-language support. Alignment models exist for the major languages:

Pass --language es etc. WhisperX auto-loads the matching alignment model.

For languages without an alignment model, transcription works but word timestamps fall back to interpolated.

Diarization is language-agnostic.

---

Setting Speaker Counts

If you know the speaker count, tell pyannote:

whisperx audio.mp3 --diarize --min_speakers 2 --max_speakers 2 --hf_token $HF_TOKEN

Hard limits dramatically improve accuracy. For two-person podcasts, set min=max=2. For meetings of unknown size, leave defaults (auto-detect, typically 1-10).

If you provide just one bound (--min_speakers 3), pyannote uses it as a hint but may produce more or fewer.

---

Batched Long-Form Inference

For long audio (1+ hours):

whisperx audio.mp3 \
    --model large-v3 \
    --batch_size 32 \
    --diarize --hf_token $HF_TOKEN

VAD pre-segmentation splits audio at silences, then transcribes 32 segments in parallel on GPU. This batched mode is what the README's 70x-realtime headline (large-v2) refers to; --batch_size is the main lever, so lower it if you run out of VRAM.

For massive archives (1,000+ hours), use the Python API with explicit chunking and async pyannote calls.

---

Integration with Video Pipelines

For video subtitle generation:

# Extract audio from video
ffmpeg -i video.mp4 -vn -acodec pcm_s16le -ar 16000 audio.wav

# Transcribe with WhisperX
whisperx audio.wav --model large-v3 --align --output_format srt

# Burn subtitles back into video
ffmpeg -i video.mp4 -vf "subtitles=audio.srt" video_subtitled.mp4

For a fully automated pipeline:

# auto-subtitle.sh
#!/bin/bash
ffmpeg -i "$1" -vn -ar 16000 -ac 1 audio.wav
whisperx audio.wav --model large-v3 --align --language en --output_format srt
ffmpeg -i "$1" -vf "subtitles=audio.srt:force_style='FontSize=22'" "${1%.*}_sub.mp4"
rm audio.wav audio.srt

---

Accuracy Tuning

To improve diarization quality:

1. Clean audio first — run Demucs or Audacity noise reduction
2. Set speaker counts — min_speakers / max_speakers
3. Use higher-quality model — large-v3, not turbo (turbo is English-tuned, less accurate for diarization edge cases)
4. Adjust VAD sensitivity — pass custom vad_options dict
5. Post-process — manually correct SPEAKER_X labels with sed/awk; cluster similar voices across recordings with voice fingerprinting

For mission-critical accuracy (court records, depositions): consider commercial Pyannote Premium or human-in-the-loop verification.

---

Performance Benchmarks

Published figures only (we have not benchmarked WhisperX ourselves):

Processing time grows roughly with audio length, so time a one-hour file on your own machine and scale from there for long archives.

---

WhisperX vs Commercial Services

For most local podcasters, journalists and researchers, WhisperX is worth testing first: full privacy and zero per-minute cost. Compare it on a few of your own recordings before deciding.

---

Tuning Recipes

Podcast / interview transcription

whisperx interview.mp3 \
    --model large-v3 --align --diarize \
    --min_speakers 2 --max_speakers 3 \
    --language en \
    --hf_token $HF_TOKEN \
    --output_format srt

YouTube auto-captioning

yt-dlp -x --audio-format wav "https://youtu.be/..."
whisperx *.wav --model large-v3 --align --output_format vtt

Meeting transcription with up to 8 speakers

whisperx meeting.wav \
    --model large-v3 --align --diarize \
    --max_speakers 8 \
    --hf_token $HF_TOKEN \
    --output_format json

Multi-language interview

whisperx multilingual.mp3 \
    --model large-v3 --align --diarize \
    --hf_token $HF_TOKEN
# Auto-detects primary language; alignment uses matching wav2vec2

---

Troubleshooting

---

FAQ

See answers to common WhisperX questions below.

---

Sources: WhisperX GitHub | pyannote.audio | WhisperX paper | pyannote speaker-diarization-community-1 model card | whisperx on PyPI. All checked November 2, 2026.

Related guides:

• Faster-Whisper Guide
• Whisper Local Speech-to-Text
• Local AI Meeting Transcription
• Local AI Podcast Production
• Air-Gapped AI Deployment
• F5-TTS Setup Guide