Dub Videos Into Any Language Locally: pyVideoTrans + Whisper + Voice Cloning

Short answer: use pyVideoTrans (18,600+ GitHub stars, v4.08, GPL-3.0) to dub a video into another language entirely on your own machine — faster-whisper transcribes, Ollama translates, and the built-in F5-TTS clones a voice for the new audio track. No API key, no upload, $0 per video, and it is the only mainstream open-source dubber that gives different speakers different voices. Pick VideoLingo (18,000+ stars, Apache-2.0) instead only when polished single-voice subtitles matter more than dubbing — its own README concedes it "cannot dub multiple characters separately."

That is the whole recommendation. The rest of this page is the working detail: what the four-stage pipeline actually does, a spec-for-spec comparison pulled from both repos in August 2026, the exact offline setup (including the two config gotchas the official docs bury), how multi-speaker dubbing works and when it falls over, and the limitations neither project's README leads with. We already cover the pieces of this pipeline separately — faster-whisper, subtitles with Whisper, F5-TTS — this is the page that chains them into finished dubbed video.

---

What Local Dubbing Actually Involves {#pipeline}

A dub is four models in a trench coat: speech recognition → subtitle translation → speech synthesis → video muxing. pyVideoTrans runs all four as one click, and every stage has a fully offline option.

The pyVideoTrans README describes its workflow exactly that way — "Speech Recognition (ASR) → Subtitle Translation → Speech Synthesis (TTS) → Video Synthesis" — and it is worth understanding what each stage does before you trust it with an hour of footage:

1. ASR. The original audio becomes timestamped subtitles. Local engines: faster-whisper (the README's own "recommended" pick), original openai-whisper, a whisper.cpp channel (v4.08 added built-in Windows support for it), and WhisperX or Parakeet when you need speaker labels.
2. Translation. The subtitle text is translated. Local options: Ollama (any model you already run) or the built-in offline M2M100 channel (the model downloads on first use). Cloud options (DeepSeek, ChatGPT, Gemini, Google) exist but defeat the point of this page.
3. TTS. The translated lines are spoken. This is where local options got good: F5-TTS is built directly into pyVideoTrans since v4.04 — voice cloning with no separate install — alongside CosyVoice, GPT-SoVITS, ChatTTS, and ChatterBox as local channels.
4. Muxing. The new audio is aligned to the original subtitle timings and embedded back into the video, optionally with dual-language subtitles burned in or soft-muxed.

Each stage is swappable, which is why this workflow has aged well: when a better local TTS model lands, it becomes a dropdown option rather than a rewrite. If you only need stage 1 — subtitles, no dub — our Whisper subtitles guide is the shorter path.

---

pyVideoTrans vs VideoLingo: The Data {#comparison}

Both projects are the same size (18.6K vs 18.1K stars) but they are not the same tool: pyVideoTrans is a dubbing machine with a subtitle feature; VideoLingo is a subtitle machine with a dubbing feature.

Everything in this table comes from the GitHub API and each project's README and docs, checked August 6, 2026:

Two rows decide most real cases. If you are dubbing — especially anything with two or more speakers — pyVideoTrans is the only one of the two that even attempts per-speaker voices, and its bundled F5-TTS means voice cloning works out of the box. If you are producing translated subtitles for a single-speaker video and want them to read like a streaming service wrote them, VideoLingo's segmentation genuinely is the better output (more in its section below).

---

Fully Offline Setup: pyVideoTrans {#setup}

Install takes four commands from source (or one .exe download on Windows), then three dropdown choices make it fully offline: faster-whisper for ASR, Ollama or M2M100 for translation, F5-TTS for dubbing.

Install

Windows 10/11 users get the shortest path: download the prepackaged release from the pyVideoTrans releases page (currently v4.08), extract, and run sp.exe — no Python environment needed.

From source (macOS, Linux, or Windows), the project moved to uv:

git clone https://github.com/jianchang512/pyvideotrans.git
cd pyvideotrans
uv sync
uv run sp.py

The README recommends Python 3.10. On an NVIDIA card, install the CUDA 12.x build of PyTorch (the README carries the exact command for the current version) — the ASR and TTS stages are the ones that benefit.

The three offline choices

1. ASR channel: faster-whisper (local). Pick it in the speech-recognition dropdown and choose a model size. The published numbers justify the README's "recommended" label: SYSTRAN's benchmark transcribes 13 minutes of audio in 59 seconds at 2.9GB VRAM (large-v2, int8, RTX 3070 Ti, CUDA 12.4) versus 2m23s at 4.7GB for original openai/whisper — "up to 4 times faster for the same accuracy while using less memory." Model-size tradeoffs are covered in our faster-whisper guide; large-v3 in int8 is the accuracy pick, small if you are on CPU.

2. Translation channel: Ollama or M2M100. Point the Ollama channel at your local instance (default port 11434) and pick a model you already run — an instruction-tuned 7-8B handles subtitle-length segments, and our 12GB VRAM model picks all qualify. No Ollama? The built-in M2M100 channel translates offline with no external service at all — quality is a step below a good LLM but the dependency count is zero. For longer-form translation workflows beyond subtitles, see offline document translation.

3. TTS channel: F5-TTS (built in). Since v4.04 there is nothing to deploy. For voice cloning, the official docs specify: record a clean 3-10 second WAV of the target voice, drop it in the f5-tts folder inside the pyVideoTrans root, and name it yourvoice.wav#the exact text spoken in the clip — filename plus transcript after the #. Models auto-download on first use (budget real time for that; the docs warn it can be slow). Full engine details in our F5-TTS setup guide.

The two config gotchas (from the official docs)

• GPT-SoVITS is not built in: you must run its own API service (api_v2.py) and point pyVideoTrans at http://127.0.0.1:9880 — the docs are explicit that the address must be 127.0.0.1, not 0.0.0.0, and the API terminal has to stay open.
• CosyVoice also runs as a separate service (webui.py at http://127.0.0.1:8000), and the stock CosyVoice3 web UI cannot be connected as-is — you must set streaming=True to streaming=False in its code or use the pre-modified build the docs link. If that sounds like friction, it is; F5-TTS being bundled is exactly why it is the default recommendation here. ChatterBox is the other local channel worth a look if F5-TTS's output does not suit your language.

Then pick source and target language, queue the video, and let it run. On processing time we will only put numbers on what is published: the ASR stage benchmark above is the one hard figure, and the TTS stage generates every subtitle line as a separate clip, so total wall-clock scales with how much is actually spoken — not just video length. Run one short video end to end before committing an hour-long file.

---

Multi-Speaker Dubbing: The Feature That Separates Them {#multi-speaker}

Switch pyVideoTrans's ASR channel to WhisperX or Parakeet and it can assign a different dubbing voice to each detected speaker — the one capability VideoLingo explicitly dropped.

Mechanically: WhisperX and Parakeet both support speaker diarization — labeling which segments belong to which speaker — and pyVideoTrans uses those labels to route each speaker's lines to a different TTS voice. An interview dubs as an interview instead of one android reading both sides. We cover the engine itself — including how its diarization actually works — in the WhisperX guide.

Now the honest half. Diarization is the flakiest stage in this whole pipeline, and you do not have to take our word for it — VideoLingo's README gives it as the reason the feature does not exist there: "whisperX's speaker distinction capability is not sufficiently reliable," so it "cannot dub multiple characters separately." pyVideoTrans ships the feature anyway and lets you judge. In practice the failure pattern is predictable:

• Works well: two speakers, clean audio, minimal overlap — podcasts, interviews, lecture Q&A.
• Degrades: crosstalk, laughter over speech, background music, more than three speakers, or speakers with similar voices. Misattributed segments mean the wrong voice speaks a line mid-conversation.

The practical workflow is to run diarization, then review the subtitle editor before synthesis — pyVideoTrans pauses between stages so you can fix speaker labels and translations before it spends GPU time speaking them. For a two-person YouTube interview that review takes minutes and the result is a genuinely usable dub. For a five-way panel over music, dub it single-voice or do not dub it at all.

---

When VideoLingo Is the Better Pick {#videolingo}

Choose VideoLingo when subtitle quality is the product: single-line Netflix-style output, NLP-based splitting, and a three-step translate-reflect-adapt pass produce translations that read noticeably better than a raw one-shot translation.

VideoLingo's pitch — "Netflix-level subtitle cutting, translation, alignment, and even dubbing" — is mostly about the first three words. It transcribes with word-level WhisperX, splits lines with NLP so subtitles break at natural phrase boundaries, maintains a terminology list for consistent translation of names and jargon, and runs each translation through a reflect-and-adapt second pass. Input languages: English, Russian, French, German, Italian, Spanish, Japanese, and Chinese. Setup is python setup_env.py then streamlit run st.py (or the Docker image on port 8501), and it runs free and local against an Ollama endpoint via its OpenAI-compatible API support.

The honesty notes for a local-first audience:

• Its README pairs "free" with Edge-TTS — which is not offline. Edge-TTS is Microsoft's hosted service; it costs nothing but your audio text goes to Microsoft's servers. VideoLingo's only fully-local dubbing path is wiring up a separate GPT-SoVITS install.
• Single voice only, per its own limitations section, and "dubbing feature may not be 100% perfect due to differences in speech rates and intonation" — their words, and fair.
• Multilingual source videos keep only the main language in transcription.

So the split is clean: VideoLingo when the deliverable is beautiful translated subtitles and any dub is a bonus; pyVideoTrans when the deliverable is the dub itself, offline, cloned, possibly multi-voice.

---

Honest Limitations {#limitations}

Local dubbing in 2026 is genuinely usable and genuinely imperfect: timing drift, no lip-sync, translation quality capped by your local model, and diarization errors are all still real.

• Speech-rate mismatch. A translated sentence rarely takes the same time to say. pyVideoTrans can stretch audio or speed playback to compensate, but aggressive correction sounds unnatural. VideoLingo's README owns the same issue for its own dubs. Languages that expand a lot in translation (English→German, English→Spanish) show it most.
• No lip-sync. This whole pipeline replaces the audio track. Mouths keep moving in the original language. For talking-head content viewers tolerate it (it is how most human dubbing ships too); just do not expect the on-screen mouth to match.
• Translation quality is your model's quality. An 8B model under Ollama produces serviceable subtitle translation, not literary translation. Idioms, jokes, and culture-specific references are where it shows. The review step between translation and synthesis exists for a reason — use it.
• Cloned-voice quality varies by language. F5-TTS cloning from a 3-10s reference is impressive in its strong languages and mediocre outside them; test your target language on one clip before batch-processing. Our local TTS model roundup covers alternatives per language.
• No end-to-end benchmark exists. Neither project publishes full-pipeline timings, and we have not published our own timed run yet — the faster-whisper ASR figure above is the only stage with hard public numbers. Treat any "dubs an hour of video in X minutes" claim you read elsewhere as unattributed.
• Ethics and rights. Voice cloning a speaker you do not have permission from is off the table, whatever the tooling makes possible. For your own channel's multi-language versions — the actual use case this pipeline is built for — you are cloning yourself, and it is fine.

---

Verdict {#verdict}

pyVideoTrans is the local dubbing stack to standardize on: fully offline at every stage, F5-TTS cloning built in since v4.04, per-speaker voices no competitor ships, and an active repo (pushed August 4, 2026).

1. Dubbing your own content into other languages: pyVideoTrans with faster-whisper + Ollama + built-in F5-TTS. One .exe on Windows, four commands elsewhere.
2. Interviews and two-person podcasts: same stack, ASR switched to WhisperX or Parakeet for per-speaker voices — and always review speaker labels before synthesis.
3. Translated subtitles as the end product: VideoLingo's single-line output and reflect-translation pass beat pyVideoTrans's subtitle quality; skip its dubbing.
4. Subtitles only, minimal moving parts: skip both and run Whisper directly.

The larger point: every stage of professional dubbing — transcription, translation, voice cloning, muxing — now runs on a mid-range consumer GPU at zero marginal cost per video. Two years ago this pipeline was an API bill; today it is a dropdown menu.

---

Sources {#sources}

• pyVideoTrans repository — stars/forks/license/activity via GitHub API, pipeline and engine lists from the README, v4.08 release notes (all checked August 2026)
• pyVideoTrans official docs — F5-TTS built-in configuration, GPT-SoVITS and CosyVoice service setup details
• VideoLingo repository — stars/forks/license via GitHub API; features, install, and quoted limitations from the README (checked August 2026)
• faster-whisper (SYSTRAN) — published large-v2 benchmark: 13 min audio in 59s / 2,926MB VRAM (int8) vs 2m23s / 4,708MB for openai/whisper, RTX 3070 Ti, CUDA 12.4

---

FAQ {#faq}