Local Speech Studio: TTS, Voice Cloning and Transcription

Speech AI gets discussed as a single subject and it is really two, with almost nothing in common beyond the file format.

Text to speech is a generation problem. You supply text and the model produces a waveform that did not exist. The interesting failures are aesthetic: wrong emphasis, a question read as a statement, an abbreviation pronounced letter by letter, a voice that is technically clear and emotionally dead. There is no correct answer to compare against, so quality is judged by ear and consistency is judged over hours of output rather than seconds.

Speech to text is a recognition problem. There is a right answer, you can measure distance from it, and the failure modes are factual. Words are missed, names are mangled, speakers are confused with each other, and — with the current generation of models — text is sometimes produced for audio that contained no speech at all.

They are joined by a third thing that gets least attention and causes the most trouble: audio engineering. Recognition models are trained on a specific sample rate and will resample whatever you give them, so a beautiful 48 kHz studio recording buys you nothing at the model boundary while a noisy phone recording costs you accuracy that no model choice will recover. Synthesis models emit at their own native rate and concatenating clips from two engines without resampling produces audible pitch and timing artifacts. Loudness normalization is not a polish step; it is the difference between an audiobook that gets accepted by a distributor and one that gets rejected.

Why local, specifically

The obvious answer is money, and for high-volume work it is a real answer, since per-character synthesis pricing and per-minute transcription pricing both scale linearly with exactly the thing you are trying to do more of. But three other reasons come up more often among people who actually make the switch.

Privacy is the first, and it is often not optional. Interview recordings under source protection, clinical dictation, legal discovery material and internal company audio frequently cannot be sent to a third-party processor at all, regardless of that processor's contractual assurances. Local processing removes the question.

Determinism is the second. A hosted voice can be retired, retuned or repriced between one chapter of a project and the next, and a narrated series that changes voice halfway through is worse than one that never launched. A checkpoint on your own disk does not change unless you change it.

Iteration cost is the third and least discussed. When each attempt is free, you regenerate a sentence twelve times to get the emphasis right. When each attempt has a price, you accept the third one. That difference compounds across a long project and it shows in the finished audio.

There is no best local TTS engine, and the people who claim otherwise are usually comparing a demo sentence rather than a workload. The engines occupy distinct positions on a curve running from tiny and instant to large and expressive, and the correct choice falls out of three questions: does it need to run on CPU, does it need to be your voice, and are you selling the result.

Piper sits at the fast end. It is a VITS-derived system exported to ONNX, built for the Rhasspy voice assistant ecosystem, tiny enough to run in real time on a Raspberry Pi, and released under a permissive license. Voices are per-speaker and per-language, downloaded individually. The output is clear, intelligible and prosodically flat. For a voice assistant, a screen reader, a notification system or anything where latency dominates, that trade is correct and nothing else comes close on resource cost.

Kokoro is the current answer to "something that sounds good and still runs on a laptop". It is a compact model — the parameter count is in the name of the release — built on a StyleTTS-lineage architecture, distributed under a permissive license, and it produces noticeably more natural prosody than its size suggests. It does not clone voices; it ships a set of trained voices. For narration, video voiceover and any application where you want a good default rather than a specific person, it is the pragmatic starting point.

XTTS v2 from Coqui was the model that made few-shot voice cloning ordinary: a handful of seconds of reference audio, multilingual output, usable quality. It also carries the ecosystem's most-misread license. The weights are released under the Coqui Public Model License, which is non-commercial, and Coqui the company wound down in early 2024, so the code lives on as community forks while the model terms did not change. People discover this after building a product on it.

F5-TTS is a flow-matching, non-autoregressive system with strong zero-shot cloning and fast inference. The code repository and the model weights carry different terms, and the base checkpoint was trained on a speech corpus published under a non-commercial license, which is why its commercial status is argued about rather than settled. Read the model card, not the repository license file.

Chatterbox from Resemble AI is the notable recent case of a genuinely capable cloning model released under a permissive license, with explicit expressiveness controls and a built-in perceptual watermark on generated audio. For anyone who needs cloning and needs to sell the output, the license position alone makes it worth evaluating first.

OpenVoice takes a different architectural route: rather than cloning end to end, it synthesizes with a base speaker and then transfers tone color onto the result, which separates what is said and how it is delivered from who appears to be saying it. GPT-SoVITS sits at the demanding end, combining few-shot cloning with optional fine-tuning, and rewards the setup effort with quality that is hard to beat locally.

The selection questions, in order

Most disappointing clones are not model failures. They are reference audio failures, and the fix is upstream of anything you can configure.

The reference clip is the whole ballgame

Zero-shot cloning models encode a short reference into a speaker embedding and condition generation on it. Everything present in that clip is a candidate for imitation: room reverb, an air conditioner, mouth noise, compression artifacts from a lossy source, the particular emotional register of those few seconds. Give it a clip recorded in a bathroom and you get a voice that sounds like it lives in a bathroom, in every sentence, forever.

What a good reference looks like: clean, close-miked, no background, no music, no reverb, consistent volume, no clipping, and — the part people miss — delivered in the register you want back. A reference clip of someone speaking excitedly produces excited output regardless of what the text says. If you want calm narration, record a calm reference. If you want range, some engines accept multiple references and you can switch between them by section.

Length matters less than quality once you are past a fairly low threshold. Zero-shot systems generally document their reference requirement in seconds rather than minutes, and adding more audio of mixed quality tends to blur the embedding rather than sharpen it. Get a short clip that is genuinely clean before you go looking for a longer one.

Prosody is where synthetic audio gives itself away

Even with a perfect clone, listeners identify synthetic speech from rhythm rather than timbre. Three specific things create the tell.

Uniform pacing. Human speech varies enormously in speed within a sentence, slowing on important words and rushing over connectives. Many synthesis engines are close to metronomic. Where the engine exposes speed or pacing controls, varying them per segment is more effective than any single global setting.

Missing or misplaced pauses. A comma is not a pause instruction, and models handle punctuation inconsistently. The practical technique is to control pauses explicitly by splitting text into segments and inserting measured silence between them, rather than trusting the model to interpret the page.

Wrong emphasis on ambiguous sentences. The model does not know which word carries the meaning, and it guesses from the training distribution. Rewriting the sentence is usually faster than fighting the model, and in narration you are allowed to do that.

Text normalization, the unglamorous half

Before a single audio sample is generated, the text has to be turned into something a model can pronounce. Numbers are the obvious case and they are context-dependent: a year, a quantity, a phone number, an ordinal and a monetary amount are all read differently and look similar. Abbreviations are worse, because expansion depends on domain — the same three letters can be an acronym read letter by letter in one paragraph and a word in the next. Roman numerals, units, section references, footnote markers, em dashes, ellipses and quotation marks all need decisions.

Every long-form project ends up with a normalization stage and a pronunciation dictionary for the proper nouns specific to that project. Building it deliberately at the start is much cheaper than discovering in chapter nine that the protagonist's surname has been mispronounced for eight chapters.

Consent is a design constraint, not an afterthought

Cloning a voice that is not yours, without permission, is a problem in several dimensions at once. Personality and publicity rights cover voice in a growing number of jurisdictions, with the Tennessee ELVIS Act being an explicit example and the EU AI Act's transparency obligations covering synthetic media disclosure. Beyond the law, platform terms increasingly prohibit undisclosed synthetic voice, and some open models embed watermarks in output precisely so that provenance can be established later. The defensible position is a documented consent record for every cloned voice and disclosure wherever the audience could reasonably be misled.

OpenAI's Whisper reset expectations for open speech recognition. The paper describes an encoder-decoder transformer trained on 680,000 hours of weakly supervised multilingual audio collected from the web, and the released weights carry a permissive license. That scale is why it handles accents, background noise and code-switching far better than what came before, and the same training approach is why it has a very specific failure mode.

Whisper hallucinates, and you need to plan for it

Because Whisper is a language model generating text conditioned on audio, it will produce fluent text when the audio contains nothing to transcribe. On silence, on music, on background hum, it emits plausible sentences. Anyone who has run it over unattended recordings has seen subtitle boilerplate appear from nowhere, or the same phrase repeat for a minute of dead air. This is not a bug in a particular build; it is what the architecture does when the acoustic evidence runs out and the decoder keeps going.

The mitigations are known and they stack. Run a voice activity detector first and only feed the model segments that contain speech, which removes the majority of the problem outright. Disable conditioning on previous text, which stops a repetition loop from feeding itself. Use the no-speech and compression-ratio thresholds to reject degenerate output. And treat any segment whose timing looks impossible — a long duration for very few words, or the reverse — as a candidate for review rather than as fact.

The implementations are not interchangeable

The original research implementation is not what you should run in production. faster-whisper reimplements inference on CTranslate2 and delivers substantially better speed and memory behavior from the same weights, which makes it the default choice for most people. whisper.cpp targets CPU and Apple hardware and is the right answer where no GPU exists. WhisperX adds two things Whisper itself does not do well: forced alignment against a phoneme model to produce genuinely accurate word-level timestamps, and a diarization stage. Whisper's own segment timestamps are approximate, and if you are producing subtitles that approximation is visible.

NVIDIA's Parakeet family is the significant alternative, built on a different architecture, English-focused in its most-used checkpoints, and very fast. Models from that family have ranked at the top of Hugging Face's Open ASR Leaderboard. The trade-off is breadth: Whisper's multilingual coverage remains its strongest argument.

Speaker labels are a separate system with separate failures

Diarization answers "who spoke when" and is not part of transcription at all. The common approach uses a pipeline such as pyannote, which segments audio, embeds each segment as a speaker vector, and clusters. It fails in predictable ways. Overlapping speech, which happens constantly in real conversation, is poorly handled by clustering approaches. Speakers with similar voices merge. A speaker who says three words in an hour may be dropped entirely or split off as a phantom. And if you supply the wrong speaker count, or let it estimate badly, everything downstream is wrong in a way that looks confident.

Practical mitigation: constrain the speaker count when you know it, expect to correct the boundaries by hand for anything published, and never present automatic speaker labels as authoritative in a legal or journalistic context without review.

Quality in, quality out

Recognition accuracy is decided before the model runs. Distance from the microphone matters more than the microphone. Reverberant rooms are far worse than quiet noise. Aggressive noise reduction applied before transcription often lowers accuracy rather than raising it, because the artifacts it introduces are unlike anything in the training data. And audio that has been through a low-bandwidth codec has lost information that no model recovers, which is the underlying reason telephone recordings transcribe worse than anything else you will handle.

Generating one good paragraph is a solved problem. Generating nine hours of consistent narration, or subtitling a back catalog, is a systems problem, and it is where most projects stall.

An audiobook pipeline, stage by stage

Text extraction comes first and is dirtier than expected. EPUB gives you structure. PDF gives you a reading-order puzzle with headers, footers, page numbers and hyphenated line breaks that will be read aloud unless you strip them. Anything scanned needs OCR and then needs proofreading, because an OCR error becomes a mispronounced word rather than a visible typo.

Segmentation comes next. Chapters become units of work, and within a chapter you split on sentence boundaries, because every synthesis model has a maximum input length and quality degrades before you reach it. Splitting badly — mid-clause, or on an abbreviation full stop — produces unnatural breaks that no amount of model quality fixes.

Then normalization and pronunciation, as described earlier, applied consistently across the whole book rather than per chapter.

Then generation, one segment at a time, with the same reference audio, the same settings and, where the engine supports it, the same seed. This is where long-form drift shows: tone wanders across hours, and a passage generated on Tuesday can sit oddly beside one generated on Monday. Keeping every generation parameter pinned and storing them alongside the output is what makes a later regeneration match.

Then assembly. Concatenation with controlled silence between segments and slightly longer between paragraphs. Crossfades of a few milliseconds at joins to avoid clicks. And then loudness work, which is where technical requirements become concrete: Audible's ACX submission specification, to take the best-documented example, sets bounds on RMS level, requires peaks below a stated ceiling, and requires a noise floor quieter than a stated threshold. Synthetic audio typically passes the noise floor test trivially and fails the RMS window unless you normalize deliberately.

Finally, packaging: a chaptered M4B with metadata and cover art, which is an ffmpeg exercise.

Subtitles have rules, and they are published

Subtitling is not transcription with timestamps. It is transcription, condensed to a reading speed, split across lines at syntactic boundaries, and timed to shot changes. Netflix publishes a timed text style guide that caps reading speed in characters per second, sets a lower cap for children's programming than for adult content, and specifies maximum line length along with minimum and maximum cue durations. It is worth reading as a reference even for work that will never go near that platform, because it makes explicit the constraints every subtitle house applies. Raw ASR output violates all of these constantly, which is why an automated subtitle pass produces something that reads badly even when every word is correct.

Dubbing is a duration problem disguised as a translation problem

Transcribe, translate, synthesize, and the pipeline appears trivial until you try to fit the result into the original timing. Translated text is systematically longer or shorter than the source depending on the language pair, and you have three options, all bad in different ways. Time-stretch the synthesized audio, which introduces artifacts and sounds rushed. Extend the segment into adjacent silence, which works only when there is silence. Or rewrite the translation to fit the available duration, which is what professional dubbing actually does and which requires a human who speaks both languages. Anyone promising fully automatic lip-synced dubbing is quietly choosing one of the first two.

Serving it as an API

Turning a local engine into a service that an application can call is mostly plumbing, with three decisions that matter. Expose an OpenAI-compatible speech endpoint so that existing client libraries work unchanged. Keep the model resident in memory rather than loading per request, because load time dominates synthesis time for short inputs. And queue requests rather than serving them concurrently on one GPU, because most of these models are not safe to run in parallel on a single device and the failure when you try is a confusing memory error rather than a clean rejection.

Speech models are small compared with language models, which changes the hardware conversation completely. A great deal of useful work here happens on machines that could not load a mid-sized chat model.

The compact synthesis engines are designed for CPU. Real-time factor better than one on a modest processor means you can narrate faster than the audio plays, which is all a voice assistant or a notification system needs, and enough for batch narration if you are willing to leave it running. Adding a GPU speeds these up and rarely changes what is possible.

Cloning models are the tier where a GPU starts to matter, both because they are larger and because iteration count is high: you will generate the same sentence repeatedly while tuning a reference clip, and the difference between two seconds and thirty seconds per attempt decides whether you tune at all.

Transcription is the tier with the widest spread. The small recognition models run comfortably on CPU. The large ones are slow enough on CPU that a long recording becomes an overnight job, and the efficient inference implementations narrow that gap substantially without closing it. If you transcribe regularly, this is the workload that justifies the card.

Memory rather than compute is usually the binding constraint, and for once the numbers are modest. The practical planning rule is the same as everywhere else in local AI — parameters times bytes per parameter, plus working space — but applied to models an order of magnitude smaller than the language models this equipment is usually bought for. A card that is unremarkable for text generation is comfortable here.

Apple Silicon deserves a specific mention because the speech ecosystem has adapted to it unusually well. Metal-accelerated recognition builds are mature, and unified memory removes the capacity anxiety entirely at this model scale. The gaps are in the cloning tier, where some projects still assume CUDA.

Storage plans should account for the audio rather than the models. Speech checkpoints are small. Uncompressed working audio for a long project is not, and a pipeline that keeps intermediate WAV files for every segment of every chapter will consume more disk than every model you have downloaded put together.

This field has an unusually wide gap between what is technically easy and what is legally clear, and the gap runs in the least convenient direction: several of the most impressive open voice models carry the most restrictive terms.

Separate the layers. The license on the code is not the license on the weights, and neither is necessarily the license on the data the weights were trained on. A repository can be MIT while its checkpoint is non-commercial, and a checkpoint can be permissively licensed while being trained on a corpus whose own license forbids commercial derivatives. All three have to work for you before the output is safely sellable, and the model card rather than the repository is where the binding terms usually sit.

The specific traps. The best-known few-shot cloning model of the previous generation is under a non-commercial model license, which surprises people who found it through tutorials that never mentioned this. At least one popular flow-matching system has permissive code and a base checkpoint trained on a non-commercially licensed corpus, leaving its commercial status contested. Meanwhile some genuinely capable cloning models have been released under plainly permissive terms, which makes them the sensible starting point for commercial work even if a demo elsewhere sounded marginally better.

Voice identity is a separate right entirely. Even with a permissive model license, the voice you cloned belongs to somebody. Personality and publicity rights covering vocal likeness are recognized in an increasing number of jurisdictions and legislation aimed specifically at synthetic voice has been passed in some of them. A model license grants you nothing with respect to the person. Written consent, scoped to the uses you intend, is the only defensible position for any voice that is not your own.

Disclosure obligations are arriving independently of licensing. The EU AI Act includes transparency requirements for synthetic media, several platforms require labeling of AI-generated audio, and audiobook and podcast distributors have their own policies on synthetic narration, some of which require declaration at submission. These change more often than model licenses do, and they apply to the distribution channel rather than the model, so they need checking per platform and per project.

Training data for a voice you build yourself. If you fine-tune on recordings, the recordings need clearance too — performer consent, and rights in the underlying material if it is a reading of somebody else's text. The output being synthetic does not launder an input you had no right to use.