Build a Local Voice Assistant: The 2026 Speech-to-Speech Stack

The best local speech-to-speech stack in 2026 is Hugging Face's open speech-to-speech pipeline: Silero VAD v5 for voice detection, NVIDIA Parakeet TDT for transcription, a local LLM served by llama.cpp or vLLM, and Qwen3-TTS or Kokoro-82M for speech output. One pip install; every stage can run on your own hardware, behind an OpenAI Realtime-compatible API.

That's the short answer. The longer one: on July 1, 2026, Hugging Face and Cerebras open-sourced a production voice-agent pipeline — the same one HF says already powers 9,000+ Reachy Mini robots — and the repo explicitly supports swapping every cloud piece for a local one. This page is the successor to our Whisper + Ollama + Piper build: same goal, far less duct tape.

---

The 2026 Stack at a Glance {#the-stack}

The design decision that matters: the LLM slot is swappable by construction. The hosted demo talks to Cerebras; your build talks to a llama.cpp server on 127.0.0.1. Nothing else changes.

---

What HF + Cerebras Shipped {#what-shipped}

On July 1, 2026, Hugging Face published the pipeline with Cerebras as the launch inference partner. The chain is Silero VAD v5 → Parakeet TDT → Gemma 4 31B (on Cerebras, explicitly swappable) → Qwen3-TTS, exposed through an OpenAI Realtime-compatible WebSocket API.

Two things make this more than another demo repo:

1. It's deployed, not hypothetical. HF states the pipeline already powers 9,000+ Reachy Mini robots in the field.
2. It's maintained. The repo (huggingface/speech-to-speech) is on release v0.2.10 as of June 11, 2026, with pip extras for every optional component.

The Realtime-compatible API is the quiet win. Any client written against OpenAI's Realtime WebSocket protocol can point at your box instead. You get the ecosystem's tooling with none of its telemetry.

---

The Fully-Local Swap {#fully-local}

The repo documents local options for every stage:

• ASR: Parakeet TDT (the default, and the right call for European languages — more below), or Whisper, Faster-Whisper, and MLX Whisper variants via --stt.
• LLM: a llama.cpp or vLLM server speaking the OpenAI API. The README's own local example uses the Gemma 4 E4B GGUF — the small sibling of the hosted 31B. See our Gemma 4 breakdown for how E4B compares.
• TTS: Qwen3-TTS for quality and voice control, or Kokoro-82M when you want TTS to cost basically nothing. Pocket TTS, ChatTTS, and MMS are also wired in via --tts.

No stage phones home. The only network traffic in a full-local run is your microphone audio crossing localhost.

---

Install and Run {#install}

Commands below are taken verbatim from the repo README. Install:

pip install speech-to-speech

Optional components ship as extras — for example:

pip install "speech-to-speech[kokoro]"
pip install "speech-to-speech[faster-whisper]"

Or from source:

git clone https://github.com/huggingface/speech-to-speech.git
cd speech-to-speech
uv sync

Heads up on the default: running bare speech-to-speech uses the OpenAI Responses API (default model gpt-5.4-mini), which needs OPENAI_API_KEY set. That's the cloud path. For local, point it at your own server.

Fully local with llama.cpp (the README's example, using the Gemma 4 E4B GGUF):

speech-to-speech \
    --model_name "ggml-org/gemma-4-E4B-it-GGUF" \
    --responses_api_base_url "http://127.0.0.1:8080/v1" \
    --responses_api_api_key ""

Fully local with vLLM:

speech-to-speech \
    --llm_backend chat-completions \
    --model_name "Qwen/Qwen3-4B-Instruct-2507" \
    --responses_api_base_url "http://localhost:8000/v1"

On a Mac:

speech-to-speech --local_mac_optimal_settings

Add --model_name "mlx-community/Qwen3-4B-Instruct-2507-bf16" to pick a specific MLX model, and use the [whisper-mlx] / [mlx-lm] extras plus --llm_backend mlx-lm for the Apple-Silicon-native path.

Serving other devices: run speech-to-speech --mode websocket --ws_host 0.0.0.0 --ws_port 8765 for the Realtime-compatible WebSocket, or --mode socket --recv_host 0.0.0.0 --send_host 0.0.0.0 with python scripts/listen_and_play.py --host <IP address of your server> as the thin client.

Component selection is one flag each: --stt parakeet-tdt (or whisper, faster-whisper, whisper-mlx, paraformer) and --tts qwen3 (or kokoro, pocket, chattts, facebook-mms).

What about Ollama? Not documented in the README — llama.cpp and vLLM are the supported local backends. But the chat-completions backend accepts any OpenAI-compatible base URL, and Ollama serves one, so pointing --responses_api_base_url at your Ollama /v1 endpoint is the obvious unofficial route. If it misbehaves, fall back to the documented llama.cpp example.

---

Hardware: What You Need {#hardware}

Honest version: the LLM is the whole hardware question. Parakeet is a 600M-parameter model. Qwen3-TTS starts at 0.6B. Kokoro is 82M and CPU-friendly. The speech stages fit on almost anything; your LLM choice sets the bill.

Gemma 4 31B — the model the hosted demo runs on Cerebras — needs a big GPU to run locally. Most readers should not try; E4B exists precisely for this. We're deliberately not quoting tokens-per-second figures here because they depend entirely on which LLM and quant you pick.

---

Component Deep-Dive {#components}

Parakeet TDT 0.6B v3 — the ASR

NVIDIA's 600M-parameter transcription model, and the reason this pipeline feels fast. Facts: 25 European languages, automatic language detection, ~6.34% average word error rate, and the highest throughput among multilingual models on the HF Open ASR Leaderboard. If your users speak a language outside those 25, swap in Whisper or Faster-Whisper with one flag.

Qwen3-TTS — the voice

Released January 22, 2026 under Apache 2.0, in 0.6B and 1.7B sizes on the 12Hz tokenizer. Three variants, and the naming is literal: Base does voice cloning, CustomVoice ships preset speakers, VoiceDesign builds a voice from a text instruction. For where it sits against everything else, see our local TTS model roundup and our Chatterbox Multilingual v3 writeup.

Kokoro-82M — the fallback

82M parameters, 6 languages, runs on CPU. This is the "my GPU is fully occupied by the LLM" option, and for a lot of assistant builds it's all you need. Setup guide: Kokoro TTS local install.

Worth watching: Nemotron 3.5 ASR Streaming 0.6B

Announced June 4, 2026: 40 language-locales in one 600M cache-aware streaming model, with configurable chunk sizes from 80 to 1120ms. NVIDIA's own benchmarks claim sub-100ms end-of-utterance latency and roughly 17x more concurrent streams — vendor numbers, so treat them as a ceiling, not a promise. It's not the pipeline default today, but it's the obvious candidate for true streaming ASR in this slot.

---

vs Whisper + Ollama + Piper {#vs-old-stack}

Our 2025 build glued faster-whisper, Ollama, and Piper together with hand-written Python. It works, and if yours is running, nothing forces a migration. Here's what actually changed:

The real upgrade isn't any single model — it's that component choice became a flag instead of a refactor.

---

Verdict {#verdict}

If you're building a local voice assistant in 2026, start from pip install speech-to-speech, not from scratch. The pipeline is open, deployed at real scale, and designed for the local swap: Parakeet for ears, your llama.cpp or vLLM model for the brain, Qwen3-TTS or Kokoro for the mouth. The one honest caveat: the out-of-the-box default is a cloud LLM, so the local build is a deliberate three-flag choice — copy the llama.cpp example above and you're offline. Budget your hardware around the LLM; the speech models are a rounding error.

---

Sources {#sources}

• Hugging Face + Cerebras: the voice-agent pipeline announcement (July 1, 2026)
• huggingface/speech-to-speech on GitHub — README (v0.2.10), source of all commands quoted above

---

FAQ {#faq}