Run Whisper Locally: Free Offline Speech-to-Text Setup

Published April 10, 2026 • Updated August 23, 2026 • 16 min read

Quick answer: Install faster-whisper (pip install faster-whisper) plus ffmpeg, then transcribe with large-v3 for accuracy or small for speed. Whisper is MIT-licensed and runs fully offline, so there is no API key, no per-minute billing, and no audio leaving your machine. OpenAI's own model card puts large-v3 at roughly 10 GB of VRAM in FP16, or about half that at INT8 — and with no GPU at all, whisper.cpp runs tiny or base on CPU.

OpenAI released Whisper as an open-source speech recognition model in September 2022 and it is still the default general-purpose speech-to-text system you can run yourself. The weights are public, the licence is permissive, and every mainstream runtime — PyTorch, C++, CTranslate2 — has a mature implementation.

This guide covers three installation routes (original Whisper, whisper.cpp, faster-whisper), how to size a model against your hardware, and workflows for batch transcription and real-time dictation.

A note on numbers: every speed and VRAM figure below is either quoted from an official model card or repository, or worked out from stated arithmetic with the formula shown. Where a number would have to come from a specific machine, this guide gives you the command to measure it on your machine instead of quoting someone else's GPU.

---

What you will learn:

• Which Whisper model size matches your hardware, with the sizing arithmetic
• Three installation methods and when each one is the right call
• Batch transcription of audio files and folders
• Real-time microphone transcription setup
• Integration with Ollama for transcribe-then-summarize pipelines
• Privacy advantages over cloud transcription services

If you are setting up Whisper on a Mac specifically, start with the Mac local AI setup guide for Apple Silicon optimization, then come back here for Whisper-specific configuration.

Table of Contents

1. What Is Whisper
2. Which Whisper model fits my hardware?
3. Method 1: Original OpenAI Whisper
4. Method 2: whisper.cpp (CPU Optimized)
5. Method 3: faster-whisper (Recommended)
6. How fast will Whisper be on my machine?
7. Real-Time Transcription
8. Batch Processing Workflows
9. Which languages does Whisper handle well?
10. How accurate is Whisper?
11. Integration with Ollama
12. Privacy Advantages

---

What Is Whisper {#what-is-whisper}

Whisper is an automatic speech recognition (ASR) model trained on 680,000 hours of multilingual audio data scraped from the web. OpenAI released it under the MIT license, meaning you can use it for anything, commercial included, with zero restrictions.

The model uses an encoder-decoder Transformer architecture. Audio goes in as mel spectrograms (80-channel log-mel features computed from 16kHz audio), and text comes out as tokens. It handles transcription (same language), translation (any language to English), and language detection in a single model.

What makes Whisper useful is not any single capability but the combination: robustness to background noise, accents, technical jargon, and multiple speakers, all in one model that also does translation and language ID. The original Whisper repository remains the canonical reference implementation, and OpenAI published the training details, evaluation methodology and per-language results in the Whisper paper (arXiv:2212.04356).

Key specifications:

• Architecture: Encoder-decoder Transformer
• Training data: 680,000 hours of labeled audio
• Languages: 100+ languages for transcription, any-to-English translation
• License: MIT (fully permissive, commercial use allowed)
• Latest version: large-v3 (released November 2023, still state-of-the-art for general use)

---

Which Whisper model fits my hardware? {#model-sizes}

Whisper ships in six sizes. Match the model to your VRAM first, then worry about accuracy.

Where each column comes from — check the working yourself:

• Parameters, FP16 VRAM and relative speed are the figures published in the openai/whisper README model table. Relative speed is measured against large as 1x, on OpenAI's hardware, so treat it as a ratio between models rather than a promise about yours.
• INT8 VRAM is arithmetic, not a measurement: INT8 stores one byte per weight where FP16 stores two, so the weights halve. Runtime buffers do not halve, which is why real usage lands slightly above half.
• ggml file size is also arithmetic: parameters x 2 bytes for an FP16 ggml conversion. 1,550M x 2 = 3.1 GB for large-v3; 39M x 2 = 78 MB for tiny. Downloaded files come out a few percent under this because not every tensor is stored at full width.

Sizing rule of thumb

The formula that matters when you are staring at a GPU spec sheet:

weights (GB) = parameters (billions) x bytes-per-weight
  FP16  -> 2 bytes    INT8 -> 1 byte
headroom -> add ~1-2 GB for activations, KV state and the audio front-end

So large-v3 at INT8 is 1.55 x 1 = 1.55 GB of weights, and the ~5 GB figure above is that plus decoder and beam-search overhead. This is why an 8 GB card runs large-v3 at INT8 without drama while a 4 GB card does not.

Which model should you use?

No GPU, or integrated graphics only: tiny or base with whisper.cpp. CPU inference stays usable up to small and gets slow beyond it.

4-6 GB VRAM (GTX 1070, RTX 3060, M1 8 GB): small or medium with faster-whisper INT8. Small is the pragmatic default — OpenAI's own relative-speed column puts it around 4x large, and the accuracy gap only really opens up on hard audio.

8-12 GB VRAM (RTX 3070, RTX 4070, M2 16 GB): large-v3 at INT8 fits comfortably. This is where the accuracy jump on noisy and accented audio starts to pay for itself.

16 GB+ VRAM (RTX 3090, RTX 4090, M3 Pro 36 GB): large-v3 at FP16, no quantization needed. See the hardware requirements guide for full VRAM tables across GPU models.

Not sure your card qualifies? turbo is the interesting middle option: it is a large-v3 derivative with a pruned decoder, so it sits at 809M parameters and ~6 GB FP16 while OpenAI's table rates it ~8x large's speed.

---

Method 1: Original OpenAI Whisper {#method-original}

The reference implementation. Use this if you want the canonical experience or need to modify the model code.

Installation

# Create a virtual environment (recommended)
python3 -m venv whisper-env
source whisper-env/bin/activate

# Install Whisper
pip install openai-whisper

# Install ffmpeg (required for audio processing)
# Ubuntu/Debian:
sudo apt install ffmpeg
# macOS:
brew install ffmpeg
# Windows:
choco install ffmpeg

Basic Usage

# Transcribe a file
whisper audio.mp3 --model small --language en

# Transcribe with translation to English
whisper japanese_meeting.mp3 --model medium --task translate

# Output specific format
whisper lecture.wav --model large-v3 --output_format srt

# Specify output directory
whisper interview.mp3 --model small --output_dir ./transcripts

Output Formats

Whisper generates multiple output files by default:

• .txt — Plain text transcript
• .vtt — WebVTT subtitles (for web video)
• .srt — SubRip subtitles (for most video players)
• .tsv — Tab-separated with timestamps
• .json — Full output with word-level timing

When to actually use it

The reference implementation is the slowest of the three and the heaviest on VRAM, because it runs the model through stock PyTorch with no inference-specific optimisation. Use it when you need to modify the model code, reproduce a paper result, or debug something that behaves differently in an optimised runtime. For everyday transcription, skip to faster-whisper below.

---

Method 2: whisper.cpp (CPU Optimized) {#method-whisper-cpp}

whisper.cpp is a C/C++ port by Georgi Gerganov (the creator of llama.cpp). It runs on pure CPU with SIMD optimizations, making it the best choice for machines without a dedicated GPU. It also supports Metal acceleration on Apple Silicon.

Installation

# Clone the repository
git clone https://github.com/ggerganov/whisper.cpp.git
cd whisper.cpp

# Build with optimizations
# For x86 Linux/Windows:
make -j$(nproc)

# For Apple Silicon Mac (Metal acceleration):
make -j$(sysctl -n hw.ncpu) WHISPER_METAL=1

# For NVIDIA GPU (CUDA):
make -j$(nproc) WHISPER_CUDA=1

# Download a model
bash models/download-ggml-model.sh large-v3

Usage

# Basic transcription
./main -m models/ggml-large-v3.bin -f audio.wav

# With language detection
./main -m models/ggml-large-v3.bin -f audio.wav -l auto

# Output SRT subtitles
./main -m models/ggml-large-v3.bin -f audio.wav --output-srt

# Use 8 threads (match your CPU core count)
./main -m models/ggml-large-v3.bin -f audio.wav -t 8

# Convert audio to required format first (16kHz WAV)
ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav

Why whisper.cpp on a Mac

whisper.cpp is the only one of the three routes with no Python runtime and no CUDA dependency, which is what makes it the practical choice on machines without an NVIDIA GPU. On Apple Silicon it has two acceleration paths documented in the project README: Metal for the GPU, and an optional Core ML encoder that offloads the heaviest part of the model to the Neural Engine. The Core ML path needs a one-time model conversion step, which the README walks through.

The general shape: on a Mac, an accelerated whisper.cpp build finishes meaningfully sooner than the same build restricted to CPU threads. How much sooner depends on your chip tier, the model size, and how many performance cores you have — measure it with the command in the next section rather than trusting anyone's number, including ours.

---

Method 3: faster-whisper (Recommended) {#method-faster-whisper}

faster-whisper reimplements Whisper on top of CTranslate2, an inference engine built specifically for Transformer models. SYSTRAN's project README states it is "up to 4 times faster than openai/whisper for the same accuracy while using less memory", with further gains available from INT8 quantization on both CPU and GPU. That is the maintainers' claim about their own project — it is the right starting expectation, not a guarantee for your file and your card.

This is the route to default to. The dedicated faster-whisper setup guide goes deeper on GPU/CPU tuning, quantization options, and running it as a server.

Installation

# Create virtual environment
python3 -m venv faster-whisper-env
source faster-whisper-env/bin/activate

# Install faster-whisper
pip install faster-whisper

# For CUDA GPU acceleration (requires CUDA 12+)
pip install faster-whisper[cuda]

Basic Usage

from faster_whisper import WhisperModel

# Load model (auto-detects GPU)
# Options: "tiny", "base", "small", "medium", "large-v3"
model = WhisperModel("large-v3", device="cuda", compute_type="int8")

# Transcribe
segments, info = model.transcribe("meeting.mp3", beam_size=5)

print(f"Detected language: {info.language} (probability: {info.language_probability:.2f})")

for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

CLI Wrapper Script

#!/usr/bin/env python3
"""Fast local transcription with faster-whisper."""
import sys
import argparse
from faster_whisper import WhisperModel

def transcribe(audio_path, model_size="large-v3", language=None, output_format="txt"):
    model = WhisperModel(model_size, device="auto", compute_type="int8")

    segments, info = model.transcribe(
        audio_path,
        beam_size=5,
        language=language,
        vad_filter=True,          # Skip silence (huge speedup)
        vad_parameters=dict(
            min_silence_duration_ms=500,
            speech_pad_ms=200
        )
    )

    print(f"Language: {info.language} ({info.language_probability:.0%})")

    if output_format == "srt":
        for i, seg in enumerate(segments, 1):
            start = format_timestamp(seg.start)
            end = format_timestamp(seg.end)
            print(f"{i}")
            print(f"{start} --> {end}")
            print(f"{seg.text.strip()}\n")
    else:
        for seg in segments:
            print(seg.text.strip())

def format_timestamp(seconds):
    h = int(seconds // 3600)
    m = int((seconds % 3600) // 60)
    s = int(seconds % 60)
    ms = int((seconds % 1) * 1000)
    return f"{h:02d}:{m:02d}:{s:02d},{ms:03d}"

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("audio", help="Path to audio file")
    parser.add_argument("--model", default="large-v3", help="Model size")
    parser.add_argument("--language", default=None, help="Language code (e.g., en, ja, de)")
    parser.add_argument("--format", default="txt", choices=["txt", "srt"])
    args = parser.parse_args()
    transcribe(args.audio, args.model, args.language, args.format)

Save as transcribe.py and use:

python transcribe.py meeting.mp3 --model large-v3 --format srt > meeting.srt

The two settings that change speed most

compute_type. int8 halves the weight memory versus float16 (one byte per weight instead of two) and is usually the single biggest lever on a memory-constrained card. int8_float16 is a middle ground on GPUs; int8 is the right default on CPU.

vad_filter=True. Voice Activity Detection drops silent regions before they reach the model, so wall-clock time tracks speech duration rather than file duration. The arithmetic is direct: if a 60-minute recording is 20 minutes of silence, VAD removes a third of the work. That is why the win is enormous on meeting recordings with long pauses and near-zero on a dense audiobook.

It also fixes a real correctness bug, not just speed — see the hallucination note in troubleshooting below.

---

How fast will Whisper be on my machine? {#how-fast}

Any published tok/s or minutes-per-hour figure is a statement about somebody else's CPU, GPU, thermal headroom and audio file. The useful number is the one from your box, and it takes about a minute to get.

The metric to use is the real-time factor (RTF):

RTF = audio duration / processing time

RTF = 1   -> transcribes exactly as fast as the audio plays
RTF = 10  -> a 60-minute file finishes in 6 minutes
RTF < 1   -> slower than real time; live dictation is off the table

Anything above RTF 1 can, in principle, keep up with a live microphone. Comfortable live dictation wants headroom — roughly RTF 3 or better — because the model also has to re-process overlapping context windows.

Measure it in one command

#!/usr/bin/env python3
"""bench.py - measure the real-time factor on YOUR hardware."""
import sys, time
from faster_whisper import WhisperModel

audio = sys.argv[1]
size = sys.argv[2] if len(sys.argv) > 2 else "small"
compute = sys.argv[3] if len(sys.argv) > 3 else "int8"

model = WhisperModel(size, device="auto", compute_type=compute)

start = time.perf_counter()
segments, info = model.transcribe(audio, beam_size=5, vad_filter=True)
segments = list(segments)          # generator is lazy; force the work
elapsed = time.perf_counter() - start

print(f"model={size} compute={compute}")
print(f"audio={info.duration:.1f}s  wall={elapsed:.1f}s  RTF={info.duration / elapsed:.1f}x")

python bench.py sample.mp3 small int8
python bench.py sample.mp3 large-v3 int8

Run it on a five-minute clip of the kind of audio you actually process, at two model sizes, and you will know more about your own setup than any benchmark table can tell you. Watch nvidia-smi (or Activity Monitor on a Mac) alongside it for the real VRAM figure.

Two things that will skew the first run: the model downloads on first use, and CUDA kernels compile on first load. Discard run one and time run two.

---

Real-Time Transcription {#real-time}

Real-time transcription captures audio from your microphone and produces text as you speak. The requirement is simple: the model must clear RTF 1 on your machine, with headroom. Run bench.py from the previous section before you write any of this code — if your chosen model does not beat real time on a recorded clip, it will not beat a live microphone either.

The practical ladder, smallest first: tiny and base clear real time on modest CPUs, small typically needs a dedicated GPU or Apple Silicon, and large-v3 needs a mid-range or better discrete GPU. Where the cutoff falls for your hardware is a measurement, not a spec — which is exactly what the benchmark script tells you.

If dictation rather than transcription is the goal, the local voice assistant guide (Whisper + Ollama + Piper) wires the same pipeline into a full speech-in, speech-out loop.

Setup with faster-whisper

#!/usr/bin/env python3
"""Real-time speech-to-text with faster-whisper."""
import numpy as np
import sounddevice as sd
from faster_whisper import WhisperModel
import queue
import threading

# Configuration
MODEL_SIZE = "small"       # Use "small" for balance of speed + accuracy
SAMPLE_RATE = 16000
CHUNK_DURATION = 3         # Process 3 seconds of audio at a time
SILENCE_THRESHOLD = 0.01

audio_queue = queue.Queue()
model = WhisperModel(MODEL_SIZE, device="auto", compute_type="int8")

def audio_callback(indata, frames, time_info, status):
    """Called for each audio chunk from the microphone."""
    audio_queue.put(indata.copy())

def process_audio():
    """Process queued audio chunks."""
    buffer = np.array([], dtype=np.float32)

    while True:
        chunk = audio_queue.get()
        audio_data = chunk.flatten().astype(np.float32)
        buffer = np.concatenate([buffer, audio_data])

        # Process when buffer has enough audio
        if len(buffer) >= SAMPLE_RATE * CHUNK_DURATION:
            # Check if there is actual speech
            if np.abs(buffer).mean() > SILENCE_THRESHOLD:
                segments, _ = model.transcribe(
                    buffer,
                    beam_size=1,          # Faster for real-time
                    language="en",
                    vad_filter=True
                )
                for seg in segments:
                    print(seg.text.strip(), end=" ", flush=True)
            buffer = np.array([], dtype=np.float32)

# Start real-time transcription
print("Listening... (Ctrl+C to stop)")
processor = threading.Thread(target=process_audio, daemon=True)
processor.start()

with sd.InputStream(samplerate=SAMPLE_RATE, channels=1, callback=audio_callback):
    try:
        while True:
            sd.sleep(100)
    except KeyboardInterrupt:
        print("\nStopped.")

Install dependencies:

pip install sounddevice numpy faster-whisper

Latency here is set by CHUNK_DURATION, not by the model: the script waits until it has that many seconds of audio buffered before it transcribes anything, so 3 seconds of chunk means at least 3 seconds before text appears, plus however long inference takes. Drop it to 1.5 seconds for a snappier feel at the cost of more fragmented output — Whisper is trained on 30-second windows and loses context when you feed it slivers.

---

Batch Processing Workflows {#batch-processing}

Transcribe an Entire Directory

#!/bin/bash
# batch_transcribe.sh - Transcribe all audio files in a directory
INPUT_DIR="$1"
OUTPUT_DIR="${2:-./transcripts}"
MODEL="${3:-large-v3}"

mkdir -p "$OUTPUT_DIR"

for file in "$INPUT_DIR"/*.{mp3,wav,m4a,flac,ogg,mp4,mkv,webm}; do
    [ -f "$file" ] || continue
    basename=$(basename "$file" | sed 's/\.[^.]*$//')
    echo "Transcribing: $file"

    python3 -c "
from faster_whisper import WhisperModel
model = WhisperModel('$MODEL', device='auto', compute_type='int8')
segments, info = model.transcribe('$file', beam_size=5, vad_filter=True)
with open('$OUTPUT_DIR/' + basename + '.txt', 'w') as f:
    for seg in segments:
        f.write(f'[{seg.start:.1f}s] {seg.text.strip()}\n')
print(f'  Language: {info.language}, Duration: {info.duration:.0f}s')
"
done
echo "All transcriptions saved to $OUTPUT_DIR"

Usage:

chmod +x batch_transcribe.sh
./batch_transcribe.sh ./recordings ./transcripts large-v3

Podcast Workflow

A three-step pipeline that takes an episode from URL to show notes without touching a cloud service. If subtitles rather than notes are the output you want, the local subtitle generation guide covers SRT timing and burn-in:

# Step 1: Download podcast episode
yt-dlp -x --audio-format mp3 "https://youtube.com/watch?v=EPISODE_ID" -o episode.mp3

# Step 2: Transcribe with faster-whisper
python3 -c "
from faster_whisper import WhisperModel
model = WhisperModel('large-v3', device='cuda', compute_type='int8')
segments, _ = model.transcribe('episode.mp3', beam_size=5, vad_filter=True)
with open('transcript.txt', 'w') as f:
    for seg in segments:
        mins = int(seg.start // 60)
        secs = int(seg.start % 60)
        f.write(f'[{mins:02d}:{secs:02d}] {seg.text.strip()}\n')
"

# Step 3: Generate summary with Ollama
ollama run llama3.2 "Summarize this podcast transcript into key points and timestamps:" < transcript.txt > summary.md

That last step is the real power move: Whisper produces the transcript, and a local LLM generates the summary. No cloud services involved. The entire pipeline runs offline.

---

Which languages does Whisper handle well? {#language-support}

Whisper supports 99 languages, and accuracy varies enormously between them. The determining factor is how many hours of that language were in the 680,000-hour training set — OpenAI is explicit about this in the paper, and publishes a per-language WER chart alongside the model.

Do not take a blog's word for your language. The authoritative source is the per-language breakdown in the Whisper paper and the language list in the repository README, which OpenAI orders by measured performance. Look yours up before you commit to a workflow — the spread between the best and worst supported languages is more than an order of magnitude in error rate.

The broad pattern is predictable: high-resource languages with large web audio footprints (Western European, plus Japanese, Korean and Mandarin) sit at the top; languages with limited training audio sit at the tail, and for those the gap between small and large-v3 is far wider than it is in English. If your target language is in the tail, always test large-v3 before concluding Whisper cannot do the job.

Language-Specific Tips

# Force a specific language (faster and more accurate than auto-detect)
segments, info = model.transcribe("audio.mp3", language="ja")

# Translate any language to English
segments, info = model.transcribe("german_lecture.mp3", task="translate")

# Initial prompt helps with domain-specific terms
segments, info = model.transcribe(
    "medical_recording.mp3",
    language="en",
    initial_prompt="This is a cardiology consultation discussing myocardial infarction, "
                   "troponin levels, and echocardiography results."
)

The initial_prompt trick is underrated. Whisper conditions on that text as if it were the preceding transcript, so seeding it with domain vocabulary — drug names, product names, the surnames of everyone in the meeting — measurably pulls recognition toward those spellings instead of phonetic guesses. Keep it short; it consumes context.

---

How accurate is Whisper? {#accuracy-benchmarks}

Word Error Rate (WER) is the standard metric: the proportion of words inserted, deleted or substituted versus a reference transcript. Lower is better, and 0% is not achievable on hard audio even for human transcriptionists.

Where to get current, independent numbers. The right source is not a blog post — it is the Open ASR Leaderboard on Hugging Face, which evaluates open speech models on the same fixed set of datasets with the same harness and is re-run as new models land. It reports WER and RTF side by side, so you can see the accuracy-versus-speed trade directly, and it includes the Whisper family alongside the models competing with it. A leaderboard that updates beats a table that was true once.

For Whisper specifically, OpenAI's own evaluation across LibriSpeech, Common Voice, TED-LIUM, Fleurs and more is in the paper, with the full per-dataset breakdown in the appendices.

What actually moves your error rate

Model size is only one of four variables, and often not the dominant one:

That last row is the practical test. Run both sizes over a representative five-minute clip, diff the two transcripts, and count the differences that would actually matter to you. If small and large-v3 disagree only on punctuation, you have your answer and you just saved most of your compute budget.

For how Whisper stacks up against the newer generation of open ASR models, see the Parakeet vs Whisper comparison.

---

Integration with Ollama {#ollama-integration}

The most powerful local AI workflow combines Whisper transcription with LLM processing. Transcribe audio locally, then use Ollama to summarize, extract action items, translate, or answer questions about the content.

Transcribe-and-Summarize Pipeline

#!/usr/bin/env python3
"""Transcribe audio and generate AI summary with Ollama."""
import sys
import requests
from faster_whisper import WhisperModel

def transcribe_and_summarize(audio_path):
    # Step 1: Transcribe
    print("Transcribing...")
    model = WhisperModel("large-v3", device="auto", compute_type="int8")
    segments, info = model.transcribe(audio_path, beam_size=5, vad_filter=True)

    transcript = ""
    for seg in segments:
        mins = int(seg.start // 60)
        secs = int(seg.start % 60)
        transcript += f"[{mins:02d}:{secs:02d}] {seg.text.strip()}\n"

    print(f"Transcribed {info.duration:.0f}s of {info.language} audio")

    # Step 2: Summarize with Ollama
    print("Generating summary...")
    prompt = f"""Analyze this transcript and provide:
1. A 3-sentence summary
2. Key topics discussed (bullet points)
3. Action items mentioned (if any)
4. Notable quotes

Transcript:
{transcript[:8000]}"""  # Trim to fit context window

    response = requests.post("http://localhost:11434/api/generate", json={
        "model": "llama3.2",
        "prompt": prompt,
        "stream": False
    })

    summary = response.json()["response"]

    # Save both outputs
    with open(audio_path.rsplit(".", 1)[0] + "_transcript.txt", "w") as f:
        f.write(transcript)
    with open(audio_path.rsplit(".", 1)[0] + "_summary.md", "w") as f:
        f.write(summary)

    print(f"\nSummary:\n{summary}")

if __name__ == "__main__":
    transcribe_and_summarize(sys.argv[1])

Total wall time is the sum of two stages you can size separately: transcription (measure it with bench.py above) plus summarization (roughly the LLM's token throughput divided into the transcript length). Both are one-off costs per recording, and both run unattended — which is why this belongs in a cron job rather than an interactive session. The local meeting transcription guide covers the end-to-end version with speaker labels. If you want it running on dedicated hardware instead of your workstation, the homelab AI server build guide walks through the build.

Meeting Minutes Automation

# Cron job: auto-transcribe any new files in ~/Recordings
# Add to crontab -e:
*/5 * * * * find ~/Recordings -name "*.mp3" -newer ~/Recordings/.last_processed -exec python3 ~/transcribe_summarize.py {} \; && touch ~/Recordings/.last_processed

---

Privacy Advantages {#privacy-advantages}

Cloud transcription services (Google Speech-to-Text, AWS Transcribe, AssemblyAI) send your audio to external servers. For many use cases, that is unacceptable.

Scenarios where local Whisper is mandatory:

• Legal: Attorney-client privileged conversations, depositions, court proceedings
• Medical: Patient consultations, therapy sessions (HIPAA compliance)
• Corporate: Board meetings, M&A discussions, proprietary strategy sessions
• Journalism: Source interviews, especially with whistleblowers
• Personal: Private conversations you simply do not want stored on someone else's servers

Local Whisper processes everything in your machine's RAM and VRAM. Audio files never leave your hardware. There is no telemetry, no logging, no data retention policy to worry about.

For a broader view of privacy implications, the run AI offline guide covers air-gapped setups where the machine has no internet connection at all.

---

Troubleshooting

Common Issues

"CUDA out of memory"

# Use a smaller model or INT8 quantization
model = WhisperModel("large-v3", device="cuda", compute_type="int8")
# Or fall back to CPU
model = WhisperModel("large-v3", device="cpu", compute_type="int8")

"No such file or directory: ffmpeg"

# Install ffmpeg
sudo apt install ffmpeg   # Ubuntu
brew install ffmpeg        # macOS

Hallucinated text during silence

This is Whisper's best-known failure mode and it is a property of the architecture, not a bug in your setup: the decoder is autoregressive and will happily produce plausible text when the encoder gives it nothing to work with. Typical symptoms are a repeated phrase looping for a minute, or a stray "Thanks for watching!" at the end of a file. Fix it by never letting silence reach the model:

segments, info = model.transcribe(
    "audio.mp3",
    vad_filter=True,
    vad_parameters=dict(min_silence_duration_ms=1000)
)

Slow performance on Mac

Make sure you built whisper.cpp with Metal support:

make clean && make -j$(sysctl -n hw.ncpu) WHISPER_METAL=1

---

Next Steps

You now have local speech-to-text running with full privacy. Here is where to go next:

1. Build an AI pipeline. Combine Whisper transcription with Ollama summarization for automated meeting notes, podcast show notes, or voice journaling. To close the loop into a talking assistant, our local voice assistant guide (Whisper + Ollama + Piper) wires speech-to-text, an LLM, and text-to-speech into one fully offline stack — the Piper TTS setup guide covers installing the speech-output half.

2. Scale up. If you are transcribing large volumes, consider a dedicated AI server that can process files around the clock without tying up your workstation.

3. Go fully offline. Follow the run AI offline guide for an air-gapped setup where both Whisper and your LLM run without any internet connection.

---

Frequently Asked Questions

Is Whisper really free to use commercially?

Yes. OpenAI released Whisper under the MIT license. You can use it in commercial products, modify the code, and redistribute it. There are no usage fees, API keys, or restrictions. The model weights are included.

How accurate is Whisper compared to Google Speech-to-Text?

Whisper large-v3 is competitive with the major cloud ASR services on clean English, and the fact that it is free and offline usually settles the decision before accuracy does. Rather than trust any single published comparison, check the Open ASR Leaderboard, which scores open models on identical datasets and updates continuously. For an even faster open alternative, see our Parakeet vs Whisper comparison — NVIDIA's Parakeet trades language coverage for speed.

Can Whisper transcribe in real-time from a microphone?

Yes, if your hardware clears a real-time factor above 1 for your chosen model. Tiny and base manage it on ordinary CPUs; small generally wants a dedicated GPU or Apple Silicon; large-v3 wants a mid-range or better discrete GPU. Do not guess — run the bench.py script in the speed section against a recorded clip first, because the exact cutoff depends on your CPU, GPU and audio. This guide includes a complete real-time transcription script.

Does Whisper work on Apple Silicon Macs?

Yes, and it is one of the better platforms for it. whisper.cpp ships both a Metal backend and an optional Core ML encoder path that uses the Neural Engine, both documented in the project README. faster-whisper also runs on Mac via CPU. Use bench.py above to find the largest model that clears real time on your specific chip.

What audio formats does Whisper support?

Whisper accepts any audio format that ffmpeg can read: MP3, WAV, M4A, FLAC, OGG, WMA, AAC, and video files (MP4, MKV, WebM, AVI). Audio is internally converted to 16kHz mono WAV. For best results, provide the highest quality source you have.

Can Whisper identify different speakers (speaker diarization)?

The base Whisper model does not perform speaker diarization. However, you can combine it with pyannote-audio for speaker identification. The pipeline runs locally: pyannote segments the audio by speaker, then Whisper transcribes each segment. This adds processing time but works well for meetings.

How much storage do Whisper models use?

An FP16 conversion is roughly parameters x 2 bytes, so: tiny ~78 MB, base ~148 MB, small ~488 MB, medium ~1.5 GB, large-v3 ~3.1 GB, turbo ~1.6 GB. Adding those up, the full set is about 6.9 GB, or about 5.3 GB if you skip turbo. Models download once and cache in ~/.cache/huggingface/ for faster-whisper, or the models/ directory for whisper.cpp. In practice most people keep two: one small model for speed and large-v3 for anything that matters.

Should I use large-v3 or turbo?

Turbo is a large-v3 derivative with a heavily pruned decoder — 809M parameters against 1,550M — which is why OpenAI's model table rates it around 8x large's relative speed at ~6 GB of FP16 VRAM. The encoder is unchanged, so acoustic robustness holds up; the compromise is in the decoder, which shows up most on translation and on the harder languages. Rule of thumb: turbo for English transcription where throughput matters, large-v3 when the transcript is going into something you cannot easily correct later.

---

Conclusion

Whisper is one of the few open models where the local version is genuinely the sensible default rather than a compromise. It is MIT-licensed, it runs on modest hardware, and it keeps your audio on your machine. faster-whisper with INT8 puts large-v3 inside an 8 GB card; whisper.cpp puts a usable model on hardware with no GPU at all.

For most people the answer is faster-whisper with small or large-v3, VAD enabled, and bench.py run once to confirm which of the two your machine actually wants. Pair it with Ollama for summarization and the whole pipeline — audio in, structured notes out — never touches a network.

The audio on your machine stays on your machine. That alone makes local Whisper worth setting up.

---

Want to build a complete local AI stack? Start with our hardware requirements guide to size your setup, then follow the Mac or Linux setup guide for your platform.