Open Source Speech to Text
Speech to text is the rare AI task where the demo lies: a model that nails a narrated podcast can fall apart on a noisy call, overlapping speakers, or a name it has never seen, and you only learn which by running your own audio through it. Accuracy is set by how well the model matches your acoustics, not by a headline benchmark. Open source engines add a second advantage that matters for recordings of calls, consultations, and interviews: the recognition runs on your own hardware, so confidential audio never leaves the machine, and you can transcribe an archive of thousands of hours without a per-minute meter running. What you trade for that is owning the tuning, the model files, and the throughput yourself.
Whisper
General-purpose speech recognition model for multilingual transcription, translation, and language identification
whisper.cpp
C/C++ speech to text inference for OpenAI's Whisper model with CPU, GPU, and on-device support
faster-whisper
Whisper transcription with CTranslate2 for faster inference and lower memory use
Vosk
Offline speech recognition toolkit with streaming transcription, small models, and speaker identification
SpeechBrain
Open-source PyTorch toolkit for speech and text processing across conversational AI tasks
Our picks
Pick by how you run inference: batch accuracy, on-device efficiency, live streaming, or custom training.
Best general-purpose baseline: Whisper Whisper is the model most projects benchmark against: one Transformer that handles multilingual transcription, speech translation, language identification, and voice activity detection. It ships several model sizes to trade accuracy for speed, installs as a Python package with a CLI, and releases both code and weights under MIT. Set your accuracy bar here, then optimize.
Best for CPU and on-device use: whisper.cpp whisper.cpp runs Whisper in plain C/C++ with no heavy dependencies, fully offline on device. It supports integer quantization, zero runtime allocations, and acceleration for Apple Silicon, CUDA, ROCm, and Vulkan, targeting macOS, Windows, Linux, Android, iOS, WebAssembly, and Raspberry Pi. The choice when you cannot assume a GPU.
Best for streaming and constrained devices: Vosk Vosk is built for offline recognition with a streaming API and zero-latency responses, suiting live captions, assistants, and subtitles. Its models are compact, around 50 MB, so it runs from a Raspberry Pi to a server cluster, covering 20-plus languages with bindings for Python, Java, Node.js, Go, and more.
Best for fine-tuning and research: SpeechBrain SpeechBrain is a PyTorch toolkit rather than a single model, with over 200 training recipes across recognition, speaker recognition, enhancement, and separation. Train from scratch or fine-tune pretrained models and plug in Hugging Face checkpoints when your audio needs domain adaptation. Apache-2.0, with a Python inference interface too.
Fitting a speech engine to your audio and hardware
Your audio decides more than the model does. A stack that scores well on clean, single-speaker recordings can still stumble on call-center lines, crosstalk in a meeting, or field audio full of background noise, so test with your own files before trusting anyone's word error rate. Decide too which outputs you actually need downstream: word timestamps, punctuation, speaker labels, confidence scores, or subtitle-ready segments. Whisper produces solid multilingual transcripts from the command line, while faster-whisper adds word timestamps and VAD filtering through CTranslate2, which matters when the transcript has to line up with video.
Match the architecture to your latency budget and your hardware. Batch jobs can load a large model and queue work; live captioning needs streaming, stable partial results, and predictable latency under load, which is where Vosk fits with its zero-latency streaming API and roughly 50 MB models that run on a Raspberry Pi. If you have no GPU, check CPU throughput honestly: whisper.cpp is built for exactly that, running Whisper in plain C/C++ with quantization and acceleration for Apple Silicon, CUDA, ROCm, and Vulkan, down to phones and WebAssembly.
Then plan for adaptation and keeping your corpus portable. Some projects are pure inference; others let you add custom vocabulary or fine-tune, which matters when your audio is full of product names, drug names, or acronyms. SpeechBrain sits at that end, a PyTorch toolkit with training recipes you can fine-tune on your own data. Whatever you pick, keep raw audio, model identifiers, and decoding settings together so you can reprocess files when a better model lands.
Related categories
Frequently asked questions
Do I need a GPU to run open source speech to text?+
It depends on volume and latency. CPU-only works for smaller models, offline queues, and low-volume jobs; whisper.cpp is designed for this, running in plain C/C++ with quantization and acceleration paths from Apple Silicon to Raspberry Pi. A GPU is often the difference between practical and painful throughput for long files or many concurrent jobs. Test real audio length, concurrency, and model size on your target hardware, and watch memory when several workers load the same model.
How accurate is open source speech to text compared with a hosted API?+
That is decided by your audio more than by the label. Clean, single-speaker recordings can be very competitive; noisy meetings, heavy accents, overlapping speakers, and specialized terms are where gaps appear. Run a blind test on representative files and score word error rate, punctuation, speaker attribution, and how long a human needs to correct the output. A transcript that is slightly less accurate but easier to fix can still be the better choice.
Can these tools tell speakers apart?+
Recognition and speaker diarization are separate problems. Vosk includes speaker identification, and SpeechBrain covers speaker recognition among its tasks, but many engines transcribe only and leave labeling to a second component. Diarization is hardest with overlapping speech, similar voices, and poor microphones. If speaker labels matter, test the whole pipeline rather than the recognizer alone, and confirm word timestamps survive after diarization and cleanup.
Can I add custom vocabulary for names and jargon?+
Sometimes, and the mechanism varies. Vosk supports a reconfigurable vocabulary, and SpeechBrain lets you fine-tune models on your own labeled audio, which is the stronger route for dense domain terms like drug or legal names. Simpler engines rely on phrase hints or a post-processing dictionary. Start with a small evaluation set, track which substitutions recur, and remember that text normalization fixes predictable spellings but not genuine acoustic confusion.
How is live captioning different from transcribing recordings?+
Live captioning needs low latency, incremental decoding, and partial text that does not rewrite itself too aggressively; it also has to survive network jitter and microphone dropouts. Vosk is built for this with a streaming API and zero-latency responses. Batch transcription can spend more time on context and punctuation because nobody is waiting on each word. If your use case is live captions, do not judge a tool only by how it handles uploaded files.
Which engine should I start with?+
For general transcription, Whisper is the common baseline, and whisper.cpp or faster-whisper give you the same model family with lower memory and better throughput on CPU or GPU. For streaming and constrained devices, Vosk is the natural pick. For research, custom training, or tasks beyond plain recognition, SpeechBrain is a full toolkit. Match the engine to whether you need batch accuracy, real-time captions, or the ability to fine-tune.
Is the model license the same as the code license?+
Not always, so check both before shipping. Whisper releases its code and weights under MIT, which is unusually clean for commercial use, but many models carry separate weight licenses or dataset restrictions even when the code is permissive. Vosk and SpeechBrain are Apache-2.0 on the code, with pretrained models published separately. Treat model licensing as its own approval step, and confirm attribution and redistribution terms rather than assuming the code license covers everything.