PrivASR is a private, self-hosted speech-to-text engine I built for teams that need to transcribe large volumes of audio without handing their recordings to a third party. At its core it runs Whisper (large-v3) served through CTranslate2, but the model on its own is only a starting point. The real work was everything I engineered around Whisper to make it accurate, fast, and cheap enough to run in production, and to do all of it inside a client’s own AWS account so the audio never leaves their environment.
The four stages an audio file passes through
The engine is a pipeline of four stages, and it is worth being precise about the order because each one exists to fix a specific weakness in the one after it.
Voice activity detection runs first, using a pyannote model that returns the spans of the recording that actually contain speech. Diarization comes second: a NeMo clustering diarizer consumes those speech spans directly rather than computing its own, which is what the oracle_vad setting does, so the two stages never disagree about where speech begins and ends. Transcription is third, faster-whisper with large-v3 on GPU. Finally the merge stage reconciles the two independent timelines the diarizer and Whisper produce.
Those four stages are packaged as two GPU workers rather than four services. Voice activity detection and diarization live together, because the second consumes the first’s output and splitting them would mean shipping speech segments over a queue for no benefit. Transcription is its own worker, because it is the expensive stage and the one whose capacity genuinely needs to scale independently.
Adding speaker diarization Whisper cannot do
Whisper transcribes words but has no notion of who is speaking, so on its own it produces an undifferentiated wall of text. I added diarization as a dedicated stage in the pipeline. A speaker-embedding model (NVIDIA NeMo’s TitaNet-L) converts short windows of audio into voice fingerprints, and a clustering step groups those fingerprints into distinct speakers, so the number of speakers does not need to be known in advance.
The load-bearing part is alignment. Whisper and the diarizer produce two independent timelines, and I merge them by maximum temporal overlap: each transcribed segment is assigned to the speaker whose turn covers the largest share of it. The rule is deliberately simple, which is the right property here, because the alternative is a heuristic that fails in unpredictable ways on crosstalk. Segments that no speaker turn overlaps fall through to an explicit UNKNOWN label rather than being silently attributed to whoever spoke last. The output is a transcript where every line carries a speaker and a timestamp.
Stopping Whisper hallucinating over silence
Whisper’s best-known failure mode is inventing text over silence and long non-speech gaps, and it is the single biggest source of garbage in a naive transcription pipeline. The engine runs Whisper with its built-in Silero voice-activity filter enabled and a 500 ms minimum silence threshold, so stretches of dead air are dropped before the decoder ever sees them. That is what removes the phantom phrases Whisper otherwise produces over silent regions, and it measurably cleans up the transcript on recordings with long pauses, which in practice means most real meetings and interviews.
Decoding runs with a beam size of 5 rather than greedy decoding. It costs a little throughput and buys back accuracy on exactly the audio where it matters, quiet speakers, accents, and overlapping turns.
What makes it cheap enough to run in production
Two things: what the model runs as, and what happens when nothing is running.
Compute type is configurable. The default is float16 on GPU, with int8 and int8_float16 available, so the same engine can trade a little accuracy for more throughput on cheaper hardware when a workload calls for it. Models are loaded once at worker startup and held resident, so per-file cost is inference only and not repeated model loading, which on large-v3 is not a trivial amount of time.
The larger lever is that most transcription workloads are bursty. Audio arrives in batches, then nothing arrives for hours. A GPU instance left running through those quiet hours costs roughly $0.526 an hour on a g4dn.xlarge and produces nothing. That idle cost, not the cost of inference, is what makes self-hosted ASR expensive in practice.
This is the change that turns Whisper from a research tool into something economical at scale. In practice it runs at roughly 10x faster than realtime and holds its cost under $0.05 per processed audio hour, against something closer to $1.44 an hour for a managed transcription service.
Scaling to zero, and the harder problem of scaling back up
Scaling a GPU fleet down to nothing is easy. Getting it back up is the hard part, and it is the piece I spent the most infrastructure work on.
AWS target-tracking autoscaling cannot scale a service from zero, because the metric it tracks is undefined when there are no tasks. So the 0 to 1 transition is handled by a dedicated Lambda that polls the queues once a minute and sets the ECS desired count to 1 the moment work appears. From 1 upward, ordinary target tracking takes over and scales on queue depth. Underneath, an ECS capacity provider watches for tasks it cannot place and grows an Auto Scaling Group whose minimum size is zero.
The result is a cold path with honest numbers attached. A GPU instance takes roughly 60 to 90 seconds to launch and join the cluster, so a job arriving into a completely cold system waits about two to three minutes before processing begins. Scale-down is deliberately slower, ten to fifteen minutes, because autoscaling cooldowns are tuned to avoid thrashing instances up and down on a stuttering queue. When every queue is empty, ECS tasks go to zero, the Auto Scaling Group goes to zero, and the compute bill for the pipeline is nothing at all.
That tradeoff is the right one for batch transcription and the wrong one for anything interactive, which is worth saying plainly. A workload that needs a transcript back in under a second keeps a warm instance and pays for it.
Integrating through infrastructure instead of an API
There is no API. That is a deliberate decision, not an omission.
The interface is the infrastructure itself: audio is uploaded to S3, a message goes onto a queue naming the input and output locations, and the result is read from the output path once it appears. A job is a small JSON document with a job_id, an input_s3_uri, an output_s3_uri, and options for language and whether diarization is wanted.
The result that lands in S3 is the full transcript, plus per-segment start, end, text and speaker, plus the audio duration and a timing breakdown for every stage: how long voice activity detection took, how long diarization took, how long transcription took, how much of the total was spent waiting in queues rather than computing.
Having no API layer means there is no service to authenticate against, no endpoint to expose, and nothing new to threat-model. It also means access control is plain S3 and SQS permissions, which is a thing every AWS team already knows how to reason about. When the goal is deploying into someone else’s account without becoming a permanent operational dependency, removing the API is the feature.
A small Streamlit app ships alongside it on Fargate as a reference implementation, talking to exactly the same S3 and SQS interface an external caller would use.
Measuring performance per stage
The primary metric is real-time factor: processing time divided by audio duration. Below 1.0 is faster than real time, 0.5 means an hour of audio in thirty minutes, 0.1 means an hour in six. RTF is published to CloudWatch per stage and per environment alongside raw processing time and audio duration, so a regression can be attributed to voice activity detection, diarization or transcription rather than to the pipeline as an undifferentiated whole. Errors go to Sentry.
The whole thing is provisioned with Terraform and deploys into a client’s account in essentially one step, which is what makes it a product rather than a bespoke engagement.
The transcripts this produces, speaker-attributed and timestamped, are what make downstream retrieval possible. That is a separate project: the Private Speech QA Agent builds a searchable, conversational layer on top of exactly this output.