Benchmark report
Voice Code Bench: Exact Tokens in Speech-to-Text
A report on how speech-to-text systems handle exact structured values in English workplace speech, from emails and URLs to command-line flags, file paths, account identifiers, dates, and measurements.
Problem
Structured tokens are easy to corrupt.
WER measures transcript similarity. Production voice interfaces need something stricter: the exact email, command, account identifier, URL, date, file path, dollar amount, product code, or measurement a downstream system must act on.
WER
WER does not measure exact-token success.
Traditional WER treats all words as part of the same transcript surface. Production systems do not. A single wrong character in an email, CLI flag, file path, URL, account identifier, product code, or measurement can route the wrong ticket, fail an account lookup, or trigger the wrong command.
The fraction of target entities whose canonical value is recoverable from the ASR transcript.
A recording succeeds only when every target entity in that recording is recovered correctly.
A supporting transcript-quality diagnostic, not the primary benchmark score.
Benchmark scope
What Voice Code Bench measures
Voice Code Bench is intentionally narrower than universal audio suites. It asks one product-critical question: did the transcript preserve every structured entity a downstream workflow needs?
human-recorded English WAV segments
34.9-122.9 sec clips
26 structured entity types
workplace workflow domains
Results
Main findings
We evaluated 18 STT systems and rank them by CTEM. OpenAI GPT Live Transcribe leads CTEM. Deepgram Nova-3 and OpenAI GPT Live Transcribe tie for the highest TSR, while AssemblyAI Universal-3 Pro has the lowest WER. Best-in-column highlights show the strongest result for each metric. That is the benchmark's core point: entity-level recovery, complete task success, and general transcript quality are related but not interchangeable.
- 01
OpenAI GPT Live Transcribe leads CTEM, showing that entity-level recovery and task-level success can disagree.
- 02
Deepgram Nova-3 and OpenAI GPT Live Transcribe tie for the highest TSR; AssemblyAI Universal-3 Pro has the lowest WER.
- 03
Commands, file paths, URLs, and environment variables expose large gaps across systems.
| Rank | Model | CTEM | TSR | WER |
|---|---|---|---|---|
| #1 | OpenAI GPT Live Transcribe (Streaming)Highest CTEM and joint-highest TSR in the released baseline suite, with 91.8% CTEM, 68.7% TSR, and 5.9% WER. | 91.8% | 68.7% | 5.9% |
| #2 | ElevenLabs Scribe v2Strong batch baseline with 91.6% CTEM, 67.7% TSR, and broad entity recovery. | 91.6% | 67.7% | 6.9% |
| #3 | Deepgram Nova-3Strong CTEM and joint-highest TSR in the released baseline suite, with 90.9% CTEM, 68.7% TSR, and 8.2% WER. | 90.9% | 68.7% | 8.2% |
| #4 | Cartesia Ink 2 (Streaming)Streaming-only baseline with 90.1% CTEM, 63.7% TSR, and 9.3% WER. | 90.1% | 63.7% | 9.3% |
| #5 | OpenAI GPT TranscribeBatch OpenAI baseline with 89.3% CTEM, 61.0% TSR, and 6.1% WER. | 89.3% | 61.0% | 6.1% |
| #6 | Deepgram Nova-3 (Streaming)Strong streaming baseline with 88.9% CTEM, 61.7% TSR, and 9.3% WER. | 88.9% | 61.7% | 9.3% |
| #7 | Google Cloud Chirp 3Strong email recovery and competitive TSR, but WER trails the top systems in this release. | 88.8% | 60.3% | 5.8% |
| #8 | Whisper Large-v3Open-source baseline with competitive TSR but weaker URL and command recovery. | 87.6% | 54.3% | 5.8% |
| #8 | Meta Muse Voice Transcribe 1.0Batch Meta baseline with 87.6% CTEM, 55.3% TSR, and 7.5% WER. | 87.6% | 55.3% | 7.5% |
| #10 | Meta Muse Voice Transcribe 1.0 (Streaming)Streaming Meta baseline with 86.8% CTEM, 55.7% TSR, and 7.4% WER. | 86.8% | 55.7% | 7.4% |
| #11 | Google Cloud Chirp 3 (Streaming)Streaming Google baseline with strong phone and IP recovery but lower overall TSR than batch Chirp 3. | 86.1% | 50.3% | 6.7% |
| #12 | AssemblyAI Universal-3 ProBatch AssemblyAI baseline with strong phone-extension and port-number recovery, but lower TSR than the top hosted systems. | 84.5% | 50.3% | 4.3% |
| #13 | Thinking Machines Lab Inkling-NVFP4Open-source batch baseline with 84.3% CTEM, 49.7% TSR, and 4.9% WER. | 84.3% | 49.7% | 4.9% |
| #14 | ElevenLabs Scribe v2 Realtime (Streaming)Realtime ElevenLabs baseline with strong person/team, version, and domain-term recovery but lower TSR than batch Scribe v2. | 84.0% | 46.3% | 6.8% |
| #15 | Inworld STT 1Batch Inworld baseline with 83.7% CTEM, 44.0% TSR, and 9.7% WER. | 83.7% | 44.0% | 9.7% |
| #16 | NVIDIA Parakeet TDT 0.6B v3Modal-hosted Parakeet baseline with 78.6% CTEM, 37.3% TSR, and 9.3% WER. | 78.6% | 37.3% | 9.3% |
| #17 | AssemblyAI Universal-3 Pro Realtime (Streaming)Streaming AssemblyAI baseline with 78.3% CTEM, 33.0% TSR, and 6.1% WER. | 78.3% | 33.0% | 6.1% |
| #18 | Amazon Transcribe (Streaming)Streaming Amazon baseline with 75.2% CTEM, 33.7% TSR, and 10.3% WER. | 75.2% | 33.7% | 10.3% |
Entity heatmap
Where systems separate by entity type
Each cell is entity-level CTEM: exact recovery for that structured-value family. The color is relative within each column, making it possible to compare the systems on the tokens a particular product depends on.
CTEM vs WER
Entity recovery and word error rate are not interchangeable
WER stays visible as a transcript-quality diagnostic. CTEM directly measures how often the canonical entities required by downstream workflows survive.
91.8% canonical entity match
0.0432 word error rate
The lowest-WER model trails the top entity-recovery model by this many percentage points on canonical entity match.
Metrics
How to read CTEM, TSR, and WER
CTEM shows the global exact-match rate across entities. TSR is stricter at the recording level because one missed entity can invalidate the task, while WER remains a supporting diagnostic for transcript quality.
- Low WER does not guarantee usable transcripts when the application depends on exact structured values.
- OpenAI GPT Live Transcribe has the highest CTEM; it and Deepgram Nova-3 tie for the highest TSR.
- Thinking Machines Inkling reaches 84.3% CTEM and 49.7% TSR as an open-source batch baseline.
- Commands, file paths, URLs, and environment variables expose large gaps across systems.
Dataset composition
Recording and entity distribution
Voice Code Bench is a test-only evaluation dataset, not a training corpus. The release covers 49 scenarios across four difficulty levels.
Entity Coverage
Domain Coverage
Duration Buckets
Recording Length Summary
Speaker metadata
Who is represented in the benchmark
Speaker ID, sex, accent, and age-bucket metadata are available for all 300 recordings. The current release includes 85 anonymized speaker IDs.
Audio quality
Audio-quality measurements for every recording
Signal-to-noise ratio, background-noise level, speech level, loudness, and click/pop metadata are available for all 300 recordings.
Quality Field Summary
Mean and median values across all recordings.
Metadata Summary
SNR Buckets
Scoring method
From raw audio to an auditable benchmark score
Audio to transcript
The provider receives no gold transcript or benchmark entity values.
Transcript to entities
The released scorer uses an LLM-assisted recoverability verifier and stores evidence and reasons.
Entities to benchmark
CTEM, TSR, entity CTEM, WER, score files, and verifier-provenance outputs are generated.
Each item includes template, acoustic, and canonical transcript layers.
Systems receive only the audio file in the main setting, without benchmark prompts, entity lists, domain labels, custom vocabulary, grammar constraints, or post-ASR correction.
Scoring asks whether the written value a downstream application needs is recoverable from the ASR transcript.
Entity scoring uses an LLM-assisted recoverability verifier with versioned prompts, response schemas, evidence, reasons, and a released audit sample.
VoiceCodeBench is intended for diagnostic ASR evaluation, provider comparison, regression tracking, and per-entity risk analysis.
The release is not intended for training, fine-tuning, speaker identification, biometric modeling, voice cloning, or demographic profiling.
Reproducibility
Files, baseline IDs, and commands
The release includes the artifacts needed to inspect the evaluation and reproduce its checked-in results.
Repository layout
- data/audio/001.wav through data/audio/300.wav
- data/metadata.json
- baselines/predictions/*.json
- baselines/results.csv
- audit/verifier_audit_samples.csv
- scripts/voice_code_bench/
- paper/voice-code-bench.pdf
- DATASET_CARD.md
Baseline IDs
- openai_gpt_live_transcribe_streaming
- elevenlabs_scribe_v2
- deepgram_nova3
- cartesia_ink_2_streaming
- openai_gpt_transcribe
- deepgram_nova3_streaming
- google_cloud_chirp_3
- whisper_large_v3
- meta_muse_voice_transcribe_1_0
- meta_muse_voice_transcribe_1_0_streaming
- google_cloud_chirp_3_streaming
- assemblyai_universal_3_pro
- modal_inkling
- elevenlabs_scribe_v2_realtime_streaming
- inworld_stt_1
- modal_nvidia_parakeet_tdt_0_6b_v3
- assemblyai_universal_3_pro_streaming
- amazon_transcribe_streaming
Commands
python -m pip install -e ../scripts/reproduce_release.shpython -m pytestvcb-run --stt-mode all --output-dir runs/full-localPaper
Technical paper for VoiceCodeBench, including the benchmark motivation, scoring protocol, baseline setup, and release details.
Context
Why this benchmark exists
Broad voice benchmarks measure general capability. Voice Code Bench is narrower on purpose: it isolates exact structured-token recovery, the failure mode that matters when speech becomes software input.
Name the task, source dataset, split, and metric explicitly.
Voice Code Bench exposes its 26 entity types and scoring rules up front, so the evaluation target is clear before the results table.Breadth creates authority but makes one-score summaries fragile.
Voice Code Bench is intentionally narrow: it measures exact structured-token recovery rather than claiming broad audio-language coverage.Instruction-following audio systems require hybrid scoring.
The LLM verifier is limited to entity-presence judgments, while canonical targets, evidence, and review files remain auditable.Fairness, toxicity, safety, and robustness belong inside the benchmark frame.
Voice Code Bench surfaces speaker metadata completeness and known imbalance as benchmark context, not appendix material.Good documentation is decision support, not decoration.
The benchmark page puts scale, provenance, schema, metadata, limitations, and prohibited uses where practitioners can evaluate fit quickly.Failure types
Common transcription failures
The purpose is not only to rank STT providers. It is to make the recurring failure patterns visible enough for product and research teams to fix.
Dataset documentation
Dataset documentation requirements
Good benchmark documentation is not decorative. It should make scale, schema, metadata, risks, and scoring rules obvious before someone downloads the data or cites the result.
Use cases
Where exact token recovery matters
Voice coding, support operations, CRM dictation, technical education, finance, healthcare, and legal workflows all depend on exact structured speech.
- 01ASR vendor selection
- 02Voice coding
- 03Support QA
- 04CRM and form dictation
- 05Technical troubleshooting
- 06Agentic voice workflows
Limitations
Current limitations
- Current release is English-only and focused on compact workplace-style dictation.
- It does not cover meetings, overlapping speech, casual conversation, broadcast audio, voice search, noisy field recordings, telephony codecs, or long-form dictation.
- Synthetic content may not capture every distributional property of real production workflows.
- The LLM-assisted verifier introduces dependence on a verifier model and prompt.
- Commercial ASR rankings are tied to the evaluation dates and provider behavior recorded in the baseline artifacts.
Roadmap
Planned extensions
The benchmark should grow in entity types, model coverage, speaker/environment diversity, and auditable slice reporting while preserving the exact-token focus.
- 01
Versioned releases that preserve prior dataset revisions
- 02
Multilingual structured-token recovery
- 03
Noisier environments and broader accent coverage
- 04
Broader conversation formats beyond compact dictation
- 05
Documented changes to entity taxonomy, scoring policy, verifier versions, and baseline systems
- 06
Human verifier calibration and expanded audit samples
Next
Dataset and paper
The dataset, baseline outputs, evaluation harness, and documentation are available on Hugging Face. The paper records the motivation, benchmark design, scoring protocol, baseline setup, and release results.