Benchmark report
Voice Code Bench: Exact Tokens in Speech-to-Text
A report on how speech-to-text systems handle exact structured values in English workplace speech, from emails and URLs to command-line flags, file paths, account identifiers, dates, and measurements.
Problem
Structured tokens are easy to corrupt.
WER measures transcript similarity. Production voice interfaces need something stricter: the exact email, command, account identifier, URL, date, file path, dollar amount, product code, or measurement a downstream system must act on.
WER
WER does not measure exact-token success.
Traditional WER treats all words as part of the same transcript surface. Production systems do not. A single wrong character in an email, CLI flag, file path, URL, account identifier, product code, or measurement can route the wrong ticket, fail an account lookup, or trigger the wrong command.
A recording succeeds only when every target entity in that recording is recovered correctly.
The fraction of target entities whose canonical value is recoverable from the ASR transcript.
A supporting transcript-quality diagnostic, not the primary benchmark score.
Benchmark scope
What Voice Code Bench measures
Voice Code Bench is intentionally narrower than universal audio suites. It asks one product-critical question: did the transcript preserve every structured entity a downstream workflow needs?
human-recorded English WAV segments
34.9-122.9 sec clips
26 structured entity types
workplace workflow domains
Results
Main findings
Deepgram Nova-3 leads TSR and has the lowest WER. ElevenLabs Scribe v2 leads CTEM, which is the benchmark's core point: general transcript quality, entity-level recovery, and complete task success are related but not interchangeable.
- 01
Deepgram Nova-3 leads the checked-in baseline set on TSR and has the lowest WER.
- 02
ElevenLabs Scribe v2 leads CTEM, showing that entity-level recovery and task-level success can disagree.
- 03
Commands, file paths, URLs, and environment variables expose large gaps across systems.
Metrics
How to read TSR, CTEM, and WER
TSR is strict because real workflows are strict: one missed entity can invalidate the recording. CTEM shows the global exact-match rate across entities, while WER remains a supporting diagnostic for transcript quality.
- Low WER does not guarantee usable transcripts when the application depends on exact structured values.
- Deepgram Nova-3 has the highest TSR; ElevenLabs Scribe v2 has the highest CTEM.
- Commands, file paths, URLs, and environment variables expose large gaps across systems.
Context
Why this benchmark exists
Broad voice benchmarks measure general capability. Voice Code Bench is narrower on purpose: it isolates exact structured-token recovery, the failure mode that matters when speech becomes software input.
Name the task, source dataset, split, and metric explicitly.
Voice Code Bench exposes its 26 entity types and scoring rules up front, so the evaluation target is clear before the results table.Breadth creates authority but makes one-score summaries fragile.
Voice Code Bench is intentionally narrow: it measures exact structured-token recovery rather than claiming broad audio-language coverage.Instruction-following audio systems require hybrid scoring.
The LLM verifier is limited to entity-presence judgments, while canonical targets, evidence, and review files remain auditable.Fairness, toxicity, safety, and robustness belong inside the benchmark frame.
Voice Code Bench surfaces speaker metadata completeness and known imbalance as benchmark context, not appendix material.Good documentation is decision support, not decoration.
The benchmark page puts scale, provenance, schema, metadata, limitations, and prohibited uses where practitioners can evaluate fit quickly.Failure types
Common transcription failures
The purpose is not only to rank STT providers. It is to make the recurring failure patterns visible enough for product and research teams to fix.
Dataset documentation
Dataset documentation requirements
Good benchmark documentation is not decorative. It should make scale, schema, metadata, risks, and scoring rules obvious before someone downloads the data or cites the result.
Use cases
Where exact token recovery matters
Voice coding, support operations, CRM dictation, technical education, finance, healthcare, and legal workflows all depend on exact structured speech.
- 01ASR vendor selection
- 02Voice coding
- 03Support QA
- 04CRM and form dictation
- 05Technical troubleshooting
- 06Agentic voice workflows
Limitations
Current limitations
- Current release is English-only and focused on compact workplace-style dictation.
- It does not cover meetings, overlapping speech, casual conversation, broadcast audio, voice search, noisy field recordings, telephony codecs, or long-form dictation.
- Synthetic content may not capture every distributional property of real production workflows.
- The LLM-assisted verifier introduces dependence on a verifier model and prompt.
- Commercial ASR rankings are tied to the evaluation dates and provider behavior recorded in the baseline artifacts.
Roadmap
Planned extensions
The benchmark should grow in entity types, model coverage, speaker/environment diversity, and auditable slice reporting while preserving the exact-token focus.
- 01
Versioned releases that preserve prior dataset revisions
- 02
Multilingual structured-token recovery
- 03
Noisier environments and broader accent coverage
- 04
Broader conversation formats beyond compact dictation
- 05
Documented changes to entity taxonomy, scoring policy, verifier versions, and baseline systems
- 06
Human verifier calibration and expanded audit samples
Next
Benchmark dashboard
The benchmark page carries the full leaderboard, heatmap, dataset composition, speaker metadata, audio quality, scoring protocol, and reproducibility details.