Benchmark report
Voice Code Bench: Exact Tokens in Speech-to-Text
Alphanumeric entities like phone number, email addresses and URL present a special challenge for Automatic Speech Recognition models. A single wrong digit can send a call to the wrong person or put the wrong item in an order.
We built VoiceCodeBench to measure ASR models' ability to transcribe alphanumeric heavy critical entities.
Overview
Alphanumeric entities are hard to transcribe and important to get right
WER measures transcript similarity. Production voice interfaces need something stricter for critical entities like email addresses, commands, account identifiers, URLs, dates, file paths, dollar amounts, product codes, and measurements. Downstream systems need the exact value to act on.
Traditional WER treats all words as part of the same transcript surface. Production systems do not. Most transcription mistakes can be fixed by downstream reasoning models, but some mistakes are irrecoverable. A single wrong character in an email, CLI flag, file path, URL, account identifier, product code, or measurement can route the wrong ticket, fail an account lookup, or trigger the wrong command.
Low WER does not guarantee usable transcripts when the application depends on exact structured values.
| Spoken | Incorrectly transcribed |
|---|---|
415-555-0101 | 415-555-OR101 |
lio.quill@access-mail.example.org | lio.curuill@access-mail.example.org |
reports/release/canary-summary.json | report slash release slash canary dash summary dot json |
The most common mistakes observed in the benchmark include:
- Omission - The transcript leaves out a value that was spoken.
- Substitution - The transcript replaces a character or part of a value with something else.
- Formatting drift - The transcript changes formatting that carries meaning, such as an underscore in an email address.
- Spelling collapse - The transcript turns separately spoken letters into a word and loses the intended spelling or capitalization.
- Near miss - The transcript gives a plausible value that differs from the exact one the workflow needs.
- Extra entity - The transcript adds a structured value that the speaker never said.
What Voice Code Bench measures
We start with the values a workflow depends on, such as phone numbers, file paths, and measurements, and write workplace scenarios around them. Each target has a spoken form and an exact written form. VoiceCodeBench gives ASR models only the audio and checks whether the transcript contains enough evidence to recover each written value.
Voice coding, support operations, CRM dictation, technical education, finance, healthcare, and legal workflows all depend on exact structured speech.
The public release supports ASR comparison, regression tracking, and analysis by entity type. It is a diagnostic test set. Training, fine-tuning, speaker identification, biometric modeling, voice cloning, and demographic profiling fall outside its intended use.
| Metric | Value | Details |
|---|---|---|
| Recordings | 300 | human-recorded English WAV segments |
| Audio | 5.587h | 34.9-122.9 sec clips |
| Target entities | 1,482 | 26 structured entity types |
| Domains | 8 | workplace workflow domains |
Benchmark details
Metric definitions for CTEM, TSR, and WER
- Canonical token/entity match (CTEM) - The fraction of target entities whose canonical value is recoverable from the ASR transcript.
- Task success rate (TSR) - A recording succeeds only when every target entity in that recording is recovered correctly.
- Word error rate (WER) - A supporting transcript-quality diagnostic, not the primary benchmark score.
Scoring method
Each recording has three reference layers that preserve the script template, how the speaker reads it aloud, and the canonical written values. We give transcription systems only the audio, with no reference transcript, entity hints, custom vocabulary, or benchmark-specific prompts. Batch systems process the whole file. Streaming systems receive it in order, and we score their final transcripts.
For each target, we check whether the transcript contains enough evidence to recover the exact written value. A phone number can pass without hyphens if all the digits are present and in order. A missing file extension or a changed unit fails because it changes the value. A versioned LLM verifier records each decision with the supporting transcript excerpt and its reasoning.
We aggregate those decisions into CTEM overall and by entity type. A recording counts toward TSR only if every target is recovered. We compute WER separately, allowing either the spoken or written form of each reference entity. The release includes scoring code, verifier prompts, predictions, and an audit sample.
Recording and entity distribution
Voice Code Bench is a test-only evaluation dataset, not a training corpus. The release covers 49 scenarios across four difficulty levels.
Recording length summary
| Measure | Value |
|---|---|
| Mean length | 67.0 sec |
| Median length | 65.5 sec |
| Range | 34.9-122.9 sec |
| Mean entities | 4.9 / recording |
| Entity range | 3-8 / recording |
Duration buckets
| Duration | Recordings | Share |
|---|---|---|
| Under 40 sec | 5 | 2% |
| 40-60 sec | 102 | 34% |
| 60-90 sec | 165 | 55% |
| 90-120 sec | 27 | 9% |
| 120+ sec | 1 | 0% |
Coverage
| Entity type | Occurrences |
|---|---|
| Reference ID | 150 |
| Spelled sequence | 97 |
| Product code | 90 |
| Date | 89 |
| Acronym/initialism | 85 |
| Currency amount | 75 |
| Email address | 65 |
| Plain number | 65 |
Speaker metadata
Speaker ID, self-identified sex, accent, and age-bucket metadata are available for all 300 recordings. The current release includes 85 anonymized speaker IDs.
| Field | Category | Recordings | Share |
|---|---|---|---|
| Self-identified sex | Female | 152 | 51% |
| Male | 144 | 48% | |
| Non-binary | 4 | 1% | |
| Age bucket | 36-45 | 102 | 34% |
| 18-25 | 76 | 25% | |
| 46-55 | 54 | 18% | |
| 26-35 | 53 | 18% | |
| 56-65 | 10 | 3% | |
| 66+ | 5 | 2% | |
| Accent | American general | 183 | 61% |
| American New York | 18 | 6% | |
| Nigerian | 16 | 5% | |
| American California | 14 | 5% | |
| American Midwest | 14 | 5% | |
| American Southern | 14 | 5% | |
| British RP | 13 | 4% | |
| Indian | 6 | 2% | |
| Neutral | 6 | 2% | |
| Canadian | 5 | 2% | |
| Spanish Mexican | 5 | 2% | |
| British Northern | 2 | 1% | |
| Kenyan | 2 | 1% | |
| Eastern European | 1 | 0% | |
| Russian Moscow | 1 | 0% |
Audio quality
Signal-to-noise ratio, background-noise level, speech level, loudness, and click/pop metadata are available for all 300 recordings.
Quality Field Summary
| Metric | Mean | Median | Range |
|---|---|---|---|
| Signal-to-noise ratio | 61.8 dB | 61.1 dB | 36.5 to 108.8 dB |
| Background noise RMS | -80.6 dBFS | -80.1 dBFS | -120.0 to -54.9 dBFS |
| Speech RMS | -18.8 dBFS | -18.8 dBFS | -27.4 to -10.1 dBFS |
| Integrated loudness | -23.0 LUFS | -23.0 LUFS | -32.0 to -15.0 LUFS |
| Click/pop events | 0.11 / min | 0.00 / min | 0.00 to 1.98 / min |
SNR buckets
| SNR | Recordings | Share |
|---|---|---|
| 80+ dB | 22 | 7% |
| 60-80 dB | 142 | 47% |
| 45-60 dB | 118 | 39% |
| Under 45 dB | 18 | 6% |
Results
These public-set results extend the September 12, 2026 paper snapshot with baseline runs through October 9, 2026. For the latest results on the separate private evaluation set, see the live leaderboard.
Main findings
We evaluated 22 STT systems and rank them by TSR. OpenAI GPT Live Transcribe leads CTEM. Deepgram Nova-3 and OpenAI GPT Live Transcribe tie for the highest TSR, while AssemblyAI Universal-3.5 Pro has the lowest WER. Best-in-column highlights show the strongest result for each metric. That is the benchmark's core point: entity-level recovery, complete task success, and general transcript quality are related but not interchangeable.
OpenAI GPT Live Transcribe (Streaming)
OpenAI GPT Live Transcribe (Streaming)
Deepgram Nova-3
AssemblyAI Universal-3.5 Pro
Public release standings
| Rank | Model | |||
|---|---|---|---|---|
| 1 | 68.7% | 90.9% | 8.2% | |
| 1 | 68.7% | 91.8% | 5.9% | |
| 3 | 67.7% | 91.6% | 6.9% | |
| 4 | 63.7% | 90.1% | 9.3% | |
| 5 | 61.7% | 88.9% | 9.3% | |
| 6 | 61.0% | 89.3% | 6.1% | |
| 7 | 60.3% | 88.8% | 5.8% | |
| 8 | 56.0% | 85.8% | 8.7% | |
| 9 | 55.7% | 86.8% | 7.4% | |
| 9 | 55.7% | 86.1% | 7.7% | |
| 11 | 55.3% | 87.6% | 7.5% | |
| 12 | 54.3% | 87.6% | 5.8% | |
| 13 | 52.3% | 86.9% | 5.5% | |
| 14 | 51.7% | 86.5% | 3.8% | |
| 15 | 50.3% | 86.1% | 6.7% | |
| 16 | 49.7% | 84.3% | 4.9% | |
| 17 | 48.0% | 83.7% | 6.6% | |
| 18 | 46.3% | 84.0% | 6.8% | |
| 19 | 44.0% | 83.7% | 9.7% | |
| 20 | 37.3% | 78.6% | 9.3% | |
| 21 | 33.7% | 75.2% | 10.3% | |
| 22 | 27.7% | 76.9% | 5.1% |
Entity heatmap
Each cell is entity-level CTEM: exact recovery for that structured-value family. The color is relative within each column, making it possible to compare the systems on the tokens a particular product depends on.
Current limitations
This release covers short English workplace dictation based on synthetic scenarios. The mix of values may differ from production. Coverage excludes meetings, overlapping speech, casual conversation, broadcast audio, voice search, noisy field recordings, telephony codecs, and long-form dictation.
Entity scoring depends on the LLM verifier and its prompt. Rankings for commercial ASR systems reflect the provider behavior and evaluation dates recorded in the baseline artifacts.
Planned extensions
We plan to add more entity types and models and extend exact-token evaluation to more languages, a wider range of accents, noisier recording environments, and conversation formats beyond short dictation.
Versioned releases would keep earlier dataset revisions available and record changes to the entity taxonomy, scoring policy, verifier versions, and baseline systems. We also plan to publish auditable results by slice and expand the human audit samples to calibrate the verifier.
Dataset and paper
The dataset, baseline outputs, evaluation harness, and documentation are available on Hugging Face. The paper records the motivation, benchmark design, scoring protocol, baseline setup, and release results.