Back

Benchmark report

Voice Code Bench: Exact Tokens in Speech-to-Text

May 1, 2026Benchmark

Alphanumeric entities like phone number, email addresses and URL present a special challenge for Automatic Speech Recognition models. A single wrong digit can send a call to the wrong person or put the wrong item in an order.

We built VoiceCodeBench to measure ASR models' ability to transcribe alphanumeric heavy critical entities.

Live leaderboardOpen Hugging FaceRead Paper

Overview

Alphanumeric entities are hard to transcribe and important to get right

WER measures transcript similarity. Production voice interfaces need something stricter for critical entities like email addresses, commands, account identifiers, URLs, dates, file paths, dollar amounts, product codes, and measurements. Downstream systems need the exact value to act on.

Traditional WER treats all words as part of the same transcript surface. Production systems do not. Most transcription mistakes can be fixed by downstream reasoning models, but some mistakes are irrecoverable. A single wrong character in an email, CLI flag, file path, URL, account identifier, product code, or measurement can route the wrong ticket, fail an account lookup, or trigger the wrong command.

Low WER does not guarantee usable transcripts when the application depends on exact structured values.

SpokenIncorrectly transcribed
415-555-0101415-555-OR101
lio.quill@access-mail.example.orglio.curuill@access-mail.example.org
reports/release/canary-summary.jsonreport slash release slash canary dash summary dot json

The most common mistakes observed in the benchmark include:

  • Omission - The transcript leaves out a value that was spoken.
  • Substitution - The transcript replaces a character or part of a value with something else.
  • Formatting drift - The transcript changes formatting that carries meaning, such as an underscore in an email address.
  • Spelling collapse - The transcript turns separately spoken letters into a word and loses the intended spelling or capitalization.
  • Near miss - The transcript gives a plausible value that differs from the exact one the workflow needs.
  • Extra entity - The transcript adds a structured value that the speaker never said.

What Voice Code Bench measures

We start with the values a workflow depends on, such as phone numbers, file paths, and measurements, and write workplace scenarios around them. Each target has a spoken form and an exact written form. VoiceCodeBench gives ASR models only the audio and checks whether the transcript contains enough evidence to recover each written value.

Voice coding, support operations, CRM dictation, technical education, finance, healthcare, and legal workflows all depend on exact structured speech.

The public release supports ASR comparison, regression tracking, and analysis by entity type. It is a diagnostic test set. Training, fine-tuning, speaker identification, biometric modeling, voice cloning, and demographic profiling fall outside its intended use.

MetricValueDetails
Recordings300human-recorded English WAV segments
Audio5.587h34.9-122.9 sec clips
Target entities1,48226 structured entity types
Domains8workplace workflow domains

Benchmark details

Metric definitions for CTEM, TSR, and WER

  • Canonical token/entity match (CTEM) - The fraction of target entities whose canonical value is recoverable from the ASR transcript.
  • Task success rate (TSR) - A recording succeeds only when every target entity in that recording is recovered correctly.
  • Word error rate (WER) - A supporting transcript-quality diagnostic, not the primary benchmark score.

Scoring method

Each recording has three reference layers that preserve the script template, how the speaker reads it aloud, and the canonical written values. We give transcription systems only the audio, with no reference transcript, entity hints, custom vocabulary, or benchmark-specific prompts. Batch systems process the whole file. Streaming systems receive it in order, and we score their final transcripts.

For each target, we check whether the transcript contains enough evidence to recover the exact written value. A phone number can pass without hyphens if all the digits are present and in order. A missing file extension or a changed unit fails because it changes the value. A versioned LLM verifier records each decision with the supporting transcript excerpt and its reasoning.

We aggregate those decisions into CTEM overall and by entity type. A recording counts toward TSR only if every target is recovered. We compute WER separately, allowing either the spoken or written form of each reference entity. The release includes scoring code, verifier prompts, predictions, and an audit sample.

Recording and entity distribution

Voice Code Bench is a test-only evaluation dataset, not a training corpus. The release covers 49 scenarios across four difficulty levels.

Recording length summary

MeasureValue
Mean length67.0 sec
Median length65.5 sec
Range34.9-122.9 sec
Mean entities4.9 / recording
Entity range3-8 / recording

Duration buckets

Duration
Recordings
Share
Under 40 sec52%
40-60 sec10234%
60-90 sec16555%
90-120 sec279%
120+ sec10%

Coverage

Entity type
Occurrences
Reference ID150
Spelled sequence97
Product code90
Date89
Acronym/initialism85
Currency amount75
Email address65
Plain number65

Speaker metadata

Speaker ID, self-identified sex, accent, and age-bucket metadata are available for all 300 recordings. The current release includes 85 anonymized speaker IDs.

FieldCategoryRecordings
Share
Self-identified sexFemale15251%
Male14448%
Non-binary41%
Age bucket36-4510234%
18-257625%
46-555418%
26-355318%
56-65103%
66+52%
AccentAmerican general18361%
American New York186%
Nigerian165%
American California145%
American Midwest145%
American Southern145%
British RP134%
Indian62%
Neutral62%
Canadian52%
Spanish Mexican52%
British Northern21%
Kenyan21%
Eastern European10%
Russian Moscow10%

Audio quality

Signal-to-noise ratio, background-noise level, speech level, loudness, and click/pop metadata are available for all 300 recordings.

Quality Field Summary

MetricMeanMedianRange
Signal-to-noise ratio61.8 dB61.1 dB36.5 to 108.8 dB
Background noise RMS-80.6 dBFS-80.1 dBFS-120.0 to -54.9 dBFS
Speech RMS-18.8 dBFS-18.8 dBFS-27.4 to -10.1 dBFS
Integrated loudness-23.0 LUFS-23.0 LUFS-32.0 to -15.0 LUFS
Click/pop events0.11 / min0.00 / min0.00 to 1.98 / min

SNR buckets

SNR
Recordings
Share
80+ dB227%
60-80 dB14247%
45-60 dB11839%
Under 45 dB186%

Results

These public-set results extend the September 12, 2026 paper snapshot with baseline runs through October 9, 2026. For the latest results on the separate private evaluation set, see the live leaderboard.

Main findings

We evaluated 22 STT systems and rank them by TSR. OpenAI GPT Live Transcribe leads CTEM. Deepgram Nova-3 and OpenAI GPT Live Transcribe tie for the highest TSR, while AssemblyAI Universal-3.5 Pro has the lowest WER. Best-in-column highlights show the strongest result for each metric. That is the benchmark's core point: entity-level recovery, complete task success, and general transcript quality are related but not interchangeable.

CTEM leader91.8%

OpenAI GPT Live Transcribe (Streaming)

TSR leaders (tie)68.7%

OpenAI GPT Live Transcribe (Streaming)

Deepgram Nova-3

WER leader3.8%

AssemblyAI Universal-3.5 Pro

Public release standings

RankModel
1Deepgram Nova-3Deepgram68.7%90.9%8.2%
1OpenAI GPT Live Transcribe (Streaming)OpenAI68.7%91.8%5.9%
3ElevenLabs Scribe v2ElevenLabs67.7%91.6%6.9%
4Cartesia Ink 2 (Streaming)Cartesia63.7%90.1%9.3%
5Deepgram Nova-3 (Streaming)Deepgram61.7%88.9%9.3%
6OpenAI GPT TranscribeOpenAI61.0%89.3%6.1%
7Google Cloud Chirp 3Google60.3%88.8%5.8%
8xAI Grok Voice Transcribe 2.0 (Streaming)xAI56.0%85.8%8.7%
9Meta Muse Voice Transcribe 1.0 (Streaming)Meta55.7%86.8%7.4%
9xAI Grok Voice Transcribe 2.0xAI55.7%86.1%7.7%
11Meta Muse Voice Transcribe 1.0Meta55.3%87.6%7.5%
12Whisper Large-v3Open source54.3%87.6%5.8%
13Mistral Voxtral Mini Transcribe 2Mistral AI52.3%86.9%5.5%
14AssemblyAI Universal-3.5 ProAssemblyAI51.7%86.5%3.8%
15Google Cloud Chirp 3 (Streaming)Google50.3%86.1%6.7%
16Thinking Machines Lab Inkling-NVFP4Thinking Machines49.7%84.3%4.9%
17Mistral Voxtral Mini Transcribe Realtime (Streaming)Mistral AI48.0%83.7%6.6%
18ElevenLabs Scribe v2 Realtime (Streaming)ElevenLabs46.3%84.0%6.8%
19Inworld STT 1Inworld44.0%83.7%9.7%
20NVIDIA Parakeet TDT 0.6B v3Modal37.3%78.6%9.3%
21Amazon Transcribe (Streaming)Amazon33.7%75.2%10.3%
22AssemblyAI Universal-3.6 Pro Realtime (Streaming)AssemblyAI27.7%76.9%5.1%

Entity heatmap

Each cell is entity-level CTEM: exact recovery for that structured-value family. The color is relative within each column, making it possible to compare the systems on the tokens a particular product depends on.

Column leaderMiddleLargest gap
ModelEmailPhoneExtPerson /TeamAddressURLIP AddressPortCommandCLI FlagFile PathEnv VarCode SymbolVersionReference IDProduct CodeAccount /RecordCurrenc -yPctMeasure -mentNumberDateTimeAcronymSpelled SeqDomain Term
OpenAI GPT Live Transcribe (Streaming)75%98%100%94%48%66%96%100%78%98%71%91%91%93%95%93%92%100%100%100%100%100%98%98%96%95%
ElevenLabs Scribe v272%98%100%100%78%60%96%100%60%93%65%94%97%93%95%93%97%100%100%100%100%99%95%96%98%100%
Deepgram Nova-371%98%100%100%68%66%96%97%58%89%61%86%86%93%95%94%97%100%100%100%100%100%97%100%98%90%
Cartesia Ink 2 (Streaming)80%98%100%94%60%56%96%97%60%89%55%91%89%97%94%90%95%96%100%100%98%100%97%99%99%95%
OpenAI GPT Transcribe69%97%100%94%55%63%72%93%76%93%67%89%89%93%89%88%91%97%100%100%98%99%98%96%97%100%
Deepgram Nova-3 (Streaming)69%98%100%100%65%58%96%90%62%86%53%80%80%93%95%92%92%97%100%100%98%98%98%98%96%90%
Google Cloud Chirp 374%98%100%100%48%65%96%97%40%91%59%89%91%87%93%92%95%99%98%100%100%100%97%95%92%95%
Whisper Large-v374%97%100%94%60%42%92%93%46%86%55%89%91%93%95%89%91%97%100%100%97%97%93%98%97%85%
Meta Muse Voice Transcribe 1.069%98%100%94%55%40%92%97%52%95%53%91%94%87%93%88%91%88%98%100%100%100%100%98%96%90%
Mistral Voxtral Mini Transcribe 268%98%100%94%55%23%84%93%54%86%61%86%86%93%90%96%92%97%96%100%98%99%97%99%97%90%
Meta Muse Voice Transcribe 1.0 (Streaming)60%98%100%100%60%34%92%100%54%91%51%91%94%93%93%83%91%91%100%100%100%99%98%96%95%85%
AssemblyAI Universal-3.5 Pro54%92%97%100%68%47%88%97%46%86%67%77%100%83%92%88%89%92%98%100%97%100%98%94%94%100%
Google Cloud Chirp 3 (Streaming)63%98%100%100%48%58%96%97%50%86%55%86%83%87%91%93%88%88%98%95%97%99%92%94%93%90%
xAI Grok Voice Transcribe 2.080%98%100%100%70%23%88%97%36%30%55%91%97%97%93%88%92%99%100%100%100%100%97%95%99%90%
xAI Grok Voice Transcribe 2.0 (Streaming)82%95%100%100%65%21%88%97%42%39%55%89%94%93%93%91%91%93%100%100%100%99%95%95%98%95%
Thinking Machines Lab Inkling-NVFP455%92%100%83%40%48%76%83%58%75%61%77%77%93%91%87%86%96%98%100%98%100%97%94%89%95%
ElevenLabs Scribe v2 Realtime (Streaming)63%88%93%100%57%34%92%97%44%82%45%54%91%100%91%80%92%89%100%98%95%97%93%98%97%100%
Inworld STT 158%92%97%94%50%34%68%83%54%84%51%91%83%97%86%90%88%92%100%95%100%96%97%96%91%85%
Mistral Voxtral Mini Transcribe Realtime (Streaming)35%95%90%94%57%26%96%93%46%84%59%86%83%83%88%89%86%99%96%98%98%99%95%98%95%90%
NVIDIA Parakeet TDT 0.6B v332%83%100%94%45%40%76%83%32%70%33%86%86%97%78%77%78%88%100%100%98%97%92%98%90%90%
AssemblyAI Universal-3.6 Pro Realtime (Streaming)38%73%100%100%53%27%72%87%36%89%59%71%83%90%71%74%68%85%98%97%94%97%98%95%78%95%
Amazon Transcribe (Streaming)48%98%100%83%45%10%88%93%8%45%8%43%54%80%84%84%86%75%100%95%97%99%98%94%95%80%

Current limitations

This release covers short English workplace dictation based on synthetic scenarios. The mix of values may differ from production. Coverage excludes meetings, overlapping speech, casual conversation, broadcast audio, voice search, noisy field recordings, telephony codecs, and long-form dictation.

Entity scoring depends on the LLM verifier and its prompt. Rankings for commercial ASR systems reflect the provider behavior and evaluation dates recorded in the baseline artifacts.

Planned extensions

We plan to add more entity types and models and extend exact-token evaluation to more languages, a wider range of accents, noisier recording environments, and conversation formats beyond short dictation.

Versioned releases would keep earlier dataset revisions available and record changes to the entity taxonomy, scoring policy, verifier versions, and baseline systems. We also plan to publish auditable results by slice and expand the human audit samples to calibrate the verifier.

Dataset and paper

The dataset, baseline outputs, evaluation harness, and documentation are available on Hugging Face. The paper records the motivation, benchmark design, scoring protocol, baseline setup, and release results.

Live leaderboardOpen Hugging FaceRead Paper