Book a Demo
AboutHow We Do itAudio DataWhy UsResearchBlogsJobsBook a Demo

Benchmark dashboard

Voice Code Bench

A speech-to-text benchmark for exact structured values in English workplace speech, including emails, command-line flags, file paths, URLs, account identifiers, dates, and measurements.

Read BlogRead PaperOpen Hugging Face
Top TSR68.7%

Deepgram Nova-3

Gold entities1,482

audited targets

STT systems12

checked-in baselines

Recordings300

5.587h audio

Domains8

workflow domains

Entity types26

structured value families

WER measures transcript similarity. Voice Code Bench measures exact entity recovery.

spokenDHCPtranscribed as DHP
spoken192.168.1.1one octet collapses
spokenP-I-N-Gnormalized to ping

Leaderboard

Model results

Default sorting is TSR. Best-in-column highlights show the strongest task success, global entity recovery, and transcript-quality scores.

TSRTask success rate

A recording succeeds only when every target entity in that recording is recovered correctly.

CTEMCanonical token/entity match

The fraction of target entities whose canonical value is recoverable from the ASR transcript.

WERWord error rate

A supporting transcript-quality diagnostic, not the primary benchmark score.

#1Deepgram Nova-3
68.7%
#2ElevenLabs Scribe v2
67.7%
#3Deepgram Nova-3 Streaming
61.7%
#4Google Cloud Chirp 3
60.3%
#5OpenAI GPT Realtime Whisper Streaming
59.3%
#6OpenAI GPT-4o Transcribe
55.7%
#7Whisper Large v3
54.3%
#8AssemblyAI Universal-3 Pro
50.3%
#9Google Cloud Chirp 3 Streaming
50.3%
#10ElevenLabs Scribe v2 Realtime Streaming
46.3%
#11Amazon Transcribe Streaming
33.7%
#12AssemblyAI Universal-3 Pro Streaming
33.0%
RankModelTSRCTEMWER
#1Deepgram Nova-3Highest TSR in the released baseline suite and lowest WER among tracked systems.68.7%90.9%8.6%
#2ElevenLabs Scribe v2Highest CTEM in the released baseline suite, with strong code/system and language-form recovery.67.7%91.6%16.1%
#3Deepgram Nova-3 StreamingStrongest streaming-mode TSR and second-lowest WER among tracked systems.61.7%88.9%9.6%
#4Google Cloud Chirp 3Strong email recovery and competitive TSR, but WER trails the top systems in this release.60.3%88.8%25.6%
#5OpenAI GPT Realtime Whisper StreamingStreaming baseline with strong command and CLI-flag recovery but lower phone-number recovery than top batch systems.59.3%88.7%16.3%
#6OpenAI GPT-4o TranscribeBatch OpenAI baseline with solid TSR, but WER is materially higher than the top Deepgram release result.55.7%87.4%24.6%
#7Whisper Large v3Open-source baseline with competitive TSR but weaker URL and command recovery.54.3%87.6%23.1%
#8AssemblyAI Universal-3 ProBatch AssemblyAI baseline with strong phone-extension and port-number recovery, but lower TSR than the top hosted systems.50.3%84.5%25.2%
#9Google Cloud Chirp 3 StreamingStreaming Google baseline with strong phone and IP recovery but lower overall TSR than batch Chirp 3.50.3%86.1%25.5%
#10ElevenLabs Scribe v2 Realtime StreamingRealtime ElevenLabs baseline with strong person/team, version, and domain-term recovery but lower TSR than batch Scribe v2.46.3%84.0%24.9%
#11Amazon Transcribe StreamingStreaming Amazon baseline with the weakest TSR and CTEM in the released baseline suite.33.7%75.2%25.6%
#12AssemblyAI Universal-3 Pro StreamingStreaming AssemblyAI baseline with the lowest TSR in the released baseline suite.33.0%78.3%25.2%

Entity heatmap

Entity-level results

Each cell is entity-level CTEM: exact recovery for that entity type. The color is relative within each column, so teams can see which model leads on the tokens their own product depends on.

Entity-level CTEMRows are STT systems. Columns are the structured token families in the benchmark.
Column leaderMiddleLargest gap
ModelEmailPhoneExtPerson/teamAddressURLIP addressPortCommandCLI flagFile pathEnv varCode symbolVersionReference IDProduct codeAccount/recordCurrencyPctMeasurementNumberDateTimeAcronymSpelled seqDomain term
Deepgram71%98%100%100%68%66%96%97%58%89%61%86%86%93%95%94%97%100%100%100%100%100%97%100%98%90%
ElevenLabs72%98%100%100%78%60%96%100%60%93%65%94%97%93%95%93%97%100%100%100%100%99%95%96%98%100%
Deepgram69%98%100%100%65%58%96%90%62%86%53%80%80%93%95%92%92%97%100%100%98%98%98%98%96%90%
Google74%98%100%100%48%65%96%97%40%91%59%89%91%87%93%92%95%99%98%100%100%100%97%95%92%95%
OpenAI68%93%93%94%48%60%76%93%68%93%65%80%91%97%94%94%92%95%100%100%98%100%100%95%91%100%
OpenAI62%97%100%100%57%50%76%93%34%91%61%83%86%93%91%94%92%100%100%100%100%100%100%96%94%95%
Open source74%97%100%94%60%42%92%93%46%86%55%89%91%93%95%89%91%97%100%100%97%97%93%98%97%85%
AssemblyAI51%88%97%94%68%47%80%97%44%80%63%51%74%87%91%84%91%92%98%100%98%100%98%98%95%100%
Google63%98%100%100%48%58%96%97%50%86%55%86%83%87%91%93%88%88%98%95%97%99%92%94%93%90%
ElevenLabs63%88%93%100%57%34%92%97%44%82%45%54%91%100%91%80%92%89%100%98%95%97%93%98%97%100%
Amazon48%98%100%83%45%10%88%93%8%45%8%43%54%80%84%84%86%75%100%95%97%99%98%94%95%80%
AssemblyAI46%82%87%94%55%34%68%90%36%89%47%66%77%93%76%79%77%79%100%97%95%98%98%93%85%100%

TSR vs WER

Task success vs word error rate

WER stays visible as a transcript-quality diagnostic. TSR is stricter because each recording only passes when every benchmark entity survives.

Highest TSRDeepgram Nova-3

68.7% task success rate

Lowest WERDeepgram Nova-3

0.0861 word error rate

Interpretation0.0 pp gap

The lowest-WER model trails the top task-success model by this many percentage points on complete entity recovery.

Dataset composition

Recording and entity distribution

Voice Code Bench is a test-only evaluation dataset, not a training corpus. The release covers 49 scenarios across 4 difficulty levels.

Recordings300

human-recorded English WAV segments

Audio5.587h

34.9-122.9 sec clips

Target entities1,482

26 structured entity types

Domains8

workplace workflow domains

Entity Coverage

Reference ID
150
Spelled sequence
97
Product code
90
Date
89
Acronym/initialism
85
Currency amount
75
Email address
65
Plain number
65
Account/record #
65
URL
62
Measurement
61
Phone number
60
Time
60
File path
51
Command
50
Percentage
50
CLI flag
44
Postal address
40
Environment variable
35
Code symbol
35
Phone extension
30
Version
30
Port number
30
IP address
25
Domain term
20
Person/team name
18

Domain Coverage

Technical IT/dev
55
Contact routing
45
Retail/logistics/order
45
Finance/billing
40
Healthcare admin
35
Legal/insurance/gov
35
Education/workplace
25
Dense mixed stress
20

Duration Buckets

Under 40 sec
5 (2%)
40-60 sec
102 (34%)
60-90 sec
165 (55%)
90-120 sec
27 (9%)
120+ sec
1 (0%)

Recording Length Summary

Mean length67.0 sec
Median length65.5 sec
Range34.9-122.9 sec
Mean entities4.9 / recording
Entity range3-8 / recording

Speaker metadata

Speaker distribution

Speaker ID, sex, accent, and age-bucket metadata are available for all 300 recordings. The current release includes 85 anonymized speaker IDs.

FieldCategoryRecordingsShare
Speaker sexFemale152
Male144
Non-binary4
Age bucket36-45102
18-2576
46-5554
26-3553
56-6510
66+5
AccentAmerican general183
American New York18
Nigerian16
American California14
American Midwest14
American Southern14
British RP13
Indian6
Neutral6
Canadian5
Spanish Mexican5
British Northern2
Kenyan2
Eastern European1
Russian Moscow1

Audio quality

Audio quality metrics

Signal-to-noise ratio, background-noise level, speech level, loudness, and click/pop metadata are available for all 300 recordings.

Quality Field Summary

Mean and median values across all recordings.

MetricMeanMedianRange
Signal-to-noise ratio61.8 dB61.1 dB36.5 to 108.8 dB
Background noise RMS-80.6 dBFS-80.1 dBFS-120.0 to -54.9 dBFS
Speech RMS-18.8 dBFS-18.8 dBFS-27.4 to -10.1 dBFS
Integrated loudness-23.0 LUFS-23.0 LUFS-32.0 to -15.0 LUFS
Click/pop events0.11 / min0.00 / min0.00 to 1.98 / min

Metadata Summary

Audio quality metadata300 / 300
Mean SNR61.8 dB
Median SNR61.1 dB
SNR range36.5-108.8 dB
Mean loudness-23.0 LUFS
Median loudness-23.0 LUFS

SNR Buckets

80+ dB
22 (7%)
60-80 dB
142 (47%)
45-60 dB
118 (39%)
Under 45 dB
18 (6%)

Scoring method

Evaluation protocol

1

Audio to transcript

The provider receives no gold transcript or benchmark entity values.

2

Transcript to entities

The released scorer uses an LLM-assisted recoverability verifier and stores evidence and reasons.

3

Entities to benchmark

TSR, CTEM, entity CTEM, WER, score files, and verifier-provenance outputs are generated.

Three transcript layers

Each item includes template, acoustic, and canonical transcript layers.

Raw audio only

Systems receive only the audio file in the main setting, without benchmark prompts, entity lists, domain labels, custom vocabulary, grammar constraints, or post-ASR correction.

Canonical recovery

Scoring asks whether the written value a downstream application needs is recoverable from the ASR transcript.

Verifier audit trail

Entity scoring uses an LLM-assisted recoverability verifier with versioned prompts, response schemas, evidence, reasons, and a released audit sample.

Diagnostic use

VoiceCodeBench is intended for diagnostic ASR evaluation, provider comparison, regression tracking, and per-entity risk analysis.

Out-of-scope uses

The release is not intended for training, fine-tuning, speaker identification, biometric modeling, voice cloning, or demographic profiling.

Reproducibility

Files and commands

Repository layout

  • data/audio/001.wav through data/audio/300.wav
  • data/metadata.json
  • baselines/predictions/*.json
  • baselines/results.csv
  • audit/verifier_audit_samples.csv
  • scripts/voice_code_bench/
  • paper/voice-code-bench.pdf
  • DATASET_CARD.md

Baseline IDs

  • deepgram_nova3
  • elevenlabs_scribe_v2
  • deepgram_nova3_streaming
  • google_cloud_chirp_3
  • openai_gpt_realtime_whisper_streaming
  • openai_gpt_4o_transcribe
  • whisper_large_v3
  • assemblyai_universal_3_pro
  • google_cloud_chirp_3_streaming
  • elevenlabs_scribe_v2_realtime_streaming
  • amazon_transcribe_streaming
  • assemblyai_universal_3_pro_streaming

Commands

Install scorer
python -m pip install -e .
Reproduce release artifacts
./scripts/reproduce_release.sh
Run tests
python -m pytest
Run new provider baselines
vcb-run --stt-mode all --output-dir runs/full-local

Paper

Technical paper for VoiceCodeBench, including the benchmark motivation, scoring protocol, baseline setup, and release details.

Read paper PDF