Benchmark report
VocalAffectBench: Can AI Hear How You Feel?
A public, test-only benchmark for evaluating whether AI audio models can identify expressed vocal emotion from raw speech — without transcripts or contextual metadata. Seven baselines. Seven emotion classes. One finding: affect recognition remains brittle.
Problem
Transcripts carry the words. Not the delivery.
A customer who says "that's fine" may be conceding, joking, or signaling frustration. Automatic speech recognition preserves the phrase but strips the vocal delivery that makes the exchange interpretable. Voice agents that apply affect analysis only after transcription measure lexical sentiment — a different question from expressed vocal emotion.
Benchmark design
Audio only. No transcripts. No metadata.
Every baseline receives the raw audio file and a fixed instruction listing the allowed labels. No transcript, no speaker context, no domain hint. For provider endpoints that do not accept prompts, the audio is submitted with default settings and the native labels are mapped to the benchmark set before scoring.
The emotion delivered in the recording, verified by a human reviewer. Not the speaker's inferred private state.
angry, disgusted, fearful, happy, neutral, sad, surprised — Ekman's basic-emotion taxonomy plus neutral.
Fraction of clips whose mapped prediction matches the reference label. Per-class precision and recall expose failure modes.
Dataset
What VocalAffectBench contains
280 human-recorded English clips from 52 speaker accounts, equally distributed across seven emotion classes. All clips are General American accent, 16 kHz mono WAV, with no noise reduction, normalization, or enhancement.
40 clips per emotion class
angry, disgusted, fearful, happy, neutral, sad, surprised
Human-recorded English WAV, General American accent
138.9 minutes of reviewed speech
General audio models, speech-specific models, and provider endpoints
Chance level for a 7-way classification task
| Emotion | Clips | Minutes | Mean (sec) |
|---|---|---|---|
| angry | 40 | 17.1 | 25.7 |
| disgusted | 40 | 19.0 | 28.5 |
| fearful | 40 | 17.5 | 26.3 |
| happy | 40 | 22.8 | 34.2 |
| neutral | 40 | 15.6 | 23.3 |
| sad | 40 | 26.0 | 38.9 |
| surprised | 40 | 21.0 | 31.6 |
Results
Main findings
The leading model reaches 44.3% on a seven-way task. The average across all seven baselines is 34.7%. That is above the 14.3% random baseline, but far from reliable emotion recognition for production use.
Inkling is a post-publication baseline. The paper remains a frozen snapshot of the original six-model release.
- 01
Gemini 3.5 Flash leads at 44.3% — roughly 3× the random baseline but still misses more than half of all clips.
- 02
Neutral is over-predicted: 47.3% of all model outputs map to neutral, causing most non-neutral errors.
- 03
Surprised and fearful are the hardest classes, with 13.2% and 14.6% aggregate recall across baselines.
- 04
Valence (positive / neutral / negative) is more tractable at 48.9% aggregate accuracy, but still unsolved.
| Rank | Model | Acc. | Angry | Disgusted | Fearful | Happy | Neutral | Sad | Surprised |
|---|---|---|---|---|---|---|---|---|---|
| #1 | Gemini 3.5 Flash | 44.3% | 57.5% | 27.5% | 32.5% | 45.0% | 75.0% | 70.0% | 2.5% |
| #2 | Hume Prosody | 38.0% | 50.0% | 5.1% | 0.0% | 62.5% | 85.0% | 22.5% | 40.0% |
| #3 | Qwen3.5 Omni+ | 37.9% | 35.0% | 25.0% | 17.5% | 47.5% | 65.0% | 65.0% | 10.0% |
| #4 | Voxtral Small | 34.3% | 17.5% | 72.5% | 17.5% | 35.0% | 47.5% | 27.5% | 22.5% |
| #5 | Thinking Machines Inkling | 32.1% | 32.5% | 2.5% | 12.5% | 35.0% | 85.0% | 55.0% | 2.5% |
| #6 | Inworld Voice | 28.6% | 37.5% | 0.0% | 12.5% | 30.0% | 97.5% | 17.5% | 5.0% |
| #7 | OpenAI Realtime | 27.9% | 32.5% | 2.5% | 10.0% | 25.0% | 87.5% | 27.5% | 10.0% |
Class difficulty
Not all emotions are equally hard.
Neutral is the easiest class by recall at 77.5%, but the hardest by precision at 23.4% — because models over-predict it. Fearful and surprised reach only 14.6% and 13.2% recall. A model that looks useful on a neutral-heavy workload can still fail on the high-salience states that matter most for escalation or intervention.
Recall — what fraction of true clips were correctly identified
Precision — when predicted, how often was it correct
Neutral bias
The dominant error is predicting neutral.
Across all scored decisions, models output neutral 927 times — 47.3% of all outputs. Of the 1,279 incorrect predictions, 710 collapse to neutral. The largest single confusion is fearful → neutral (138 times), followed by disgusted → neutral (133) and sad → neutral (124).
A conservative neutral prediction can make a voice product miss the affective states that should trigger escalation, empathy, or additional caution.
Valence analysis
Coarser signal, still unsolved.
Some applications only need a positive / neutral / negative signal rather than a seven-way label. Mapping the benchmark to three valence buckets (excluding surprised, whose valence is ambiguous) raises aggregate accuracy to 48.9%. The best model reaches 66.2%. Valence is more tractable than discrete emotion — but it is not solved.
| Model | Neg. out. | Pos. out. | Neutral out. | Ambig. out. | Output skew |
|---|---|---|---|---|---|
| Gemini 3.5 Flash | 51.4% | 10.7% | 36.8% | 1.1% | Negative |
| Hume Prosody | 24.4% | 27.6% | 36.2% | 11.8% | Neutral |
| Qwen3.5 Omni+ | 42.9% | 12.9% | 41.8% | 2.5% | Negative |
| Voxtral Small | 50.7% | 16.4% | 20.0% | 12.9% | Negative |
| Thinking Machines Inkling | 30.0% | 7.5% | 59.3% | 3.2% | Neutral |
| Inworld Voice | 16.8% | 14.6% | 67.5% | 1.1% | Neutral |
| OpenAI Realtime | 17.9% | 6.1% | 69.6% | 6.4% | Neutral |
- Three baselines are negative-skewed; four are neutral-skewed. None is positive-skewed.
- Two models with similar aggregate accuracy can create different product risks if one over-predicts negative affect and another over-predicts neutral delivery.
- Output valence skew is descriptive: it summarizes what labels the model emits under this benchmark, not an intrinsic model property.
Protocol
How baselines are evaluated
Raw audio input
The model receives the WAV file and a fixed instruction listing the seven allowed labels. No transcript, metadata, or context is provided.
Label mapping
Provider-native labels (e.g. calmness → neutral, anxiety → fearful) are mapped to the benchmark set using the released mapping before scoring.
Score and report
Accuracy, per-class precision and recall, confusion counts, and valence accuracy are computed from the mapped predictions and stored in predictions.csv.
Limitations
What this benchmark cannot claim
- English-only, General American accent. Multilingual and cross-accent evaluation requires additional collection.
- Seven discrete labels. Human affect is richer; a single class cannot capture every perceived state.
- Balanced 40-clip classes. This enables fair comparison but does not reflect real traffic distributions.
- Performed emotional speech. Results cannot be interpreted as evidence of spontaneous emotion detection.
- Single reviewer per clip. Inter-rater reliability analysis is not supported by this release.
Next
Dataset and paper
The dataset, baseline predictions, evaluation harness, and documentation are released under the MIT License on Hugging Face. The paper describes the benchmark design, evaluation protocol, baseline setup, and full results.