VocalAffectBench: Can AI Hear How You Feel?
A public, test-only benchmark for evaluating whether AI audio models can identify expressed vocal emotion from raw speech — without transcripts or contextual metadata. Ten baselines. Seven emotion classes. One finding: affect recognition remains brittle.
Overview
Emotion matters
A customer who says "that's fine" may be conceding, joking, or signaling frustration. Automatic Speech Recognition (ASR) models preserve the phrase but strip the vocal delivery that makes the exchange interpretable. Voice agents that apply affect analysis only after transcription measure lexical sentiment — a different question from expressed vocal emotion.
A number of omni and emotion-detection specific models promise to solve this problem. Given an audio clip, these models should be able to detect the correct emotion. How accurate are their predictions?
| words spoken | "that's fine" | spoken with emotions like conceding, joking, or frustrated |
|---|---|---|
| what ASR captures | text only | tone, pace, pauses, and intensity are lost after transcription |
| what voice agents need | expressed affect | which emotion was delivered, not inferred from the words |
What VocalAffectBench contains
280 human-recorded English clips from 52 speaker accounts, equally distributed across seven emotion classes and sourced from acted and naturalistic recordings. All clips are General American accent, 16 kHz mono WAV, with no noise reduction, normalization, or enhancement.
| Metric | Value | Details |
|---|---|---|
| Audio clips | 280 | 40 clips per emotion class |
| Emotion classes | 7 | angry, disgusted, fearful, happy, neutral, sad, surprised |
| Speaker accounts | 52 | Human-recorded English WAV, General American accent |
| Total audio | 2.32 hrs | 138.9 minutes of reviewed speech |
| Models evaluated | 10 | General audio models, speech-specific models, and provider endpoints |
| Random baseline | 14.3% | Chance level for a 7-way classification task |
Benchmark details
Benchmark design
Every model receives the raw audio file and a fixed instruction listing the allowed labels. No transcript, no speaker context, no domain hint. For provider endpoints that do not accept prompts, the audio is submitted with default settings and the native labels are mapped to the benchmark set before scoring.
| Field | Value | Details |
|---|---|---|
| Target | Expressed emotion | The emotion delivered in the recording, verified by a human reviewer. Not the speaker's inferred private state. |
| Label set | 7 classes | angry, disgusted, fearful, happy, neutral, sad, surprised — Ekman's basic-emotion taxonomy plus neutral. |
| Metric | Accuracy | Fraction of clips whose mapped prediction matches the reference label. Per-class precision and recall expose failure modes. |
Scoring method
The model receives the WAV file and a fixed instruction listing the seven allowed labels. No transcript, metadata, or context is provided. Provider-native labels (e.g. calmness → neutral, anxiety → fearful) are mapped to the benchmark set using the released mapping before scoring. Accuracy, per-class precision and recall, confusion counts, and valence accuracy are computed from the mapped predictions and stored in predictions.csv.
Dataset composition
| Emotion | Clips | Minutes | Mean (sec) |
|---|---|---|---|
| Angry | 40 | 17.1 | 25.7 |
| Disgusted | 40 | 19.0 | 28.5 |
| Fearful | 40 | 17.5 | 26.3 |
| Happy | 40 | 22.8 | 34.2 |
| Neutral | 40 | 15.6 | 23.3 |
| Sad | 40 | 26.0 | 38.9 |
| Surprised | 40 | 21.0 | 31.6 |
Results
Main findings
- 01
Hopper Audio Perception-1 leads at 59.6% (167/280), ahead of Oruk Resonance 2 at 49.3%. It still misses about two in five clips.
- 02
Neutral is over-predicted: 43.6% of all scored outputs map to neutral, causing most non-neutral errors.
- 03
Surprised and fearful are the hardest classes, with 18.0% and 18.3% aggregate recall across baselines.
- 04
Valence (positive / neutral / negative) is more tractable at 55.2% aggregate accuracy, but still unsolved.
Public release standings
| Rank | Model | |
|---|---|---|
| 1 | 59.6% | |
| 2 | 49.3% | |
| 3 | 45.7% | |
| 4 | 44.3% | |
| 5 | 38.0% | |
| 6 | 37.9% | |
| 7 | 34.3% | |
| 8 | 32.1% | |
| 9 | 28.6% | |
| 10 | 27.9% |
Not all emotions are equally hard.
Neutral is the easiest class by recall at 80.8%, but the hardest by precision at 26.5% — because models over-predict it. Surprised and fearful reach only 18.0% and 18.3% recall. A model that looks useful on a neutral-heavy workload can still fail on the high-salience states that matter most for escalation or intervention.
Recall — what fraction of true clips were correctly identified
| Emotion | Recall |
|---|---|
| Neutral | 80.8% |
| Happy | 48.0% |
| Sad | 45.0% |
| Angry | 41.8% |
| Disgusted | 26.6% |
| Fearful | 18.3% |
| Surprised | 18.0% |
Precision — when predicted, how often was it correct
| Emotion | Precision |
|---|---|
| Neutral | 26.5% |
| Happy | 49.9% |
| Sad | 41.3% |
| Angry | 61.6% |
| Disgusted | 48.6% |
| Fearful | 81.1% |
| Surprised | 40.2% |
The dominant error is predicting neutral.
Across all scored decisions, models output neutral 1,220 times — 43.6% of all outputs. Of the 1,686 incorrect predictions, 897 collapse to neutral. The largest single confusion is disgusted → neutral (168 times), followed by fearful → neutral (167) and sad → neutral (163).
A conservative neutral prediction can make a voice product miss the affective states that should trigger escalation, empathy, or additional caution.
| Actual → predicted | Count |
|---|---|
| Disgusted → Neutral | 168 |
| Fearful → Neutral | 167 |
| Sad → Neutral | 163 |
| Surprised → Neutral | 146 |
| Happy → Neutral | 127 |
| Angry → Neutral | 126 |
| Fearful → Sad | 93 |
| Surprised → Happy | 88 |
Valence analysis
Some applications only need a positive / neutral / negative signal rather than a seven-way label. Mapping the benchmark to three valence buckets (excluding surprised, whose valence is ambiguous) raises aggregate accuracy to 55.2%. The best model, Hopper Audio Perception-1, reaches 80.8%. Valence is more tractable than discrete emotion — but it is not solved.
| Rank | Model | |
|---|---|---|
| 1 | 80.8% | |
| 2 | 66.3% | |
| 3 | 65.0% | |
| 4 | 64.2% | |
| 5 | 55.0% | |
| 6 | 52.5% | |
| 7 | 47.7% | |
| 8 | 47.1% | |
| 9 | 38.3% | |
| 10 | 35.4% |
| Model | Neg. out. | Pos. out. | Neutral out. | Ambig. out. | Output skew |
|---|---|---|---|---|---|
| Hopper Audio Perception-1 | 51.4% | 13.2% | 22.9% | 12.5% | Negative |
| Oruk Resonance 2 | 38.2% | 13.6% | 37.5% | 10.7% | Negative |
| Gemini 3.8 Flash | 38.9% | 15.0% | 44.3% | 1.8% | Neutral |
| Gemini 3.5 Flash | 51.4% | 10.7% | 36.8% | 1.1% | Negative |
| Hume Prosody | 24.4% | 27.6% | 36.2% | 11.8% | Neutral |
| Qwen3.5 Omni+ | 42.9% | 12.9% | 41.8% | 2.5% | Negative |
| Voxtral Small | 50.7% | 16.4% | 20.0% | 12.9% | Negative |
| Thinking Machines Inkling | 30.0% | 7.5% | 59.3% | 3.2% | Neutral |
| Inworld Voice | 16.8% | 14.6% | 67.5% | 1.1% | Neutral |
| OpenAI Realtime | 17.9% | 6.1% | 69.6% | 6.4% | Neutral |
5 models are negative-skewed. 5 are neutral-skewed. None is positive-skewed.
- Two models with similar aggregate accuracy can create different product risks if one over-predicts negative affect and another over-predicts neutral delivery.
- Output valence skew is descriptive: it summarizes what labels the model emits under this benchmark, not an intrinsic model property.
Current limitations
- English-only, General American accent. Multilingual and cross-accent evaluation requires additional collection.
- Seven discrete labels. Human affect is richer; a single class cannot capture every perceived state.
- Balanced 40-clip classes. This enables fair comparison but does not reflect real traffic distributions.
- Performed emotional speech. Results cannot be interpreted as evidence of spontaneous emotion detection.
- Single reviewer per clip. Inter-rater reliability analysis is not supported by this release.
Dataset and paper
The dataset, baseline predictions, evaluation harness, and documentation are released under the MIT License on Hugging Face. The paper describes the benchmark design, evaluation protocol, baseline setup, and full results.