Back

VocalAffectBench: Can AI Hear How You Feel?

Jul 7, 2026Benchmark

A public, test-only benchmark for evaluating whether AI audio models can identify expressed vocal emotion from raw speech — without transcripts or contextual metadata. Ten baselines. Seven emotion classes. One finding: affect recognition remains brittle.

Live leaderboardOpen Hugging FaceRead Paper

Overview

Emotion matters

A customer who says "that's fine" may be conceding, joking, or signaling frustration. Automatic Speech Recognition (ASR) models preserve the phrase but strip the vocal delivery that makes the exchange interpretable. Voice agents that apply affect analysis only after transcription measure lexical sentiment — a different question from expressed vocal emotion.

A number of omni and emotion-detection specific models promise to solve this problem. Given an audio clip, these models should be able to detect the correct emotion. How accurate are their predictions?

words spoken"that's fine"spoken with emotions like conceding, joking, or frustrated
what ASR capturestext onlytone, pace, pauses, and intensity are lost after transcription
what voice agents needexpressed affectwhich emotion was delivered, not inferred from the words

What VocalAffectBench contains

280 human-recorded English clips from 52 speaker accounts, equally distributed across seven emotion classes and sourced from acted and naturalistic recordings. All clips are General American accent, 16 kHz mono WAV, with no noise reduction, normalization, or enhancement.

MetricValueDetails
Audio clips28040 clips per emotion class
Emotion classes7angry, disgusted, fearful, happy, neutral, sad, surprised
Speaker accounts52Human-recorded English WAV, General American accent
Total audio2.32 hrs138.9 minutes of reviewed speech
Models evaluated10General audio models, speech-specific models, and provider endpoints
Random baseline14.3%Chance level for a 7-way classification task

Benchmark details

Benchmark design

Every model receives the raw audio file and a fixed instruction listing the allowed labels. No transcript, no speaker context, no domain hint. For provider endpoints that do not accept prompts, the audio is submitted with default settings and the native labels are mapped to the benchmark set before scoring.

FieldValueDetails
TargetExpressed emotionThe emotion delivered in the recording, verified by a human reviewer. Not the speaker's inferred private state.
Label set7 classesangry, disgusted, fearful, happy, neutral, sad, surprised — Ekman's basic-emotion taxonomy plus neutral.
MetricAccuracyFraction of clips whose mapped prediction matches the reference label. Per-class precision and recall expose failure modes.

Scoring method

The model receives the WAV file and a fixed instruction listing the seven allowed labels. No transcript, metadata, or context is provided. Provider-native labels (e.g. calmness → neutral, anxiety → fearful) are mapped to the benchmark set using the released mapping before scoring. Accuracy, per-class precision and recall, confusion counts, and valence accuracy are computed from the mapped predictions and stored in predictions.csv.

Dataset composition

EmotionClipsMinutesMean (sec)
Angry4017.125.7
Disgusted4019.028.5
Fearful4017.526.3
Happy4022.834.2
Neutral4015.623.3
Sad4026.038.9
Surprised4021.031.6

Results

Main findings

  1. 01

    Hopper Audio Perception-1 leads at 59.6% (167/280), ahead of Oruk Resonance 2 at 49.3%. It still misses about two in five clips.

  2. 02

    Neutral is over-predicted: 43.6% of all scored outputs map to neutral, causing most non-neutral errors.

  3. 03

    Surprised and fearful are the hardest classes, with 18.0% and 18.3% aggregate recall across baselines.

  4. 04

    Valence (positive / neutral / negative) is more tractable at 55.2% aggregate accuracy, but still unsolved.

Public release standings

RankModel
1Hopper Audio Perception-159.6%
2Oruk Resonance 249.3%
3Gemini 3.8 Flash45.7%
4Gemini 3.5 Flash44.3%
5Hume Prosody38.0%
6Qwen3.5 Omni+37.9%
7Voxtral Small34.3%
8Thinking Machines Inkling32.1%
9Inworld Voice28.6%
10OpenAI Realtime27.9%

Not all emotions are equally hard.

Neutral is the easiest class by recall at 80.8%, but the hardest by precision at 26.5% — because models over-predict it. Surprised and fearful reach only 18.0% and 18.3% recall. A model that looks useful on a neutral-heavy workload can still fail on the high-salience states that matter most for escalation or intervention.

Recall — what fraction of true clips were correctly identified

Emotion
Recall
Neutral80.8%
Happy48.0%
Sad45.0%
Angry41.8%
Disgusted26.6%
Fearful18.3%
Surprised18.0%

Precision — when predicted, how often was it correct

Emotion
Precision
Neutral26.5%
Happy49.9%
Sad41.3%
Angry61.6%
Disgusted48.6%
Fearful81.1%
Surprised40.2%

The dominant error is predicting neutral.

Across all scored decisions, models output neutral 1,220 times — 43.6% of all outputs. Of the 1,686 incorrect predictions, 897 collapse to neutral. The largest single confusion is disgusted → neutral (168 times), followed by fearful → neutral (167) and sad → neutral (163).

A conservative neutral prediction can make a voice product miss the affective states that should trigger escalation, empathy, or additional caution.

Actual → predicted
Count
Disgusted → Neutral168
Fearful → Neutral167
Sad → Neutral163
Surprised → Neutral146
Happy → Neutral127
Angry → Neutral126
Fearful → Sad93
Surprised → Happy88

Valence analysis

Some applications only need a positive / neutral / negative signal rather than a seven-way label. Mapping the benchmark to three valence buckets (excluding surprised, whose valence is ambiguous) raises aggregate accuracy to 55.2%. The best model, Hopper Audio Perception-1, reaches 80.8%. Valence is more tractable than discrete emotion — but it is not solved.

RankModel
1Hopper Audio Perception-180.8%
2Gemini 3.5 Flash66.3%
3Oruk Resonance 265.0%
4Gemini 3.8 Flash64.2%
5Voxtral Small55.0%
6Qwen3.5 Omni+52.5%
7Hume Prosody47.7%
8Thinking Machines Inkling47.1%
9Inworld Voice38.3%
10OpenAI Realtime35.4%
ModelNeg. out.Pos. out.Neutral out.Ambig. out.Output skew
Hopper Audio Perception-151.4%13.2%22.9%12.5%Negative
Oruk Resonance 238.2%13.6%37.5%10.7%Negative
Gemini 3.8 Flash38.9%15.0%44.3%1.8%Neutral
Gemini 3.5 Flash51.4%10.7%36.8%1.1%Negative
Hume Prosody24.4%27.6%36.2%11.8%Neutral
Qwen3.5 Omni+42.9%12.9%41.8%2.5%Negative
Voxtral Small50.7%16.4%20.0%12.9%Negative
Thinking Machines Inkling30.0%7.5%59.3%3.2%Neutral
Inworld Voice16.8%14.6%67.5%1.1%Neutral
OpenAI Realtime17.9%6.1%69.6%6.4%Neutral

5 models are negative-skewed. 5 are neutral-skewed. None is positive-skewed.

  1. Two models with similar aggregate accuracy can create different product risks if one over-predicts negative affect and another over-predicts neutral delivery.
  2. Output valence skew is descriptive: it summarizes what labels the model emits under this benchmark, not an intrinsic model property.

Current limitations

  • English-only, General American accent. Multilingual and cross-accent evaluation requires additional collection.
  • Seven discrete labels. Human affect is richer; a single class cannot capture every perceived state.
  • Balanced 40-clip classes. This enables fair comparison but does not reflect real traffic distributions.
  • Performed emotional speech. Results cannot be interpreted as evidence of spontaneous emotion detection.
  • Single reviewer per clip. Inter-rater reliability analysis is not supported by this release.

Dataset and paper

The dataset, baseline predictions, evaluation harness, and documentation are released under the MIT License on Hugging Face. The paper describes the benchmark design, evaluation protocol, baseline setup, and full results.

Live leaderboardOpen Hugging FaceRead Paper