Gemini 3.5 Flash
Benchmark dashboard
Vocal Affect Bench
A vocal emotion benchmark for evaluating whether emotion detection or omni models can identify expressed affect from raw speech without transcripts or metadata.
angry, disgusted, fearful, happy, neutral, sad, surprised
emotion detection and omni models
40 per emotion class
across all 7 models
uniform 7-class chance
Detecting the correct vocal emotion is critical to having the correct response. Vocal Affect Bench evaluates emotion detection accuracy in the latest SOTA models.
words spoken"that's fine"conceding, joking, or frustrated — the transcript is identical
what ASR capturestext onlytone, pace, pauses, and intensity are lost after transcription
what voice agents needexpressed affectwhich emotion was delivered, not inferred from the words
Leaderboard
Model results
Ranked by seven-way accuracy. Best model reaches 44.3% — roughly 3× the 14.3% random baseline, but still misses more than half of all clips. Inkling is a post-publication baseline; the paper remains a snapshot of the original release.
| Rank | Model | Acc. | Angry | Disgusted | Fearful | Happy | Neutral | Sad | Surprised |
|---|---|---|---|---|---|---|---|---|---|
| #1 | Gemini 3.5 Flash | 44.3% | 57.5% | 27.5% | 32.5% | 45.0% | 75.0% | 70.0% | 2.5% |
| #2 | Hume Prosody | 38.0% | 50.0% | 5.1% | 0.0% | 62.5% | 85.0% | 22.5% | 40.0% |
| #3 | Qwen3.5 Omni+ | 37.9% | 35.0% | 25.0% | 17.5% | 47.5% | 65.0% | 65.0% | 10.0% |
| #4 | Voxtral Small | 34.3% | 17.5% | 72.5% | 17.5% | 35.0% | 47.5% | 27.5% | 22.5% |
| #5 | Thinking Machines Inkling | 32.1% | 32.5% | 2.5% | 12.5% | 35.0% | 85.0% | 55.0% | 2.5% |
| #6 | Inworld Voice | 28.6% | 37.5% | 0.0% | 12.5% | 30.0% | 97.5% | 17.5% | 5.0% |
| #7 | OpenAI Realtime | 27.9% | 32.5% | 2.5% | 10.0% | 25.0% | 87.5% | 27.5% | 10.0% |
Class difficulty
Per-class recall and precision
Most models can detect neutral emotion correctly at 77.5% recall. The recall for detecting fearful and surprised is only 14.6% and 13.2%. A model that looks useful on neutral-heavy traffic can still fail on the states that matter most for escalation.
Recall — what fraction of true clips were correctly identified
Precision — when predicted, how often was it correct
Neutral bias
Top confusions
710 of 1,279 incorrect predictions collapse to neutral. The dominant error is over-predicting neutral rather than mis-labeling between non-neutral emotions.
Valence analysis
Coarse positive / neutral / negative accuracy
Instead of predicting the distinct seven classes of emotions, it is easier for models to detect positive, neutral and negative. The accuracy now sits at 48.9% on average, but still not great.
| Model | Neg. out. | Pos. out. | Neutral out. | Ambig. out. | Output skew |
|---|---|---|---|---|---|
| Gemini 3.5 Flash | 51.4% | 10.7% | 36.8% | 1.1% | Negative |
| Hume Prosody | 24.4% | 27.6% | 36.2% | 11.8% | Neutral |
| Qwen3.5 Omni+ | 42.9% | 12.9% | 41.8% | 2.5% | Negative |
| Voxtral Small | 50.7% | 16.4% | 20.0% | 12.9% | Negative |
| Thinking Machines Inkling | 30.0% | 7.5% | 59.3% | 3.2% | Neutral |
| Inworld Voice | 16.8% | 14.6% | 67.5% | 1.1% | Neutral |
| OpenAI Realtime | 17.9% | 6.1% | 69.6% | 6.4% | Neutral |
Dataset composition
Clip and class distribution
VocalAffectBench is a test-only evaluation dataset, not a training corpus. 280 clips across 7 balanced classes, sourced from acted and naturalistic recordings.
| Emotion | Clips | Minutes | Mean (sec) |
|---|---|---|---|
| angry | 40 | 17.1 | 25.7 |
| disgusted | 40 | 19.0 | 28.5 |
| fearful | 40 | 17.5 | 26.3 |
| happy | 40 | 22.8 | 34.2 |
| neutral | 40 | 15.6 | 23.3 |
| sad | 40 | 26.0 | 38.9 |
| surprised | 40 | 21.0 | 31.6 |
Scoring method
Evaluation protocol
Audio input
The model receives raw audio only — no transcript, label hint, or context metadata.
Emotion label output
The model must return one of seven canonical labels: angry, disgusted, fearful, happy, neutral, sad, or surprised.
Exact-match scoring
Seven-way accuracy, per-class precision/recall, and valence-bucket accuracy are computed against gold labels.