A curated collection of benchmarks for omni-modal (audio + video) multimodal LLMs, plus the officially reported results of open omni models, plus reading notes on the papers themselves.
Everything here is compiled from official sources only — benchmark papers, official GitHub/HuggingFace releases, public repo issues, and model technical reports. Each entry links a per-benchmark page with task format, scale, the official evaluation protocol (when published), and officially reported scores with their sources.
benchmarks/
├── long-video-av/ long-form audio-visual omni benchmarks
├── av-qa/ short & medium clip audio-visual QA
├── long-video-visual/ long-video benchmarks (vision-only / subtitle-based)
├── social/ social & pragmatic understanding
├── emotion/ emotion & sentiment
├── action/ related action / video source datasets
└── models/ official model report cards
paper/ 39 papers — the details a first read misses
├── omni-models/ Qwen-Omni family, MiniCPM-o 4.5, AV-Flamingo, JavisGPT
├── omni-efficiency/ audio-visual token & KV-cache compression
├── adaptive-reasoning/ thinking budgets, latent reasoning, inverse scaling
├── frame-selection/ query-aware frame selection & test-time perception
├── modality-grounding/ cross-modal shortcuts, interference, preference repair
├── av-agents/ tool-orchestration agents & the Interspeech 2026 challenge
├── temporal-grounding/ long-audio and long-video localization
└── emotion-models/ AffectGPT series, Emotion-LLaMA v2, Emotion-Qwen
Paper notes — paper/
Not summaries. Each page records what a careful second pass turns up and a first read misses: the buried hyper-parameter, the protocol caveat that makes two numbers non-comparable, the ablation that was run and dropped, the claim a paper's own table does not support, the training cost nobody discloses. Official sources only.
| Benchmark | Modalities | Scale | Avg duration | Best reported |
|---|---|---|---|---|
| MMOU | A+V | 20,000 QA / 11,877 videos | ~8.7 min | Gemini-2.5-Pro 64.2 (human 84.3) |
| VideoOdyssey | A+V | 1,062 AV + 1,618 V QA / 100 videos | ~109 min | Gemini-3.1-Pro 46.1 (AV track) |
| TraceAV-Bench | A+V | 2,200 QA / 578 videos | 35.2 min | Gemini-3.1-Pro 68.29 |
| LVOmniBench | A+V | 1,014 QA / 275 videos | 34.5 min | Gemini-3.0-Pro 65.8 |
| OmniVideoBench | A+V | 1,000 QA / 628 videos | ~6.4 min | Gemini-3.0-Pro 61.8 (human 82.7) |
| LongInsightBench | A+V+T | 4,781 QA / 1,001 videos | ~9 min | Gemini-2.5-Flash 65.17 |
| Video-MME-v2 | A+V (mixed) | 3,200 QA / 800 videos | 10.4 min | — |
| WorldSense | A+V | 3,172 QA / 1,662 videos | ~2.35 min | Gemini 2.5 Pro 65.1 |
| Daily-Omni | A+V | 1,197 QA / 684 videos | ~45 s | Qwen3.5-Omni 84.6 |
| Video-Holmes | V (+A ablation) | 1,837 QA / 270 films | 1–5 min | MiniCPM-o 4.5 64.3 (w/ audio) |
| AVHBench | A+V | 5,302 QA (+1,106 captioning) | ~10 s | SFFL-tuned Qwen3-Omni-30B 81.29 |
| MUSIC-AVQA | A+V | 45,867 QA / 9,288 videos | ~60 s | AVST 71.59 |
| AVQA (Yang 2022) | A+V | 57,335 QA | ~10 s | HCRN+HAVF ~89 |
| OmniBench | A+Image+T | 1,142 QA | — | Omni-AutoThink-7B 0.69 (989-q subset) |
- MMOU — largest long AV omni benchmark; 10-choice MCQ with evidence timestamps (NVIDIA + UMD)
- VideoOdyssey — all videos >1 h; introduces Continuous Clue Length (CCL)
- TraceAV-Bench — multi-hop trajectory reasoning + cross-modal hallucination traps
- LVOmniBench — fully human-authored long AV QA (10–90 min videos)
- OmniVideoBench — modality-complementary QA with annotated reasoning chains
- LongInsightBench — human-centric long-video omni benchmark (FineVideo)
- Video-MME-v2 — Video-MME follow-up with mixed AV / V / A questions
- LongVALE — omni event localization + captioning (vision/audio/speech aligned)
- WorldSense — synchronized AV understanding, 26 task categories
- Daily-Omni — cross-modal temporal alignment over daily-life clips
- Video-Holmes — suspense-film reasoning (incl. social reasoning)
- AVHBench — audio↔visual cross-modal hallucination (ICLR 2025)
- MUSIC-AVQA — music-performance AVQA with modality-typed questions (CVPR 2022)
- FortisAVQA — robustness extension of MUSIC-AVQA (head/tail + rephrasings)
- AVQA (Yang et al. 2022) — VGGSound-based MCQ with modality-dependency labels
- AVUT / ACVUBench — audio-centric QA designed to remove text shortcuts
- OmniBench — tri-modal (text + audio + static image) MCQ
- AV-Odyssey — short-clip AV MCQ
- JointAVBench — medium-short clip AV MCQ
- Video-MME — short/medium/long tiers, w/ and w/o subtitles
- MLVU — multi-task long video understanding (CVPR 2025)
- LVBench — every video >30 min (ICCV 2025)
- CG-Bench — clue-grounded long-video QA (ICLR 2025)
- LongVideoBench · HourVideo · MoVQA · EgoSchema · CinePile · MMBench-Video · VideoChatGPT-Bench · ActivityNet-QA · MovieChat-1K · NExT-QA
- SocialOmni — turn-taking who/when/how axes with an AV-inconsistency probe
- Social-IQ 2.0 — base MCQ social intelligence dataset
- Social Genome — grades the reasoning trace itself; vocal cues as evidence
- Social Caption — MCQ + generative social description on Social-IQ 2.0
- SIV-Bench — social scene / state / dynamics over TikTok-YouTube clips
- R³-VQA — belief / intent / desire / emotion with social-causal chains
- MimeQA — nonverbal Theory of Mind on silent mime videos (NeurIPS 2025)
- HumanVBench — human-centric 16 sub-tasks incl. AV alignment
- EgoIntent — egocentric step-level What / Why / Next intent
- MUStReason — sarcasm reasoning with per-modality cue annotations (LREC 2026)
- RivaBench — reasoning-intensive AV incl. stand-up comedy understanding
- MER-NOISE (MER2023/2024) — robustness track with official corruption protocol
- MER2025 Track 1 · MER2026 — MER challenge series
- CA-MER — controlled audio-visual conflict subsets (modality bias)
- EMER · OV-MER · EmoPrefer — explainable / open-vocabulary / preference emotion (AffectGPT series)
- MELD · IEMOCAP · RAVDESS · DFEW · MAFW — classic emotion corpora
- CH-SIMS v2.0 — Chinese sentiment with modality-isolated labels
- MME-Emotion · EmoBench-M — aggregated emotional-intelligence benchmarks for MLLMs
- AV-EMO-Reasoning — spoken-response emotion reasoning (speech out)
Kinetics-400/700 · AVA · Charades · ActivityNet v1.3 & ActivityNet-QA
- Qwen3-Omni — full benchmark tables from the technical report (arXiv:2509.17765) + published third-party evaluations
- Qwen2.5-Omni — full benchmark tables from the technical report (arXiv:2503.20215)
- Scores are quoted exactly as reported by the cited source (paper table, official leaderboard, repo issue, or model tech report); different sources use different settings and are generally not directly comparable.
- Contributions welcome — please cite an official source (paper / official repo / leaderboard) for every number added.