Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey
Paper
• 2505.15957 • Published
• 3
Evaluations of Large Audio-Language Models (LALMs)
This collection contains papers for various LALM evaluation frameworks.
Note Our survey paper associated with this collection. We provide a detailed overview of the current progress and potential future direction in the field of LALM evaluations. Please consider to cite our paper if you find this collection helpful.
-
Roadmap towards Superhuman Speech Understanding using Large Language Models
Paper • 2410.13268 • Published • 33 -
StressTest: Can YOUR Speech LM Handle the Stress?
Paper • 2505.22765 • Published • 17 -
Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks
Paper • 2411.05361 • Published • 5 -
AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension
Paper • 2402.07729 • Published -
AudioBench: A Universal Benchmark for Audio Large Language Models
Paper • 2406.16020 • Published -
MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models
Paper • 2408.01337 • Published • 11 -
FinAudio: A Benchmark for Audio Large Language Models in Financial Applications
Paper • 2503.20990 • Published • 19 -
Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialogue
Paper • 2409.04927 • Published -
Advancing Singlish Understanding: Bridging the Gap with Datasets and Multimodal Models
Paper • 2501.01034 • Published -
AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?
Paper • 2412.02611 • Published • 25 -
Audio-FLAN: A Preliminary Release
Paper • 2502.16584 • Published • 36 -
QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions
Paper • 2503.20290 • Published • 1 -
SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning
Paper • 2504.09081 • Published • 16 -
Listen, Think, and Understand
Paper • 2305.10790 • Published • 2 -
Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering
Paper • 2204.09634 • Published -
ASR-EC Benchmark: Evaluating Large Language Models on Chinese ASR Error Correction
Paper • 2412.03075 • Published -
NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics
Paper • 2411.07186 • Published • 1 -
CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following
Paper • 2506.12285 • Published • 54 -
BLAB: Brutally Long Audio Bench
Paper • 2505.03054 • Published • 1 -
Textually Pretrained Speech Language Models
Paper • 2305.13009 • Published • 4 -
S2SBench: A Benchmark for Quantifying Intelligence Degradation in Speech-to-Speech Large Language Models
Paper • 2505.14438 • Published • 1 -
MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
Paper • 2410.19168 • Published • 24 -
VoiceBench: Benchmarking LLM-Based Voice Assistants
Paper • 2410.17196 • Published -
VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models
Paper • 2501.04962 • Published -
URO-Bench: A Comprehensive Benchmark for End-to-End Spoken Dialogue Models
Paper • 2502.17810 • Published -
Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models
Paper • 2412.05167 • Published -
MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark
Paper • 2506.04779 • Published -
MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix
Paper • 2505.13032 • Published • 4 -
CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models
Paper • 2310.08753 • Published -
SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Information
Paper • 2505.13237 • Published • 1 -
Audio Entailment: Assessing Deductive Reasoning for Audio Understanding
Paper • 2407.18062 • Published -
GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities
Paper • 2406.11768 • Published • 24 -
Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems
Paper • 2505.15000 • Published -
Multi-Domain Audio Question Answering Toward Acoustic Content Reasoning in The DCASE 2025 Challenge
Paper • 2505.07365 • Published -
Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations
Paper • 2402.12786 • Published -
SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words
Paper • 2406.13340 • Published -
Mind the Gap! Static and Interactive Evaluations of Large Audio Models
Paper • 2502.15919 • Published • 4 -
Does Your Voice Assistant Remember? Analyzing Conversational Context Recall and Utilization in Voice Interaction Models
Paper • 2502.19759 • Published • 1 -
S2S-Arena, Evaluating Speech2Speech Protocols on Instruction Following with Paralinguistic Information
Paper • 2503.05085 • Published • 47 -
AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large Language Models
Paper • 2505.16211 • Published • 18 -
Tune In, Act Up: Exploring the Impact of Audio Modality-Specific Edits on Large Audio Language Models in Jailbreak
Paper • 2501.13772 • Published -
Best-of-N Jailbreaking
Paper • 2412.03556 • Published -
Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models
Paper • 2505.15406 • Published • 5 -
The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio
Paper • 2410.12787 • Published • 30