OmniAssistBench Exposes Weaknesses in Omni-LLMs

Researchers from Nanjing University have introduced OmniAssistBench, a novel evaluation framework designed to test omni-modal large language models in real-time video assistant scenarios. Unlike traditional static benchmarks, this tool evaluates a model's ability to continuously perceive its environment and guide users through multi-turn conversations to achieve specific goals. To build the dataset, the creators spent more than 1,000 expert person-hours reverse-engineering existing Internet videos, segmenting them into multi-turn clips, and establishing logical user goals.

To address the challenge of diverging interaction paths, where a single goal can be reached in multiple ways, the benchmark provides models with predefined priors. This forces the systems to guide users along specific, standardized routes. In initial evaluations, Google's proprietary Gemini-3-Pro model scored 66.4 out of 100. Meanwhile, the open-source Qwen3-Omni-Instruct model achieved a score of 51.2.

The benchmark's results highlight significant gaps in current AI capabilities. Although the tested models generally understood user inputs, they frequently delivered incorrect or incomplete guidance. Specifically, the models struggled to interpret visual prompts like hand gestures, failed to maintain historical context over the course of multi-turn interactions, and struggled to delay their responses until a target event actually occurred.

For AI practitioners and developers, OmniAssistBench provides a standardized diagnostic tool to move beyond passive video understanding. It shows that building a truly interactive assistant requires focusing on temporal context retention and fine-grained visual perception rather than just basic instruction-following. Developers can use this benchmark to target these specific failure modes, paving the way for more reliable, real-time agents in robotics, customer support, and hands-free tutorials.