OmniAssistBench

💡 Introduction

Recent advances in Omni-LLMs are paving the way for real-time video assistant applications, where models constantly perceive the environment and guide users to achieve certain goals through multi-turn conversations. However, evaluations under these assistant-style interaction scenarios are still challenging. OmniAssistBench aims at addressing this challenge by proposing an annotation pipeline which allows annotators to build test samples from existing Internet videos.

OmniAssistBench proposes an annotation pipeline which allows annotators to build test samples from existing Internet videos.

The main evaluation challenge lies in dataset construction: In multi-turn interaction with the assistant, user will do what the model suggests. This means that the model's unpredictable response dynamically changes the subsequent video contents, which static offline datasets cannot accommodate. As shown in Fig. (b) above, when the user asks for instructions on making latte coffee, the model may suggest extracting espresso first then frothing the milk, or vice versa. Although both suggestions are valid, static dataset may only contain subsequential videos of extracting espresso before frothing milk.

To address this challenge, OmniAssistBench proposes an annotation pipeline which provides the model with prior knowledges to enforce a fixed interaction path. As shown in Fig. (c) above, annotators first summarize the scource video into a list of steps to achieve the user goal. Then, multi-turn user questions and ground truth answers are designed. Scource videos are cut into clips based on the designed intercation process, whith prior knowledges embedded as on-screen subtitles and user questions embedded as audios. This pipeline allows annotators build evaluation data from raw Internet videos.

The data construction process of OmniAssistBench.

🧠 Benchmark Constructions

Task construction and statistics of OmniAssistBench.

• 🟢 Basic Tier: Core perception ablities that are necessary especially in interactive situations (e.g., social understanding, temporal perception, and gesture-based prompts).

• 🔵 Advanced Tier: Higher-level, user goal-driven interactive tasks derived from common applications (e.g., procedural guidance and proactive response).

<details>
<summary>
Click here for the examples of typical tasks in OmniAssistBench.</summary>

Examples of typical tasks in OmniAssistBench.

</details>

<details>
<summary>
Click here for the key plots of the real world cases filmed by our team. Each case is sepcially designed to cover a set of ablities evaluated in the benchmark.</summary>

Examples of key plots and questions from the three Real World Cases.

</details>

📊 Dataset Highlights

• 📈 Scale: 685 open-ended question-answer pairs covering 7 major task types and 16 fine-grained tasks. Video domains cover common daily topics such as sports, cooking, lectures, DIYs, and talk shows.
• 🎥 Custom-Filmed Real-World Cases: We design and film 3 cases with an average of over 15 interaction turns. Each case specifically targets the combinations of a group of the abilities evaluated in our benchmark to reflect genuine assistant usage.
• 🔊 Naturally Embedded Prompts: Unlike traditional benchmarks with text-only prompts, all user questions are embedded directly into the videos as realistic audio or typing/handwriting.
• ⏳ High Quality & Effort: Creating the dataset required careful choices of source videos and heavy video editing to balance video length and to embed the user prompts. Every sample was rigorously annotated by human experts. It takes 1000+ expert-hours to build the benchmark.

🏆 Leaderboard

OmniAssistBench requires candidate models to be capable of processing videos along with the corresponding audios. All models are graded using our LLM-as-a-Judge pipeline on a 0 - 5 scale. Reported scores have been normalized to 0-100.

OmniAssistBench is highly challenging. Current evaluations show that although most models can understand what to do, they struggle to provide correct and comprehensive responses.

📍Evaluation Pipeline

Our evaluation process is standardized into three steps:

1. Get Model Response: Because inference processes vary across different model, the exact evaluation code depends on the model. The result should be saved as a CSV file including two columns: video_name and output_text.

Key point is to include the following system prompt:

You are a helpful video assistant. For the above input video(s), if you decide to output, organize your output as if you are directly talking
to the user. Otherwise, if you decide to keep quiet, output exactly "[KEEP QUIET]".

We provide two example scripts of evaluating Gemini3 Pro via API and evaluating Qwen3-Omni via local inference. Please refer to the comments for handling multi-turn test samples:

• /eval_qwen3omni.py
• /eval_gemini3.py

2. LLM-as-a-Judge Scoring: We utilize an advanced LLM as a judge to grade the model's output_text from 0 to 5. The output of this step is a CSV file containing the video_name, video_class, and score.

Here is a brief summary of what each score represents. Please refer to score_rubric.txt for the detailed scoring criteria.

An example judgment script using GPT-5 is provided in/score.py.

3. Calculate Task Averages: Once the grading is complete, you can aggregate the results to see how the model performs across different tasks. We provide a script (/score_cont.py) that reads the judgment CSV and automatically calculates the average scores at sub-task level, major-task level and tier-level.

Citation

If you find this work helpful, please consider citing our paper:

BibTeX:

@misc{sun2026omniassistbench,
      title={OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs}, 
      author={Xianyun Sun and Chaoyou Fu and Zhengye Zhang and Feiyang Duan and Qingyuan Cao and Yonghui Niu and Sihang Yuan and Ge Zhang and Caifeng Shan},
      year={2026},
      eprint={2608.21360},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2608.21360}, 
}