MoSciBench

# MoSciBench: Multimodal Scientific Discovery Benchmark [](https://www.python.org/downloads/) [](#data-information) [](#repository-layout-brief) [](LICENSE)

              

A practical benchmark framework for multimodal scientific discovery.

News

• 2026-03-01: Official benchmark release of MoSciBench. We welcome issues and will maintain an open leaderboard to continuously improve LLM agents for multimodal scientific discovery.
  官方发布 MoSciBench 基准。欢迎大家提交 issues；我们会维护一个开源榜单，致力于持续提升 LLM agents 在多模态科学发现中的能力。

TL;DR

MoSciBench is the first benchmark focused on end-to-end multimodal scientific discovery with LLM agents. Built from peer-reviewed studies via a four-stage pipeline, it spans 6 scientific domains, 7 data modalities, 5 discovery question types, and 88 tasks. Each task requires cross-modal hypothesis verification across heterogeneous data.

Problem Setting

Given a scientific hypothesis and its associated multimodal dataset, the agent must generate executable analysis code, run it on raw data, interpret the resulting numerical outputs, and determine whether the hypothesis is supported or refuted.

Requirements

• Python 3.9+ (recommended 3.10/3.11)
• Access to an LLM service (configure in config/config2.yaml)

Install

pip install -r MoSciBench/requirements.txt

Optional extras for file previews:

• openpyxl for Excel, pyarrow for Parquet

Configure Models

cp MoSciBench/config/config2.example.yaml MoSciBench/config/config2.yaml
# Edit MoSciBench/config/config2.yaml and fill in api_key/base_url/model entries

Tip: You can point to a custom config via env var MOSCI_CONFIG (if supported in your deployment).

Download Datasets

• List datasets:

  python MoSciBench/scripts/utils/download_data.py --list

• Download and extract (example):

  python MoSciBench/scripts/utils/download_data.py health_spa

• Useful flags: --status, --no-extract, --keep-zip, --force

Dataset Access

The datasets are also available for direct download at Google Drive.

Quick Start (Baseline)

• Single run:

  python MoSciBench/run.py --dataset health_spa --baseline_name ReAct

• Multiple runs (for statistics):

  python MoSciBench/run.py --dataset health_spa --baseline_name ReAct --num_runs 3

Common flags (selection):

• --dataset: dataset name (hyphens normalized to underscores)
• --baseline_name: one of ReAct, DataVoyager, Reflexion, NoDataGuess
• --optimized_path: output root (default workspace)
• --initial_round_path: workflows source (default agents)
• --exec_model_name: model key defined in config2.yaml
• --include_workflow / --include_domain_knowledge: include extra context in Analysis prompt
• --analysis_timeout: execution timeout in seconds (default 1800)
• --max_retries: max code refine retries (default 3)

Outputs

• Logs: logs/
• Trajectory: trajectory/<dataset>/baseline_<baseline>_<timestamp>.jsonl
• Results: workspace/<baseline>/<dataset>/workflows/round_* (configs, CSVs, operator artifacts)

Repository Layout (brief)

• agents/: baselines and workflow templates (Analysis / CodeGenerate / CodeExecute / CodeRefine)
• benchmarks/: evaluation base classes and helpers
• config/: model configs (config2.yaml)
• data/: dataset root (download/unzip here)
• scripts/: runtime core (async LLM, logs, trajectory, workspace setup, downloader)
• workspace/, logs/, trajectory/: generated during runs

Data Information

License

See MoSciBench/LICENSE.

⭐ Star History

Citation

@inproceedings{liutowards,
  title={Towards Multimodal Data-Driven Scientific Discovery Powered by LLM Agents},
  author={Liu, Fan and Zeng, Xiaozhao and Liu, Hao},
  booktitle={The Fourteenth International Conference on Learning Representations}
}