ASI-Bench

Paper

ASI-Bench: At the Dawn of Artificial Superintelligence

arXiv:2608.17271

Figure 1: Overview of ASI-Bench. Left: B3 performance across agents. Right: scores from B1 to B4, where B1 provides full methods, B2 only the method name, B3 only the research goal and data, and B4 further adds distractors.

Overview

ASI-Bench is the first benchmark to jointly evaluate general intelligence,
innovation, and autonomous execution, and the first to progressively withdraw
human methodological guidance within the same research project. It tests how
far an AI agent can independently select methods, conduct end-to-end research,
and produce verifiable scientific results instead of merely following a
human-specified procedure.

The scientific objective, input data, required artifacts, and scoring criteria
remain fixed across four matched guidance conditions:

These are long-horizon investigations rather than isolated questions. Across
the 60 tasks, complete research trajectories involve more than 2,600 interaction
turns and 2,400 execution steps, spanning over 35 hours of agent execution.

The benchmark was distilled from more than 1,300 candidate research ideas
through five review rounds, over 1,100 review assignments, more than 2,000 task
revisions, and over 1,500 sandbox runs. Its construction and validation involved
more than 31,000 human-hours.

Resources

Quick start

1. Install ASI-Bench

pip install asibench
asibench --help

Requirements: Python 3.11 or later. The default installation is lightweight
and includes everything needed to download public benchmark instances. For
local agent execution or task authoring, install the optional full stack:

asibench is the canonical CLI. The legacy ai4sci-bench command remains only
as a backwards-compatibility alias.

pip install 'asibench[full]'

2. Download a benchmark set

Each fixed-seed dataset contains its own prompts and input data. seed31415
also publishes references for local scoring; seed42 keeps references private.
Choose one seed for each run:

mkdir asi-bench-run
cd asi-bench-run

asibench task pull \
  --repo seed31415 \
  --output-dir hf_instances_seed31415/

3. Run an agent

asibench run \
  --instances-dir hf_instances_seed31415/ \
  --agent-cmd 'python my_agent.py --workspace {workspace}' \
  --sandbox linux_ns \
  --output-dir out_seed31415/

The command above evaluates all four prompt levels (b1,b2,b3,b4). Use
--prompt-levels only when intentionally running a subset.

4. Score locally or submit

seed31415 can be scored locally with its public references and the scoring
contracts in this GitHub checkout:

asibench score \
  --repo seed31415 \
  --results-dir out_seed31415/ \
  --instances-dir hf_instances_seed31415/ \
  --tasks-dir /path/to/ASI-Bench/tasks/

The command writes a separate local_score_seed31415.json and never overwrites
the produce-only result. Local scores are reproducible but non-official.

For seed42, repeat the pull and run with separate directories, then submit for
private-reference scoring:

asibench login
asibench submit --results-dir out_seed42/ \
  --benchmark-repo Apexintelligence-AI/ASI-Bench-seed42

submit creates an authenticated draft on the ASI-Bench website. Review the
completeness summary and confirm the draft to enter the official scoring queue.
Only seed42 result directories are accepted: the CLI checks every instance ID
before building a bundle or authenticating, and rejects seed31415, unknown, or
mixed-seed results. If --benchmark-repo is supplied, it must identify the
official seed42 dataset.
asibench score --repo seed42 is rejected before reading local inputs because
seed42 GT is not public. Local benchmark runs never calculate official scores.

Run configuration

Platform note: The examples use Linux Bash, and the linux_ns sandbox
requires Linux. Windows users should run it through WSL2. PowerShell and
Command Prompt use different line-continuation and JSON-escaping syntax.

See Getting Started for agent configuration,
sandbox selection, and platform-specific commands.

Package and CLI compatibility

• asibench is the only published Python distribution and the canonical CLI.
• Existing Python integrations may continue to import ai4sci_bench.
• ai4sci-bench remains available as a legacy CLI alias.
• Benchmark instances are downloaded separately with asibench task pull.
• Runtime and declared-output metadata come from tasks/ or --tasks-dir.
• Retired internal orchestration commands are not part of the public CLI. Use
  run, report, batch-report, review, and the Portal workflows.

Built-in CLI agents — installation

Install the CLI agent you want to evaluate, then select its ASI-Bench adapter
with --agent.

• New model, existing harness. When a supported harness such as Codex CLI,
  Claude Code, or Kimi Code adds a model, select the new model through
  --agent-config. If the model is served through a third-party endpoint and
  the adapter supports that route, also set api_base, api_key, and
  api_protocol (openai or anthropic). No ASI-Bench code change is normally
  needed while the harness accepts the model and the endpoint remains protocol
  compatible.
• New agent harness. A new CLI or runtime can be used immediately through
  --agent-cmd when it can read the prepared workspace and write result files;
  this generic mode supports none and linux_ns. First-class built-in
  integration—its own --agent name, harness-specific authentication and
  configuration, proxy wiring, and Docker-based os support—requires ASI-Bench
  maintainers to implement and test a new adapter.

Example with Kimi Code:

asibench run --agent kimi_code_cli \
  --agent-config '{"model": "kimi-k2.7"}' \
  --output-dir out/

Framework-managed model proxies use text-only input by default. For endpoints
that accept images, set "supports_image_input": true in --agent-config.
Without this flag, image attachments are replaced with a short notice before
the next model request. Direct endpoints outside the framework proxy remain the
responsibility of the connected agent.

Contribute a task

We welcome project-level scientific tasks from the research community. Authors
can draft a task locally or in the Portal, validate it, and submit a frozen
revision for review.

Quick start for task authors

# 1. Create a task scaffold
asibench task create --domain physics --name my_new_task

# 2. Edit the generated files in tasks/physics/my_new_task/:
#    - task_meta.yaml   — task metadata (public)
#    - task_eval.yaml   — evaluation and generation config during authoring
#    - task_submission.yaml — Portal-only author evidence (private; never exported)
#    - generate_gt.py   — ground-truth generator (private)
#    - custom_scorer.py — custom scoring logic, if needed
#    - prompt_b1.md … prompt_b4.md — local starter prompts for the guidance gradient

# 3. Required pre-submit validation
asibench validate --pre-submit tasks/physics/my_new_task/

# 4. Required same-model trial across B1, B2, B3, and B4
asibench difficulty-check --task physics.my_new_task

# 5. Upload an exact Draft snapshot, then review and submit it in the Portal
asibench task submit --task-dir tasks/physics/my_new_task/

Submission requirements

• A complete task package must define the scientific objective, B1–B4 prompts,
  agent-visible inputs and required artifacts, reproducible reference generation,
  evaluation gates and weighted scorers, runtime dependencies, and local-testing
  evidence.
• Complete validate --pre-submit and a same-model difficulty-check across
  B1–B4 before submission.
• Use lowercase letters, digits, and underscores for task IDs. The canonical
  form is domain.task_name.
• asibench task submit --task-dir ... exact-syncs the local files to an
  owner-only Portal Draft. It verifies the returned relative paths and SHA-256
  hashes, and prints the recoverable Draft URL if synchronization fails. It
  never sends the Draft for review by itself.
• On first use, asibench login opens Settings → CLI tokens. Create a PAT,
  copy it, and paste it into the hidden prompt; the CLI validates and saves it
  in ~/.asibench/credentials with mode 0600. Later submissions reuse it.
• Set ASIBENCH_NO_BROWSER=1 to print URLs without opening a browser. For CI,
  set ASIBENCH_SUBMIT_TOKEN; tokens are intentionally not accepted as command
  arguments so they do not enter shell history.
• The Portal requires local_testing_done and one finite score for each level
  before freezing a revision.
• Authors do not create repository pull requests. Administrators publish
  accepted tasks after review.

The Portal also provides a 15-step Guided Flow for constructing a Task entirely
in the browser:
asibench.apexin.ai/submit/proposals/new.
Use --endpoint or ASIBENCH_SUBMIT_ENDPOINT to select another Portal.

Runnable samples from the official benchmark

The following final tasks can be used to verify the runner workflow:

Pull and run one in produce-only mode:

asibench task pull \
  --repo seed42 \
  --tasks astronomy.nbody_close_encounters \
  --output-dir example_instances/

asibench run \
  --instances-dir example_instances/ \
  --tasks astronomy.nbody_close_encounters \
  --agent-cmd 'python my_agent.py --workspace {workspace}' \
  --sandbox linux_ns \
  --output-dir example_results/

asibench submit --results-dir example_results/

For official benchmark tasks, GitHub publishes the catalog metadata, scoring
configuration, and scorer implementations for auditability. Prompts and input
data come from Hugging Face. Ground-truth generators, generator settings,
reference specifications, and private solver assets remain exclusively on the
scoring service and are not present in this repository. seed42 reference
answers are private. seed31415 reference answers are intentionally public on
Hugging Face, so its GitHub scorers can produce reproducible local,
non-official scores.

Five opt-in sample tasks are fully public examples. Their B1–B4 prompts,
ground-truth generators, and scorer implementations or configurations are
available under tasks/:

• robotics.minimum_snap_trajectory_conditioning
• chemistry.bsse_counterpoise_cbs_extrapolation
• materials.phonon_dispersion
• medicine.ethics_disclosure_diagnosis_1670
• medicine.jama_id0014_malignant_an

The examples were synchronized from
apexin-ai/Agent-AI4Sci-Bench
at revision 2ba9258442bf53ad6c4911957234e03e767476ad.

Dataset layout

Pull the two fixed-seed datasets separately:

asibench task pull --repo seed42 --output-dir hf_instances_seed42/
asibench task pull --repo seed31415 --output-dir hf_instances_seed31415/

seed42 and seed31415 are the only official Hugging Face contracts, and
--repo is required. They share task schemas and declared outputs but contain
different generated inputs. Matching schemas live in
tasks/**/task_meta.yaml; matching public scoring contracts live in
task_eval.yaml and optional custom_scorer.py files. seed31415 additionally
contains public reference/ directories. Private manifests, ground-truth
generators, generator settings, seed42 references, and solver assets stay on
the scoring side.

The task workflow uses the grouped asibench task create, task pull, and
task submit commands. See Contribute a Task
for the complete authoring guide.

How scoring works (and stays fair)

• Public execution framework: agent adapters, sandboxes, instance loading,
  output collection, and submission packaging are auditable in this repository.
• Public local scoring: seed31415 publishes references on Hugging Face and
  uses the GitHub scoring configuration/custom scorers through asibench score.
• Private official scoring: seed42 never publishes references;
  asibench submit accepts only seed42 outputs and sends them to the website
  for authenticated scoring.
• Score status: seed31415 local reports are explicitly non-official;
  its results cannot be packaged by asibench submit, and self-reported scores
  are not accepted as leaderboard results.

Sandboxing & reproducibility

• Runs support task, os, and linux_ns sandboxes with pinned per-task
  runtimes.
• Every run records agent, model, effort, sandbox, and framework provenance.
• Source checkouts are installed editable into task environments; wheel-based
  runs install the matching asibench version.
• Mutable caches live under ~/.asibench/, never under site-packages.
• Namespace and Docker prerequisites are checked before execution. Missing
  prerequisites produce a non-zero CLI exit.

Documentation

Installation, execution, submission, task contribution, catalog, and
leaderboard documentation are available on the
ASI-Bench website.

License

See LICENSE.