Promote Sign in Sign up

Repository

opencode-zen-bench

smoke test for free LLM models run through . It runs each model through five fixed coding tasks, scores them with deterministic gates (no human judgment in the loop), and emits a C · License MIT · Maintenance unknown

virgiliojr94
Repository
License
MIT
Category
Task
C1
C1
C2
C2

Specs

  • CategoryTask
  • C1C1
  • C2C2
  • C3C3
  • C4C4
  • HARDHARD

README

opencode-zen-bench

A small, objective smoke test for free LLM models run through
OpenCode. It runs each model through five fixed coding
tasks, scores them with deterministic gates (no human judgment in the loop), and
emits a CSV + summary. A separate clean Claude session then reads only the
numbers and picks the best model per activity — so the verdict is unbiased.

Why

"Which free model is best?" is usually answered by vibes. This answers it with
identical prompts, pass/fail gates, and latency, so the comparison is fair and
reproducible on any machine.

What it tests

Category Task Gate (objective)
C1 research: latest stable Go version, fixed format output matches VERSION=… URL=http…
C2 boilerplate: write pytest tests for a function the tests exist, cover the empty case, and pass
C3 debug: fix a buggy scorer.py pytest goes green
C4 review: spot a real bug in a diff answer starts BUG: and names the right cause
HARD debug: fix merge_intervals (8 edge cases) pytest goes green

Prerequisites

  • OpenCode installed and authenticated for the providers
    you want to test (opencode auth login).
  • python3 (the runner creates a local .venv with pytest automatically).
  • A POSIX shell with bash, grep, timeout, column.

Setup

  1. List the free models on your station and fill models.txt (one
    provider/model per line):

    opencode models            # everything you're authed for
opencode models opencode   # the OpenCode Zen catalog

    Put the exact IDs into models.txt. The placeholders shipped there are
    guesses — wrong IDs are recorded as invoke-error, not a real score.

  2. Clean environment matters (avoid false positives). This repo ships an
    opencode.json with "permission": "allow" so headless edits never stall on
    a permission prompt. It defines no MCP servers — and MCP tool schemas can
    eat tens of thousands of tokens and even break some models. If your global
    OpenCode config loads MCP servers, disable them for the run, or set
    OC_FLAGS=--pure after confirming that flag's exact position with
    opencode run --help on your station.

Run

./run.sh                 # full matrix: every model × every category, N=2 reps
N=3 TIMEOUT=900 ./run.sh # more reps / longer timeout for slow free models
./consolidate.sh         # -> results/summary.md (pass-rate + median latency)

results/runs.csv has every run (model,class,run,pass,wall_ms,note) and each
run's raw output is saved under work/<model>_<class>_<n>/output.txt for audit.

Judge it (clean session)

The runner never decides a winner. Follow HANDOFF.md: open a fresh
Claude session, paste the neutral prompt + summary.md + runs.csv, and let it
rank best-per-activity strictly from the data.

Fairness notes (read before trusting a result)

  • Identical prompts across models (prompts.sh). Don't tweak between runs.
  • C1 needs a web/fetch tool. A model with none will FAIL C1 — that's "no web
    access", not "bad model". The clean-session prompt is told to flag this.
  • Small samples. Default N=2; bump it for a real verdict.
  • invoke-error rows are setup/availability failures (wrong ID, not authed,
    timeout), not capability — exclude them from quality judgments.

MIT.