workbuddy-bench

English · 简体中文

Overview

WorkBuddy Bench is a benchmark for evaluating coding agents on real-world work,
reverse-engineered from actual developer, PM, algo, QA, ops, and security tasks.
Given a task and a sandboxed workspace, an agent is asked to produce the correct
change (a patch, an artifact, a report) and is graded against a test suite.

The benchmark ships four subsets:

This repository is the benchmark's evaluation framework. It drops an agent
CLI (a harness) into a local Docker sandbox, runs it against a batch of tasks,
captures what happened (patches, trajectories, test results, efficiency), and
scores the run. It is built on top of
Harbor.

Quick start

The easiest way to run WorkBuddy Bench is to let a coding agent drive the whole
flow for you. This repo ships a wbbench-run-setup skill (under
.agents/skills/, also linked at .claude/skills/) that takes
you end-to-end — from an unconfigured checkout to a running, analyzed evaluation:
datasets, environment, model config, .env credentials, the job file, launching
the run, and finally the analysis report. Open the repo in a skill-aware agent
(Claude Code, CodeBuddy Code, …), invoke the skill (e.g. /wbbench-run-setup) or
simply ask it to set up a WorkBuddy Bench run, and it guides you one step at a
time.

Prefer to configure things by hand — or want to see what the skill is doing under
the hood? The Setup and Usage sections below walk through the same steps
manually.

Setup

Every task runs in Docker for reproducibility. You need Python ≥ 3.12,
uv, and Docker.

uv sync            # install dependencies
cp .env.example .env

The task datasets are not in this repo. They live on HuggingFace at
tencent/workbuddy-bench
as one archive per subset. Download the subsets you want into datasets/:

./scripts/dataset/fetch-dataset.sh code   # or: web / office / sec / all

This fetches, checksum-verifies, and extracts each subset to
datasets/wb-bench-<subset>-v1.0/. See datasets/README.md
for the archive list, sizes, and the WB_BENCH_HF_REPO override.

Point the framework at a model. Copy the template and fill it in:

cp configs/models/_template.model.yaml configs/models/<provider>/<slug>.yaml

model:
  name: Hy3                         # the model id your backend expects
  protocols: [openai]               # openai or anthropic
  backend_url_env: MODEL_BASE_URL   # the .env var holding the base URL
  backend_key_env: MODEL_API_KEY    # the .env var holding the API key

backend_url_env and backend_key_env name the env vars; put the actual URL and
key in .env, where secrets stay out of version control.

Usage

A job is one evaluation run: a model, a harness, and a dataset brought
together. Copy the template and fill it in:

cp configs/jobs/_template.job.yaml configs/jobs/<slug>.yaml

model: <provider>/<slug>                  # a model under configs/models/
harness: codebuddy-code/<version>         # the agent CLI to evaluate
dataset: datasets/wb-bench-code-v1.0/tasks

Then run it:

uv run ./scripts/run.sh --help                  # list available jobs
uv run ./scripts/run.sh --job <slug> --dry-run  # preview what will run, then exit
uv run ./scripts/run.sh --job <slug>            # run it

Results land under results/. The full set of job and model fields is documented
in configs/README.md.

Citation & license

Licensed under the Tencent license. See LICENSE.

If you find WorkBuddy Bench helpful, please cite it:

@misc{workbuddybench2026,
    title  = {WorkBuddy Bench: Evaluating Coding Agents on Real Role-Played Work},
    author = {Tencent Youtu Lab and Keen Security Lab and WorkBuddy and Yunding Security Lab},
    year   = {2026},
    note   = {https://github.com/Tencent/workbuddy-bench}
}