HuggingEnvs

Open source RL environments for LLM agents

Build  ·  Deploy  ·  Train  ·  Eval  — end to end, reproducible, in the open.

Why this exists

The RL environment ecosystem is moving fast. New frameworks land every few weeks, each with its own
vocabulary for the same handful of ideas, and most of what's written about them is either a launch post
or a spec. Meanwhile the actual bottleneck in RL for LLMs has quietly shifted: the algorithm isn't the
hard part any more — the environment is.

So we're building the resource we wanted. Open source, end to end, and reproducible: how to design an
environment, how to build it, how to deploy it, how to train against it, and how to scale it to
thousands of concurrent sessions. Real code you can run, not diagrams of code someone else ran.

Every environment here works. Every rollout has been executed. Every training curve came from a job you
can launch yourself, in one command, without a GPU of your own.

Where things live: source, recipes and notebooks in this repo · environments, datasets, models and
demos on 🤗 huggingface.co/HuggingEnvs.

Projects

Each numbered folder is a self-contained, end-to-end project — its own environments, notebooks,
results and README, plus the Hub repos it owns. They read in order but stand alone.

00 · RL Environments 101  _{3 environments · 6 frameworks · 8 live Spaces}

One env, six ways. Three environments, each implemented six times — same logic, six framework
dialects. Diff any two server.py files and the differences are the lesson.

Across openenv · ors · nemo_gym · verifiers · skyrl_gym · gem — 18 implementations, 8 of
them deployed as Spaces you can hit right now.

01 · LaTeX OCR  _{train a VLM against a served reward}

The full loop. Project 00 shows you what an environment is; this one takes a single environment
all the way to a trained model.

Qwen3-VL-2B learns to read rendered math into LaTeX with GRPO, scored by a reward served from a live
OpenEnv Space. Correctness is checkable —
render the prediction, compare — so the reward is honest and there is very little to game. Runs on a
GPU you spin up in one command; no cluster, no local GPU.

More coming. Each new project is another end-to-end recipe: an environment, a training run, and
the artifacts on the Hub. Proposals and contributions welcome →

Articles & talks

Long-form writing and conference talks. Sources live in content/; each one ships to the
Hub as a Space.

<table>
<tr>
<td width="33%" valign="top">

<a href="https://huggingface.co/spaces/AdithyaSK/rl-environments-guide">

</a>

The Ultimate Guide to RL Environments

Building and scaling RL environments in the LLM era — how frameworks are built, how rewards are
wired, and how they scale to thousands of concurrent sessions.

_{📂 content/articles/rl-environments-guide/}

</td>
<td width="33%" valign="top">

<a href="https://huggingface.co/spaces/AdithyaSK/rl-environments-101-slides">

</a>

RL Environments 101

From "what is an env?" to training your own. RL fundamentals → environment anatomy → OpenEnv →
training with TRL. The original 30-minute talk.

_{📂 content/slides/rl-environments-101/}

</td>
<td width="33%" valign="top">

<a href="https://huggingface.co/spaces/AdithyaSK/scaling-rl-for-llms-amd-ai-dev-day">

</a>

Scaling RL for LLMs

What an environment actually is, how reward hacking happens, and how to build and train against your
own. The 20-minute cut, for AMD AI Dev Day.

_{📂 content/slides/scaling-rl-amd/}

</td>
</tr>
<tr>
<td width="33%" valign="top">

<a href="https://huggingface.co/spaces/AdithyaSK/multi-harness-training-slides">

</a>

Multi-Harness Training

OpenEnv × Harbor — why an environment's failure model decides whether it can be trained against:
in-process agent loops vs. an HTTP boundary, and what it takes to capture trainable tokens.

_{📂 content/slides/multi-harness-training/}

</td>
<td width="33%" valign="top">
</td>
<td width="33%" valign="top">
</td>
</tr>
</table>

Quickstart

Run an environment. Wordle is pure Python with no external backend — the fastest full rollout:

git clone https://github.com/adithya-s-k/HuggingEnvs
cd HuggingEnvs
cp .env.example .env          # HF_TOKEN, plus E2B_API_KEY for sandbox-backed envs

cd 00-environments-101/envs/wordle/verifiers
uv sync && uv run python rollout.py

Train a model against one. No GPU, no cluster, no setup — one command spins up a GPU with the
notebooks loaded and prints a JupyterLab URL:

curl -sSL https://raw.githubusercontent.com/adithya-s-k/HuggingEnvs/main/tools/jupyter_launch.py | python3 -

_{Windows (PowerShell): irm https://raw.githubusercontent.com/adithya-s-k/HuggingEnvs/main/tools/jupyter_launch.py | python -. Set FLAVOR=t4-small for a cheaper GPU. Track jobs at huggingface.co/settings/jobs.}

Build your own environment

Five SKILL.md-spec agent skills turn a plain-English description
into a runnable RL environment across four frameworks. They work in any project — with Claude Code,
Cursor, Codex, OpenCode, Gemini CLI and others.

npx skills add adithya-s-k/HuggingEnvs

"make me an env where the agent plays connect-four" — that's the whole interface.

Repository layout

HuggingEnvs/
├── 00-environments-101/     3 environments × 6 frameworks
├── 01-latex-ocr/            train a VLM against a served reward
├── content/
│   ├── articles/            long-form sources (Astro → Docker Space)
│   └── slides/              talk decks (Vite → static Space)
├── tools/                   launcher, Space deploy, index generation
├── assets/                  brand + content thumbnails
└── .claude/skills/          the five environment-authoring agent skills

Inside a project the folders always mean the same thing: envs/ (implementations, shared logic in
core/), train/ (configs + launch), notebooks/, results/.

Contributing

We're actively looking for new end-to-end recipes — a task, an environment, a training run, and
honest results. Domains we don't cover yet are especially welcome: web browsing, SQL, games,
robotics sims, tool-use over real APIs, long-horizon software engineering.

Half-finished counts. A recipe with real numbers and a gap beats a polished one nobody ran — open an
issue and we'll help you land it. New framework ports, reproductions that disagree with ours, and
corrections to the guide are all just as welcome.

See CONTRIBUTING.md. The fastest path to a new environment is the
rl-env-from-description skill.

Citation

@misc{huggingenvs,
  author = {Kolavi, Adithya S},
  title  = {HuggingEnvs: Open Source RL Environments for LLM Agents},
  year   = {2026},
  url    = {https://github.com/adithya-s-k/HuggingEnvs}
}

License

Apache 2.0