Microduck RL

RL training environments for Microduck —
a ~800 g, ~25 cm tall bipedal robot — built on
mjlab (MuJoCo Warp) with PPO.
Policies are trained here at 50 Hz, exported to ONNX, and deployed on the real
robot by the runtime in pollen-robotics/microduck.

https://github.com/user-attachments/assets/50c3d537-8db2-4005-9d9c-3472faeec4d0

The repo encodes the full sim2real recipe: BAM
actuator physics, domain randomization, backlash simulation, and the
reward-design lessons that made it work
(see AGENTS.md for the distilled playbook).

Quickstart

Requires a CUDA GPU (training runs through MuJoCo Warp) and uv.

On ARM boxes (DGX Spark / GB10, Jetson): uv sync pulls ~2 GB of CUDA
wheels on first run and uv's default 30 s HTTP timeout can abort mid-download.
Export UV_HTTP_TIMEOUT=600 for the first sync.

git clone https://github.com/pollen-robotics/microduck_rl
cd microduck_rl

# train the walking policy (uses your GPU; ~1-2 h for a usable gait at 4096 envs)
uv run train Mjlab-Velocity-Flat-MicroDuck --env.scene.num-envs 4096

# watch a trained policy in the viewer
uv run play Mjlab-Velocity-Flat-MicroDuck --wandb-run-path <entity/project/run_id>

# export to ONNX for deployment
uv run scripts/export.py Mjlab-Velocity-Flat-MicroDuck --wandb-run-path <...>
uv run publish --onnx output.onnx --repo <user>/microduck-<name> --kind episodic --duration-s 4.0   # share it (see "Publishing a policy")

# drive the exported policy in CPU MuJoCo with the keyboard
uv run scripts/infer_policy.py --walking output.onnx

Resume from a checkpoint:

uv run train Mjlab-Velocity-Flat-MicroDuck --env.scene.num-envs 4096 \
    --agent.run-name resume --agent.load-checkpoint model_29999.pt --agent.resume True

No GPU? Add --hf-jobs to any train command to run it on Hugging Face Jobs
instead of locally (see scripts/hf/README.md).

Tasks

uv run list-envs prints the live registry. Flat/Rough variants exist where noted.

At deployment the runtime hot-swaps these policies (walk / recover / trick)
behind a shared 61-dimensional observation contract, so any of them can take
over the robot at any moment. scripts/infer_policy.py rehearses exactly that:

uv run scripts/infer_policy.py --walking walk.onnx --standing stand.onnx \
    --sitstand sitstand.onnx --roulade roulade.onnx --new-cmd-obs

Keyboard-driven (velocity commands, G ground pick, Y sit/stand, R roulade,
K/L kicks); --debug, --save-csv, --record support sim2real comparisons.
The servos are simulated with the same BAM M6 XL330 model the policies are
trained against (voltage control + load-dependent friction, via
bam.mujoco.MujocoController); --vin / --vin-drop-gain / --kp-fw pin the
training DR ranges to one value, --no-bam falls back to the XML PD actuators.

Backlash variants

Every main task has a Backlash twin that trains on a model with ±1° of gear
play (2° total) in series with each of the 14 servo joints: insert -Backlash
before MicroDuck in the task id, e.g. Mjlab-Velocity-Flat-Backlash-MicroDuck.

The backlash is modeled properly for sim2real: each servo gets an unactuated
passive_<joint>_backlash hinge, and because the real encoder sits on the
output side of the play, both the firmware PD emulation
(BacklashEncoderBamActuator) and the joint_pos/joint_vel observations
read through the backlash (qpos[servo] + qpos[backlash]). Observation and
action dims are unchanged, so ONNX export and the runtime need no changes.
See src/mjlab_microduck/tasks/backlash.py.

Actuator model

All tasks use the BAM M6 actuator model for
the Dynamixel XL330 (voltage control law, back-EMF, Coulomb/Stribeck/load-dependent
friction), with per-env domain randomization on battery voltage, voltage sag
under load, command delay, and friction magnitude
(FrictionDRBamActuator in src/mjlab_microduck/actuator/).

At this scale — tiny servos driving a ~800 g biped — actuator fidelity is most
of the sim2real gap, which is why the actuator is modeled down to its voltage
control law instead of an ideal PD.

Robot models

MJCF models live in src/mjlab_microduck/robot/microduck/ and are exported
from Onshape with onshape-to-robot,
one config_mjcf_*.json per model:

scene*.xml files wrap the robots with a floor + keyframes (STAND/SIT/FOLD)
for quick viewing and for infer_policy.py.

Project structure

src/mjlab_microduck/
├── robot/
│   ├── microduck/                    # MJCF exports, export configs, scenes, add_backlash.py
│   └── microduck_constants.py        # robot cfgs, HOME frame, BAM actuator cfg
├── actuator/friction_dr_bam.py       # BAM + friction DR + backlash encoder feedback
├── tasks/
│   ├── __init__.py                   # task registration (base + backlash variants)
│   ├── mdp.py                        # rewards, events, observations, custom classes
│   ├── backlash.py                   # make_backlash_variant() env-cfg wrapper
│   └── microduck_*_env_cfg.py        # one cfg module per task family
├── train_cli.py                      # `train` script (identical to mjlab's)
├── train_hook.py                     # intercepts `train ... --hf-jobs`
└── hf_jobs.py                        # Hugging Face Jobs submission

Conventions worth knowing:

• The observation layout is shared across every policy (61-dim actor obs:
  48 proprioception + commands [twist(3), head_pose(4), body_pose(6)]), which
  is what makes runtime policy hot-swapping possible. Envs that don't use a
  command slot zero-pad it rather than dropping it.
• Unactuated joints are all named passive_* (roller wheels, backlash
  hinges); actuators, joint observations and pose rewards select servo joints
  with ^(?!passive_).*.
• Domain-randomization toggles are ENABLE_* booleans at the top of each
  env cfg file.
• Joint layout (14 servos): 0–4 left leg (hip_yaw, hip_roll, hip_pitch, knee,
  ankle), 5–8 neck/head (neck_pitch, head_pitch, head_yaw, head_roll),
  9–13 right leg.
• The exporter bakes the observation normalizer into the ONNX graph — always
  deploy ONNX produced by scripts/export.py, never a hand-converted
  checkpoint, or the policy sees unnormalized observations at runtime.

AGENTS.md documents the env-building workflow and the reward-design
rules learned across the project (also aimed at AI coding agents working in
this repo).

Publishing a policy

uv run publish puts a policy on the Hugging Face Hub in the shape the robot's
daemon loads: one policy.onnx with the observation normalizer baked in, a
manifest.json following schema 2 of the
microduck policy manifest,
and a README saying how to run it. Anyone with a microduck can then install it
with one command, no daemon release needed.

# From a wandb run — exports through the one safe path, then uploads
uv run publish --task Mjlab-PoliteBow-Flat-MicroDuck \
    --wandb-run-path <entity/project/run_id> --checkpoint 3000 \
    --repo <user>/microduck-polite-bow --kind episodic --duration-s 4.0 \
    --description "Bows from a two-foot stand and comes back up."

# From an ONNX you already exported (validated, not re-exported)
uv run publish --onnx output.onnx --repo <user>/microduck-flamingo \
    --kind perpetual --unwind-s 1.5 --twist-help "[flag, side, 0]"

# A new gait for a slot
uv run publish --onnx output.onnx --repo <user>/microduck-my-walk --kind perpetual --slot walk

# See what would be uploaded without touching the Hub
uv run publish --onnx output.onnx --repo <user>/microduck-bow --kind episodic --duration-s 4.0 --dry-run

Then on a robot:

sudo robotctl policy add polite-bow <user>/microduck-polite-bow   # episodic: length comes from the manifest
sudo robotctl policy add flamingo <user>/microduck-flamingo --hold 5   # held pose: you pick how long
sudo robotctl policy load walk <user>/microduck-my-walk                # gait: into the walk slot
robotctl robot do polite-bow

What --kind means, and what each needs:

• episodic — runs for --duration-s and returns itself to a standing pose
  (kicks, roulade, a bow). Add --chain if holding the button should repeat it.
• perpetual — runs until told otherwise. Two shapes:
  • a gait (a new walk or stand): add --slot walk (or stand) and
    nothing else; the owner installs it with robotctl policy load walk <repo>.
  • a held pose (the flamingo): give --unwind-s, how long the daemon
    drives the idle twist (--idle, zeros by default) before handing back to
    the gait, so the robot is not let go of on one foot. The owner runs it as a
    one-shot with policy add ... --hold <seconds>.

Before anything is uploaded, publish checks the graph is [1,61] -> [1,14]
(a 51-D legacy policy is refused with a message), runs it on plausible inputs
and refuses NaNs or a constant output, fills the training block from git and
wandb (task, commit, branch, dirty flag, run, checkpoint), and refuses to
overwrite an existing .onnx in the repo without --force. Repos are created
private; --no-private for public, --tag v1 to tag the revision.

Only constant-command policies are publishable this way. Phase-driven moves
(the ground pick) and the posture-flag sit↔stand are driven by the daemon
itself and live in the official set, pollen-robotics/microduck-policies.

Tests

uv run --with pytest pytest tests/

CPU-only config-invariant and reward-function regression tests — they lock in
joint-index mappings, reward sign conventions, and NaN guards.

Related projects

• microduck — the Microduck project home, including the onboard runtime that runs the exported policies
• mjlab — the training framework (MuJoCo Warp + rsl_rl)
• BAM — better actuator models, by Rhoban

License

This project is licensed under the Apache 2.0 License. See the LICENSE file for details.
3D model files are licensed under Creative Commons BY-SA-NC.