Train an SO-101 Robot From Sim-to-Real With NVIDIA Isaac

Welcome to this workshop on sim-to-real transfer for the SO-101 robot!

This repository contains the assets and code to accompany this learning content.

The rest of this README will help you setup the environment and ensure everything is installed correctly.

You can also use this repo as a basis for trying out your own tasks.

Requirements

This content was tested on the following GPUs:

NVIDIA RTX 6000 Pro (Blackwell)
NVIDIA RTX 5090 (Blackwell)
NVIDIA RTX 6000 (Ada)

OS and Software tested:

Ubuntu Linux >22.04
Docker
CUDA Toolkit
NVIDIA Container Toolkit

Installation

Create directory and clone this repo

cd ~/
git clone https://github.com/isaac-sim/Sim-to-Real-SO-101-Workshop.git

Building the Docker images

Navigate to the repo
```
cd ~/Sim-to-Real-SO-101-Workshop
```

Teleop & Simulation container

From the repo root directory, run:

docker build -t teleop-docker -f docker/sim/Dockerfile .

Real Robot & Inference Server - this may take a while to build

For Blackwell architecture GPUs:

From the repo root directory, run:
```
./docker/real/build.sh blackwell
```
For Ada architecture GPUs:
From the repo root directory:
```
./docker/real/build.sh ada
```
Continue with the course instructions here.

Starting the images

To start the Teleop & Simulation container:

xhost + 
docker run --name teleop -it --privileged --gpus all -e "ACCEPT_EULA=Y" --rm --network=host \
   -e "PRIVACY_CONSENT=Y" \
   -e DISPLAY \
   -v /dev:/dev \
   -v /run/udev:/run/udev:ro \
   -v $HOME/.Xauthority:/root/.Xauthority \
   -v ~/docker/isaac-sim/cache/kit:/isaac-sim/kit/cache:rw \
   -v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw \
   -v ~/docker/isaac-sim/cache/pip:/root/.cache/pip:rw \
   -v ~/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \
   -v ~/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \
   -v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \
   -v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw \
   -v ~/docker/isaac-sim/documents:/root/Documents:rw \
   -v ~/.cache/huggingface/lerobot/calibration:/root/.cache/huggingface/lerobot/calibration \
   -v ./docker/env:/root/env \
   -v $(pwd)/source:/workspace/Sim-to-Real-SO-101-Workshop/source \
   -v $(pwd)/outputs:/workspace/Sim-to-Real-SO-101-Workshop/outputs \
   -v $(pwd)/datasets:/workspace/Sim-to-Real-SO-101-Workshop/datasets \
   teleop-docker:latest

To start the Real Robot & Inference Server:

docker run -it --rm --name real-robot --network host --privileged --gpus all \
    -e DISPLAY \
    -v /dev:/dev \
    -v /run/udev:/run/udev:ro \
    -v $HOME/.Xauthority:/root/.Xauthority \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -v ~/.cache/huggingface/lerobot/calibration:/root/.cache/huggingface/lerobot/calibration \
    -v ./docker/env:/root/env \
    -v ~/models:/workspace/models \
    -v $(pwd)/docker/real/scripts:/workspace/Isaac-GR00T/gr00t/eval/real_robot/SO100 \
    real-robot \
    /bin/bash

Models and Datasets

Downloading model weights

First, install the HuggingFace command-line-interface (CLI)

The models used in the course are listed in the course instructions here.

You can either download them ahead of time, or as you get to them in the course.

Tasks

Debug envs

Lerobot-So101-Teleop-Base : Teleop debug
Lerobot-So101-Teleop-Task : Lightbox, cameras, non-task related debug

Tasks

Lerobot-So101-Teleop-Vials-To-Rack - Main task for the workshop - pick up the vial and place it in the yellow rack
Lerobot-So101-Teleop-Vials-To-Rack-DR - Same as above but with domain randomization

Eval

Lerobot-So101-Teleop-Vials-To-Rack-Eval - Evaluation without domain randomization (fixed orange robot, no lighting/mat DR)
Lerobot-So101-Teleop-Vials-To-Rack-DR-Eval - Evaluation with full domain randomization

Commands

list_envs - List environments in this repo
zero_agent - Debug script with zero actions
random_agent - Debug script with random actions
lerobot_agent - LeRobot SO101 teleop script
lerobot_eval - Model evaluation script
lerobot_push_dataset - LeRobot Dataset push to hub script

Contributions

We are not currently accepting contributions for this project.