Self-Distilled RLVR (RLSD)

Reproduction: RLSD vs GRPO on MATH via the Tinker API

Claim tested. The paper's headline empirical claim — RLSD beats GRPO on reasoning accuracy (paper: +2.32 pts average, 56.18 vs 53.86 over five multimodal benchmarks) — plus its mechanism claims (RLSD maintains higher entropy; credit-clip ratio 3–6%; OPSD, the distribution-matching predecessor, is unstable and leaks privileged information).

What was done. GRPO / RLSD / OPSD were re-implemented on the Tinker API (hosted LoRA training + sampling of Qwen3-8B) and run on MATH Level-4–5 problems, up to 150 steps × 64 prompts × 8 rollouts, evaluated on MATH500 (n=500). Runs were orchestrated with orx on a local machine; all model compute is remote. Downscaling vs the paper: text-only instead of multimodal, LoRA instead of full fine-tuning, ~10% of the paper's prompt budget, lr 1e-5 (LoRA-appropriate; the paper's 1e-6 is for full FT and produced no measurable learning here), MATH reference solutions as privileged information. OPSD is implemented as the policy-gradient estimator of the per-token reverse-KL distillation (Tinker exposes sampled-token logprobs, not full-vocab logits).

Result. Headline claim not observed at this scale, across 19 runs in 6 configurations: GRPO +10.8 vs RLSD +9.0 at 150 steps (71.0 vs 69.0 on MATH500), and RLSD never led at any dose — including a high-headroom Qwen3.5-4B test (+23.5 vs +22.5) and a 2-seed replay of RLSD's strongest window (+1.35 vs +1.3). Mechanism claims reproduced: RLSD entropy stayed ~2× GRPO's late in training (0.24 vs 0.13 nats/token), credit-clip ratio sat in the paper's 3–6% band, zero privileged-info leakage in RLSD runs, and OPSD collapsed (61.6% → 7.6%) at an lr where GRPO/RLSD trained stably — directionally supporting the paper's core diagnosis. Keeping λ's reweighting active all run (instead of the paper's 25% decay) hurt accuracy, validating that design choice.

Read more: full visual report · interactive notebook (marimo) — run locally with marimo edit notebooks/rlsd_repro.py (repo is private, so no Molab link)

Experiment log

Branch	Purpose / change	Run command	Outcome	Compute
orx/baseline-grpo-on-math-tinker-qwen3-8b	Baseline GRPO, 40×32, L4-5, lr 1e-6	bash repro/run.sh	60.8 → 62.4 (+1.6); flat — under-trained	Tinker, 20 min
orx/rlsd-on-math-tinker-qwen3-8b	RLSD round 1	bash repro/run.sh	61.2 → 62.4 (+1.2); clip ~5% ✓	Tinker, 25 min
orx/grpo-l5-80-step-round-2 / orx/rlsd-l5-80-step-round-2	Round 2: L5-only, 80×64	bash repro/run.sh	Both flat (±0.8) — lr 1e-6 confirmed too low for LoRA	Tinker, 41+52 min
orx/grpo-lr1e-5-l5-round-3 / orx/rlsd-lr1e-5-l5-round-3	Round 3: lr 1e-5, 60×64	bash repro/run.sh	GRPO +5.6, RLSD +5.2 — first real learning; tie within noise	Tinker, 35+39 min
orx/grpo-lr1e-5-seed-2-round-4 / orx/rlsd-lambda-full-60-step-round-4	Seed variance; λ-full ablation	bash repro/run.sh	GRPO s2 +6.6 (seed var ~1pt); λ-full +2.0 — full-run reweighting hurts	Tinker, 43+60 min
orx/grpo-lr1e-5-150-step-round-5 / orx/rlsd-lr1e-5-150-step-round-5	Final dose: 150×64	bash repro/run.sh	GRPO +10.8 (71.0), RLSD +9.0 (69.0); RLSD entropy 0.24 vs GRPO 0.13	Tinker, 66+90 min
orx/opsd-on-math-tinker-qwen3-8b / orx/opsd-solution-teacher-60-step / orx/opsd-solution-teacher-lr1e-5-round-5	OPSD failure-mode probes	bash repro/run.sh	lr 1e-6: no failure (nothing trains); lr 1e-5: collapse 61.6 → 7.6 via generative drift; phrase-leakage 0.0	Tinker, 24+39+42 min
orx/grpo-45-step-seed3-window-replay + 3 siblings	2-seed replay of RLSD's λ-active window	bash repro/run.sh	In-window lead was seed luck: means +1.35 (GRPO) vs +1.3 (RLSD)	Tinker, 4×20–25 min
orx/grpo-qwen3-1-7b-round-6 / orx/rlsd-qwen3-1-7b-solution-teacher-round-6	High-headroom test: Qwen3.5-4B, RLSD w/ solution-trace teacher (parallel session)	bash repro/run.sh	GRPO +23.5, RLSD +22.5 — big learning, still no separation; RLSD late slope steeper	Tinker, 26+32 min
main	Not run as an experiment (publication surface)	—	—	—

Metrics also on wandb project rlsd-2604.03128.

Chinese

Method

RLSD (Self-Distilled RLVR) reframes self-distillation from a distribution-matching objective into a token-level credit assignment signal for RLVR. The teacher model can use privileged information, but that information does not directly decide which tokens should be reinforced or penalized, nor does it change the direction of parameter updates. It only adjusts how much credit each token receives within the same trajectory.

Given a student-sampled trajectory $y=(y_1,\dots,y_T)$, RLSD computes the log-probability of each token under both the student context $x$ and the teacher context $(x,r)$, then defines the privileged information gain:

$$
\Delta_t=\texttt{sg}\left(\log P_T(y_t)-\log P_S(y_t)\right)
$$

where $\texttt{sg}$ denotes stop-gradient. $\Delta_t$ measures the marginal contribution of the privileged information $r$ to the current token: a positive value means the teacher-side information supports the token more, while a negative value means it does not support the token. Because $\Delta_t$ is stop-gradient, it is used only as a weighting signal and does not introduce an additional optimization path.

RLSD then constructs a direction-aware evidence weight according to the sign of the sequence-level advantage:

$$
w_t=\exp(\mathrm{sign}(A)\cdot\Delta_t)=\left(\frac{P_T(y_t)}{P_S(y_t)}\right)^{\mathrm{sign}(A)}
$$

When $A>0$, tokens more supported by the teacher receive larger positive credit. When $A<0$, the weight is inverted so that tokens less supported by the teacher receive larger blame. Since $w_t$ is always positive, RLSD never flips the sign of the advantage: the environment reward still decides whether the whole trajectory is reinforced or penalized, while the teacher only redistributes credit inside the trajectory.

Finally, RLSD clips the evidence weight to limit the influence of any single token during training:

$$
\hat{A}_t=A\cdot\mathrm{clip}(w_t,1-\epsilon_w,1+\epsilon_w)
$$

This design is analogous to the clipped surrogate objective in PPO/GRPO. GRPO clips the policy ratio to constrain the update step size, while RLSD clips the evidence ratio to constrain the magnitude of credit redistribution. This allows RLSD to use teacher-side information while preserving stable training.

Performance

Installation

This project depends on the environment from EasyVideoR1: Easier RL for Video Understanding. Please first create the base environment following EasyVideoR1, then install this project in the same environment.

Step 1: Create a Conda Environment

conda create -n rlsd python=3.11
conda activate rlsd

Step 2: Install the EasyVideoR1 Base Environment

git clone https://github.com/cyuQ1n/EasyVideoR1.git
cd EasyVideoR1
pip install -e .
pip install flash-attn==2.8.3 --no-build-isolation

Step 3: Install This Project

git clone <this-repo-url>
cd rlsd_verl
pip install -e .

If you already have this repository locally, install it directly from the current project directory:

cd /pfs/qcy/video_rl/rlsd_verl
pip install -e .

Dependencies

This project follows EasyVideoR1's veRL, Ray, vLLM, and Qwen-VL dependencies, and maintains additional dependencies in this repository's requirements.txt. Key versions include:

qwen-vl-utils[decord]==0.0.14
transformers==4.57.3
vllm==0.11.0
flash-attn==2.8.3

Quick Start

The minimal training workflow is as follows.

Step 1: Prepare Data

The training and validation data can be downloaded from the Hugging Face dataset iieycx/rlsd-train-MMFineReason-123K.

The training data should be in JSON/JSONL format. Each example should contain at least the question, answer, and image fields.

Step 2: Use the Provided Configs and Scripts

This repository provides ready-to-use RLSD and GRPO baseline configuration files and training scripts under examples/visual_rl/:

The training scripts already bind the corresponding YAML config, prompt template, and reward function. Default data paths, model paths, and output paths can be checked in the scripts, and can also be overridden with environment variables.

Step 3: Start Training

Run RLSD:

bash examples/visual_rl/rlsd_train.sh

Run the GRPO baseline:

bash examples/visual_rl/grpo_baseline_train_MMFineReason.sh

For multi-node training, set WORLD_SIZE, RANK, and MASTER_ADDR on each node, then run the corresponding script:

# RLSD head node
WORLD_SIZE=2 RANK=0 MASTER_ADDR=<head_node_ip> bash examples/visual_rl/rlsd_train.sh

# RLSD worker node
WORLD_SIZE=2 RANK=1 MASTER_ADDR=<head_node_ip> bash examples/visual_rl/rlsd_train.sh

Project Structure

rlsd_verl/
|-- verl/                       # Core RL training framework
|   |-- trainer/                # GRPO / OPSD / RLSD training entry points and loops
|   |-- workers/                # Actor, rollout, reward, and critic workers
|   |-- models/                 # Qwen-VL model adaptation
|   `-- utils/                  # Dataset, tokenizer, FSDP, logging, and checkpoint utilities
|-- examples/
|   `-- visual_rl/              # Visual RL configs, training scripts, prompts, and rewards
|-- requirements.txt            # Python dependencies
|-- setup.py                    # Editable installation entry point
`-- README_zh.md

Example Pipelines

RLSD (examples/visual_rl/)

The RLSD training script uses verl.trainer.opsd_main by default, and handles rewards for different visual reasoning tasks through examples/visual_rl/reward_function/unified.py.

bash examples/visual_rl/rlsd_train.sh

GRPO Baseline (examples/visual_rl/)

The standard GRPO baseline uses the same data, prompt, and reward interfaces, making it convenient to compare against RLSD.

bash examples/visual_rl/grpo_baseline_train_MMFineReason.sh

Acknowledgements

This project is built on the following work:

• EasyVideoR1 - Easier RL for Video Understanding
• EasyR1 - An efficient and scalable multimodal RL training framework
• veRL - High-performance RL and HybridEngine

Citation

If you use this project, please cite Self-Distilled RLVR, EasyVideoR1, EasyR1, and veRL:

@misc{yang2026selfdistilledrlvr,
      title={Self-Distilled RLVR},
      author={Chenxu Yang and Chuanyu Qin and Qingyi Si and Minghui Chen and Naibin Gu and Dingyu Yao and Zheng Lin and Weiping Wang and Jiaqi Wang and Nan Duan},
      year={2026},
      eprint={2604.03128},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2604.03128},
}

@misc{qin2026easyvideor1easierrlvideo,
      title={EasyVideoR1: Easier RL for Video Understanding},
      author={Chuanyu Qin and Chenxu Yang and Qingyi Si and Naibin Gu and Dingyu Yao and Zheng Lin and Peng Fu and Nan Duan and Jiaqi Wang},
      year={2026},
      eprint={2604.16893},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2604.16893},
}

License

This project follows the same license as EasyR1.

Self-Taught RLVR

This series is built around one core question:

How can large models guide themselves and evolve iteratively?

The Self-Taught line explores three complementary dimensions:

• The first work, RLSD, studies the informed self: the model teaches itself with privileged information.
• The second work, NPO, focuses on the temporal self: the model is taught by its near-future self.
• The third work, CoPD, explores the parallel self: the model is taught by another version of itself that takes a different path.

These three works address key questions in RLVR and OPD:

• RLSD: How can a model better absorb useful privileged information?
• NPO: How can RLVR introduce more suitable auxiliary learning signals?
• CoPD: How can a single model better absorb the capabilities of multiple experts?

All of these questions share the same underlying theme: how to introduce better learning signals and make them effectively absorbable by the model.

The Self-Taught RLVR series gives the same answer:

Let the model provide itself with learning signals that match its current capability and are easier to absorb.

Welcome to upvote these works on Hugging Face:

• Hugging Face: https://huggingface.co/papers/2604.03128
• Hugging Face: https://huggingface.co/papers/2604.20733
• Hugging Face: https://huggingface.co/papers/2604.27083

☎️ We're Hiring!

We're hiring multimodal research scientists and TGT interns at JD Explore Academy! If you have top-tier publications and are passionate about video understanding and VLMs, please send your resume to: [email protected]. We'd love to hear from you!

WeChat Group