OPDVR

📖 Overview

Reinforcement Learning with Verifiable Rewards (RLVR) provides sparse task-level feedback, while
on-policy distillation (OPD) provides dense token-level guidance but ignores trajectory correctness.
OPDVR combines the two with an extremely simple mechanism and zero extra hyperparameters.
Vanilla OPD rewards each token by the teacher–student log-ratio,

$$
R_{\text{OPD}}(o_t) = \log \frac{\pi_T(ot \mid q, o{\lt t})}{\pi_\theta(ot \mid q, o{\lt t})},
$$

equivalently, decomposed by trajectory correctness,

$$
R_{\text{OPD}}(o_t)=
\begin{cases}
\log \dfrac{\pi_T(ot \mid q, o{\lt t})}{\pi_\theta(ot \mid q, o{\lt t})} \cdot (+1), & \text{trajectory correct},\
\log \dfrac{\pi_\theta(ot \mid q, o{\lt t})}{\pi_T(ot \mid q, o{\lt t})} \cdot (-1), & \text{trajectory incorrect},
\end{cases}
$$

a dense signal that is completely blind to trajectory correctness — the log-ratio is applied
verbatim regardless of whether the trajectory is valid. OPDVR instead passes it through a
ReLU gate keyed by the verifier:

$$
R_{\text{OPDVR}}(o_t) =
\begin{cases}
\max(0, \log \dfrac{\pi_T(ot \mid q, o{\lt t})}{\pi_\theta(ot \mid q, o{\lt t})}) \cdot (+1), & \text{trajectory correct},\
\max(0, \log \dfrac{\pi_\theta(ot \mid q, o{\lt t})}{\pi_T(ot \mid q, o{\lt t})}) \cdot (-1), & \text{trajectory incorrect}.
\end{cases}
$$

• correct trajectories → non-negative rewards
• incorrect trajectories → non-positive rewards

The verifier (task reward) determines the direction of every token update; the teacher controls
its magnitude. Tokens whose learning direction conflicts with the verifier are simply masked out.

GRPD (Group Relative Policy Distillation): because OPDVR is now a proper RLVR reward,
the binary signal can be replaced by any policy-gradient advantage — we instantiate it
with GRPO-style group-relative advantages.

Experiments on six reasoning benchmarks (AIME24/25, AMC, MATH500, Minerva, OlympiadBench) show
OPDVR consistently outperforms standard OPD.

📊 Main Results (avg@16)

Same-architecture distillation (Qwen3-4B ← Qwen3-4B-RL):

Cross-architecture distillation (Qwen3-1.7B-Base ← Qwen3-4B-Base-RL):

✨ Repo Structure

OPDVR/
├── opdvr.sh                    # OPDVR training (the paper's main method)
├── grpd.sh                     # GRPD variant (group-relative advantage, G=8)
├── opd_baseline.sh             # vanilla sampled-token OPD baseline
├── ablation_inverse_gated.sh   # inverse-gate ablation (Table: ablation study)
├── verl/                       # fork of verl v0.7.0 containing the OPDVR implementation
└── datasets/                   # training + eval data (parquet)

🔧 Where the Method Lives

Key config switches (all env-overridable in the run scripts):

🚀 Getting Started

Environment

conda create -n verl python==3.12
conda activate verl
cd verl/
USE_MEGATRON=0 bash scripts/install_vllm_sglang_mcore.sh
pip install --no-deps e .

Train

Everything is launched from the scripts in the repo root:

bash opdvr.sh                    # OPDVR
bash grpd.sh                     # GRPD (group-relative advantage variant)
bash opd_baseline.sh             # OPD baseline
bash ablation_inverse_gated.sh   # inverse-gate ablation

All knobs (models, data, method switches) are environment variables set at the top of each
script — edit them or override on the command line, e.g. ACTOR_MODEL_PATH=... REWARD_MODEL_PATH=... bash opdvr.sh.
Training data and the val set are under datasets/.

Evaluate

Evaluation runs inside the training loop (verl's built-in validation):
val_kwargs.n=16 gives the paper's avg@16 protocol on datasets/valid_final_unique.parquet,
reported as val-core/... in wandb / console every TEST_FREQ (default 22) steps.
No separate eval repo is needed — checkpoints saved under checkpoints/<experiment> can also
be evaluated offline with any vLLM harness against the same parquet.

valid_final_unique.parquet (1,590 problems) combines the paper's six benchmarks:
AIME24 (30), AIME25 (30), AMC (83), MATH-500 (500), Minerva (272),
OlympiadBench (675).

Acknowledgements

• OPD (Rethinking On-policy Distillation of Large Language
  Models) — this repo is developed on top of it; the OPD training framework comes from there.
• verl (v0.7.0) — the underlying RL training framework.