<br>

Code-A1 jointly trains a Code LLM and a Test LLM with opposing objectives, enables white-box adversarial test generation without self-collusion, and uses a Mistake Book replay mechanism to retrieve historical failure cases.

Table of Contents

• Motivation
• Highlights
• Repository Structure
• Installation
• Dataset
• Quick Start
• Main Results
• Citation
• Acknowledgement

Motivation

Reinforcement learning for code generation typically depends on unit-test pass rates as verifiable rewards. In practice, this creates three persistent issues:

• Static golden tests are limited in coverage and quickly saturate as the code model improves.
• Black-box generated tests are often too generic to expose implementation-specific bugs.
• Single-model self-play introduces self-collusion: the same model can generate easy tests that inflate rewards without improving code quality.

Code-A1 addresses this by separating the two roles. A Code LLM is optimized to solve programming problems, while a Test LLM is optimized to expose errors in the generated code. This makes white-box test generation useful rather than dangerous: the Test LLM can inspect candidate implementations and generate targeted adversarial tests. The framework further stabilizes co-evolution with a Mistake Book replay mechanism and a composite reward that balances test validity and adversarial difficulty.

✨ Highlights

• Adversarial co-evolution: jointly optimizes a Code LLM and a Test LLM with opposite objectives instead of collapsing both roles into one model.
• White-box test generation without self-collusion: the Test LLM conditions on candidate code to synthesize implementation-specific attack tests.
• Mistake Book replay: historical failure cases are replayed during training so the Code LLM does not forget earlier weaknesses.
• Composite reward for Test LLM: balances executable test validity with adversarial difficulty.
• Strong empirical gains: improves both code generation and test generation, and generated tests can even serve as competitive static supervision.

🛠 Installation

The project uses Python 3.10 and uv for environment management. The provided scripts create two separate environments: one for RL training and one for evaluation.

1. RL environment

cd Code-A1
bash rl_env.sh
source .venv-rl/bin/activate

This script creates .venv-rl, installs the local verl fork in editable mode, and installs the main runtime dependencies including ray, vllm/sglang support, and sandbox_fusion.

2. Evaluation environment

cd Code-A1
bash eval_env.sh
source .venv-eval/bin/activate

This script creates .venv-eval and installs the packages needed for BigCodeBench evaluation, mutation-based tooling, vllm==0.11.0, wandb, and sandbox_fusion.

3. Runtime prerequisites

Before training or evaluation, configure environment variables in Code-A1/code/rl/run/set_env.sh:

export SANDBOX_FUSION_ENDPOINT=YOUR_SANDBOX_IP
export WANDB_API_KEY=YOUR_WANDB_API_KEY

The training scripts also expect:

• A reachable sandbox_fusion service for secure code execution.
• GPUs compatible with the selected model scale and FSDP-based training.
• Access to the base models specified in the YAML configs, such as Qwen/Qwen2.5-Coder-1.5B-Instruct.

📊 Dataset

The training data is stored in Code-A1/code/rl/train_data/kodcode_hard_dual_model_training_data_mix.parquet and is built from 9,688 hard-difficulty questions from KodCode-V1.

🚀 Quick Start

Sandbox check

Validate that the sandbox executor is reachable before running training:

cd Code-A1
bash test_sandbox.sh

Expected output should contain a successful RunCodeResponse with return_code=0.

RL training

Provided launch scripts:

cd Code-A1
source .venv-rl/bin/activate

bash code/rl/run/1.5B_A1.sh
bash code/rl/run/3B_A1.sh
bash code/rl/run/7B_A1.sh

For example, the default 1.5B config uses:

• Qwen/Qwen2.5-Coder-1.5B-Instruct as the Code LLM
• Qwen/Qwen2.5-Coder-1.5B-Instruct as the Test LLM
• alpha: 0.5 in the composite test reward
• n_gpus_per_node: 4 for both Code LLM and Test LLM
• rollout.n: 8 during training and n: 32 during validation sampling

Evaluation

For the Code LLM:

• HumanEval+ and MBPP+ are included in the validation data and are evaluated during training.
• BigCodeBench is evaluated separately with:

cd Code-A1
source .venv-eval/bin/activate
bash code/rl/run/eval.sh

For the Test LLM, evaluation is conducted with UnLeakedTestBench

📈 Main Results

Code generation

Code-A1 outperforms both the Golden Tests baseline trained on human annotations and the Self-Play approach on HumanEval+, MBPP+, and BigCodeBench.

Test generation

The Test LLM also improves substantially under adversarial co-evolution. Notably, the 3B Test LLM trained with Code-A1 reaches Mul = 15.29, exceeding the unoptimized 7B base model (14.72).

📄 Citation

@misc{wang2026codea1adversarialevolvingcode,
      title={Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learning}, 
      author={Aozhe Wang and Yuchen Yan and Nan Zhou and Zhengxi Lu and Weiming Lu and Jun Xiao and Yueting Zhuang and Yongliang Shen},
      year={2026},
      eprint={2603.15611},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.15611}, 
}

🙏 Acknowledgement

The RL training stack is built on top of the excellent verl framework, which is included in this repository under Code-A1/verl. Many thanks to the verl team for open-sourcing the infrastructure that this project extends.