aacr-bench

📋 Introduction

AACR-Bench is the industry's first multilingual, repository-level context-aware code review evaluation dataset, designed to assess the performance of large language models in automated code review tasks. The dataset comprises 200 real Pull Requests from 50 active open-source projects, covering 10 mainstream programming languages. Each instance includes not only code changes but also preserves complete repository context, authentically reproducing the entire code review process. Through human-LLM collaborative review combined with multi-round expert annotation, we ensure high quality and comprehensiveness of the data.

✨ Core Features

🌍 Multi-language Coverage

Covers 10 mainstream programming languages used in projects:

• System-level languages: C++, Rust, Go
• Enterprise languages: Java, C#, TypeScript
• Scripting languages: Python, JavaScript, Ruby, PHP

📁 Repository-level Context

• Preserves complete project structure
• Supports cross-file references and inter-module interaction analysis
• Includes PR metadata (description, title, comments, etc.)

🤖 Human Expert + LLM Enhanced Annotation

Professional Annotation Team

• 80+ senior software engineers with 2+ years of experience
• Covers frontend, backend, architecture, and other domains
• Three rounds of cross-validation

LLM Intelligent Enhancement

• Systematic issue discovery
• Comprehensive defect identification
• Improvement suggestion generation

Quality Assurance Process: GitHub human comments → LLM enhancement → Expert multi-round cross-annotation → Consistency validation

🎯 Evaluation Capabilities and Applications

AACR-Bench provides systematic evaluation capabilities across four core dimensions, supporting diverse research and application scenarios:

Evaluation Dimension System

Typical Application Scenarios

Model Development

• Performance benchmarking: Evaluate new models' code review capabilities using unified standards
• Weakness analysis: Discover model shortcomings through fine-grained metrics (e.g., low recall in certain languages)
• Iterative optimization validation: Quantify improvement effects, guide continuous model optimization

Academic Research

• Comparative studies: Fair comparison of different model architectures/training methods
• Ablation experiments: Analyze the impact of different context levels on review quality
• New method validation: Provide standardized evaluation environment for innovative algorithms

Engineering Practice

• Model selection: Help teams choose review models suitable for project characteristics
• Pre-deployment validation: Ensure model reliability in production environments
• Continuous monitoring: Track model performance changes in actual use

🚀 Quick Start

Clone Repository and Download Dataset

git clone https://github.com/alibaba/aacr-bench.git
cd aacr-bench

Install Dependencies

pip install -r requirements.txt

Configure Claude CLI

Edit configs/config.json and set the Claude CLI installation path:

{
  "cli_path": "your_path_to_claude.cmd",
  "data_path": "your_path_to_positive_samples.json"
}

Configure Evaluation Environment Variables

Create a .env file in the evaluator_runner/utils/ directory:

LLM_MODEL_URL="your_llm_model_url"
LLM_MODEL="your_llm_model"
LLM_API_KEY="your_llm_api_key"

EMBEDDING_MODEL_URL="your_embedding_model_url"
EMBEDDING_MODEL="your_embedding_model"
EMBEDDING_API_KEY="your_embedding_api_key"

Prepare Dataset

For the first run, you need to convert the raw data to task format. Uncomment and run in main.py:

if __name__ == "__main__":
    load_data_as_task()  # First run: generate task file

This will:

• Read the raw dataset (specified by data_path)
• Add finish flag to each PR for progress tracking
• Generate tmp_data.json task file

Run Code Review

cd claude-code-demo
python main.py

Run Evaluation

import asyncio
from evaluator_runner import (
    get_evaluator_ans_from_json,
    load_generated_comments_from_file,
    EvaluatorConfig
)

async def main():
    # Load AI-generated comments to evaluate
    generated_comments = load_generated_comments_from_file("path/to/comments.txt")

    # Reference comments (load from positive_samples.json)
    reference_comments = [...]

    # Run evaluation
    result = await get_evaluator_ans_from_json(
        github_pr_url="https://github.com/owner/repo/pull/123",
        generated_comments=generated_comments,
        good_comments=reference_comments
    )

    print(f"Location Match Rate: {result['positive_line_match_rate']}")
    print(f"Semantic Match Rate: {result['positive_match_rate']}")

asyncio.run(main())

Batch Evaluation

# Configure parameters in evaluator_runner/example_test.py, then run:
python evaluator_runner/example_test.py

📈 Data Overview

Dataset Scale

Classification Statistics

Language Distribution

Data Format

{
  "type": "array",
  "item": {
    "change_line_count": {"type": "integer", "description": "Number of changed lines"},
    "project_main_language": {"type": "string", "description": "Project main language"},
    "source_commit": {"type": "string", "description": "Source commit"},
    "target_commit": {"type": "string", "description": "Target commit"},
    "githubPrUrl": {"type": "string", "description": "GitHub PR URL"},
    "comments": {
      "is_ai_comment": {"type": "boolean", "description": "Whether it's an AI comment"},
      "note": {"type": "string", "description": "Comment content"},
      "path": {"type": "string", "description": "File path"},
      "side": {"type": "string", "description": "Comment anchor position"},
      "source_model": {"type": "string", "description": "Source model"},
      "from_line": {"type": "integer", "description": "Start line number"},
      "to_line": {"type": "integer", "description": "End line number"},
      "category": {"type": "string", "description": "Issue category: Security/Defect/Maintainability/Performance"},
      "context": {"type": "string", "description": "Comment scope: diff/file/repo level"}
    }
  }
}

📏 Evaluation Metrics

We employ a multidimensional metric system to comprehensively evaluate code review model performance. For complete metric definitions, calculation methods, and language-specific statistics, please refer to metrics.md.

Core Metrics

🤝 Contributing

We welcome community contributions! If you want to contribute to AACR-Bench, please follow these steps:

1. Fork this repository
2. Create feature branch (git checkout -b feat/add-new-prs)
3. Commit changes (git commit -m 'feat: add new PRs')
4. Push to branch (git push origin feat/add-new-prs)
5. Create Pull Request

For detailed contribution guidelines, please refer to CONTRIBUTING.md

👥 Authors and Maintainers

📄 License

This project is licensed under the Apache License 2.0. For details, please see the LICENSE file.

📚 Citation

If you use AACR-Bench in your research, please cite our paper:

@misc{zhang2026aacrbenchevaluatingautomaticcode,
      title={AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context}, 
      author={Lei Zhang and Yongda Yu and Minghui Yu and Xinxin Guo and Zhengqi Zhuang and Guoping Rong and Dong Shao and Haifeng Shen and Hongyu Kuang and Zhengfeng Li and Boge Wang and Guoan Zhang and Bangyu Xiang and Xiaobin Xu},
      year={2026},
      eprint={2601.19494},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2601.19494}, 
}

🗺️ Roadmap

• [x] v1.0 (2026.01): Initial release - 200 PRs, 10 languages

🌟 Acknowledgments

• Thanks to all contributors who participated in data annotation, especially core contributors who completed 15+ valid annotations. Full list in CONTRIBUTORS.md.
• Thanks to open-source project maintainers for providing original PR data.