xllm

English | 中文

[](https://docs.xllm-ai.com/) [](https://quay.io/repository/jd_xllm/xllm-ai?tab=tags) [](https://opensource.org/licenses/Apache-2.0) [](https://arxiv.org/abs/2510.14686) [](https://deepwiki.com/jd-opensource/xllm)

📢 News

• 2026-07-06: 🎉 xLLM is officially donated to the OpenAtom Foundation!
• 2026-06-13: 🎉 We day-0 support the MiniMax-M3 model, please refer to the Deployment Document for deployment.
• 2026-04-24: 🎉 We day-0 support the DeepSeek-V4 model, please refer to the Deployment Document for deployment.

<details>
<summary>More News</summary>

• 2026-02-12: 🎉 We day-0 support high-performance inference for the GLM-5 model, please refer to the Deployment Document for deployment.
• 2025-12-21: 🎉 We day-0 support high-performance inference for the GLM-4.7 model.
• 2025-12-08: 🎉 We day-0 support high-performance inference for the GLM-4.6V model.
• 2025-12-05: 🎉 We now support high-performance inference for the GLM-4.5/GLM-4.6 series models.
• 2025-12-05: 🎉 We now support high-performance inference for the VLM-R1 model.
• 2025-12-05: 🎉 We build hybrid KV cache management based on Mooncake, supporting global KV cache management with intelligent offloading and prefetching.
• 2025-10-16: 🎉 We recently have released our xLLM Technical Report on arXiv, providing comprehensive technical blueprints and implementation insights.

</details>

Overview

xLLM is an efficient LLM inference framework, specifically optimized for Chinese AI accelerators, enabling enterprise-grade deployment with enhanced efficiency and reduced cost.

Highlights

• Top-tier Performance: Delivers high-throughput, low-latency inference through many advanced features.
• Mainstream Hardware Support: Purpose-built and deeply optimized for Chinese AI accelerators.
• Service-Engine Decoupled Architecture: Service layer handles scheduling and availability; engine layer handles computation.
• Enterprise-grade Deployment: Battle-tested at scale across JD.com's core retail business.

Hardware Support

Getting Started

• Quick Start
• Launch xLLM
• Online Service
• Offline Inference
• Supported Models

Community & Support

Acknowledgment

This project was made possible thanks to the following open-source projects:

• ScaleLLM - xLLM draws inspiration from ScaleLLM's graph construction method and references its runtime execution.
• Mooncake - Build xLLM hybrid KV cache management based on Mooncake.
• brpc - Build high-performance http service based on brpc.
• tokenizers-cpp - Build C++ tokenizer based on tokenizers-cpp.
• safetensors - xLLM relies on the C binding safetensors capability.
• Partial JSON Parser - Implement xLLM's C++ JSON parser with insights from Python and Go implementations.
• concurrentqueue - A fast multi-producer, multi-consumer lock-free concurrent queue for C++11.

Thanks to the following collaborating university laboratories:

• THU-MIG (School of Software, BNRist, Tsinghua University)
• USTC-Cloudlab (Cloud Computing Lab, University of Science and Technology of China)
• Beihang-HiPO (Beihang HiPO research group)
• PKU-DS-LAB (Data Structure Laboratory, Peking University)
• PKU-NetSys-LAB (NetSys Lab, Peking University)
• TJU-TANKLab (TANK Lab, Tianjin University)

Thanks to all the following developers who have contributed to xLLM.

<a href="https://github.com/jd-opensource/xllm/graphs/contributors">
<img src="https://contrib.rocks/image?repo=jd-opensource/xllm" />
</a>

Citation

If you think this repository is helpful to you, welcome to cite us:

@article{liu2025xllm,
  title={xLLM Technical Report},
  author={Liu, Tongxuan and Peng, Tao and Yang, Peijun and Zhao, Xiaoyang and Lu, Xiusheng and Huang, Weizhe and Liu, Zirui and Chen, Xiaoyu and Liang, Zhiwei and Xiong, Jun and others},
  journal={arXiv preprint arXiv:2510.14686},
  year={2025}
}