TextCall

<h1 align="center">TextCall</h1>

Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning

TextCall is the official implementation for training and evaluating
visual-reasoning models that retain tool calls as textual reasoning context.

Contents

• Overview
• Availability
• Quick Start
• Configuration
• Data
• Evaluation
• Training
• Implementation Notes
• Documentation
• Related Work
• License
• Acknowledgments
• Citation

Overview

• Tool-call traces remain in the context even when returned images are omitted.
• The repository includes SFT, RL, inference, and benchmark entry points.
• Training and evaluation frameworks are pinned as Git submodules. Compatibility
  fixes live under patches/ instead of modifying upstream code.

Availability

Quick Start

Requirements

Clone the repository with its pinned third-party dependencies, then install the
full environment with uv:

git clone --recurse-submodules https://github.com/ant-research/TextCall.git
cd TextCall

UV_EXTRA_INDEX_URL=https://download.pytorch.org/whl/cu124 uv sync --extra all
cp .env.example .env
# Edit .env, then load it into the current shell.
source .env

Alternatively, the setup script initializes submodules, installs dependencies,
and runs the checks and offline smoke pipeline:

bash scripts/setup/run_all.sh

See the quickstart for smoke tests and the
environment guide for detailed hardware requirements.

Configuration

Set local paths and credentials in .env, then run source .env; never commit
API keys. The most
frequently used settings are:

The complete template is .env.example. Evaluation-specific
options are documented in docs/evaluation.md.

Data

Training data comes from the public DeepEyesV2 Hugging Face datasets:

• SFT: honglyhly/DeepEyesV2_SFT
• RL: honglyhly/DeepEyesV2_RL

Download the datasets from Hugging Face and set the local paths in .env or on
the command line.

Evaluation

Evaluate a checkpoint on a supported benchmark:

bash scripts/benchmark/run_public_eval.sh /path/to/checkpoint VStarBench

For TextCall bypass checkpoints, set BYPASS=1. A multi-GPU example with four
data-parallel replicas and tensor parallelism of two is:

DP_REPLICAS=4 TP=2 NPROC=16 BYPASS=1 \
  bash scripts/benchmark/run_public_eval.sh \
  /path/to/checkpoint "VStarBench,HRBench4K,MMStar"

See docs/evaluation.md for supported benchmarks, modes,
environment variables, and troubleshooting.

Training

Supervised Fine-Tuning

TEXTCALL_BASE_MODEL=/path/to/Qwen2.5-VL-7B-Instruct \
  bash scripts/training/run_public_sft.sh \
  configs/llamafactory/public_sft_lora.yaml

Reinforcement Learning

TEXTCALL_MODEL_PATH=/path/to/sft_checkpoint \
  bash scripts/training/run_public_rl.sh configs/verl/public_rl.yaml

For the complete training and evaluation workflow, follow
docs/full_reproduction.md.

Implementation Notes

<details>
<summary><strong>Repository structure</strong></summary>

TextCall/
├── third_party/              # Clean, pinned upstream submodules
│   ├── verl/                 # RL training framework
│   ├── LlamaFactory/         # SFT training framework
│   └── VLMEvalKit/           # Evaluation framework
├── patches/                  # Runtime compatibility patches
├── eval/                     # VLMEvalKit extensions
├── rl/                       # RL data, rewards, and agent loop
├── configs/                  # SFT, RL, and evaluation configs
├── scripts/                  # Setup, training, and benchmark entry points
├── data/                     # Sample inputs for offline smoke tests
├── tests/                    # Lightweight data and evaluation tests
└── docs/                     # Reproduction documentation

</details>

Third-party code is never edited in place. Framework compatibility changes are
applied through patches/, and custom VLMEvalKit files are installed from
eval/vlmevalkit_extensions/ during setup. Dependency provenance is recorded
in THIRD_PARTY.md, and the patch policy is documented in
PATCHES.md.

Documentation

• Quickstart
• Environment
• Evaluation
• Full reproduction
• Troubleshooting

Related Work

• Thinking with images —
  OpenAI's overview of models integrating visual information directly into the
  reasoning process.
• DeepEyes
  (code) — learns active visual
  perception through reinforcement learning.
• DeepEyesV2
  (code) — extends this direction
  to multimodal agents using code execution and search.

License

TextCall is released under the Apache License 2.0. Third-party
submodules retain their respective upstream licenses; see
THIRD_PARTY.md for versions and provenance.

Acknowledgments

This repository builds on
Qwen2.5-VL,
DeepEyesV2,
verl,
LLaMA-Factory, and
VLMEvalKit. We thank their
contributors for making these resources available.

Citation

@misc{shao2026thinkingtoolspixelstool,
  title         = {Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning},
  author        = {Jiahao Shao and Yuanbo Yang and Yiyi Liao and Yujun Shen and Ceyuan Yang and Yinghao Xu},
  year          = {2026},
  eprint        = {2608.09682},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2608.09682}
}