TurboFuzzLLM

Turbocharging Mutation-based Fuzzing for Effectively Jailbreaking LLMs in Practice

A state-of-the-art tool for automatic red teaming of Large Language Models (LLMs) that generates effective adversarial prompt templates to identify vulnerabilities and improve AI safety.

⚠️ Responsible Use

This tool is designed for improving AI safety through systematic vulnerability testing. It should be used responsibly for defensive purposes and developing better safeguards for LLMs.

Our primary goal is to advance the development of more robust and safer AI systems by identifying and addressing their vulnerabilities. We believe this research will ultimately benefit the AI community by enabling the development of better safety measures and alignment techniques.

📖 Table of Contents

• 🚀 Getting Started
• 🎯 Key Features
• 🔧 Method Overview
• 🔄 Architecture and Data Flow
• 📊 Results
• 🛡️ Applications
• ⚙️ Configuration
• 🤖 Supported Models
• 🧑‍💻 Development
• 📁 Codebase Structure
• 📂 Understanding Output
• 🔧 Troubleshooting
• 👥 Meet the Team
• Security
• License
• Citation

🚀 Getting Started

Prerequisites

• Python 3.8+ and pip
• Provider access: OpenAI API key for gpt-*/o1-*, AWS credentials for Bedrock/SageMaker (configure with aws configure)
• Optional local models: Hugging Face-compatible checkpoints (e.g., Gemma/Zephyr) for offline judge/target use

Install

git clone https://github.com/amazon-science/TurboFuzzLLM.git
cd TurboFuzzLLM
python -m venv .venv && source .venv/bin/activate   # optional but recommended
pip install --upgrade pip
pip install -e .

Network/cost safety: SageMaker endpoint deployment and Bedrock validation are blocked by default; pass --allow-endpoint-deploy explicitly when you intend to enable them.

Quick Start

1. Download seed templates:

   python3 scripts/get_templates_gptfuzzer.py

2. Run an interactive attack:

   python3 src/__main__.py answer --target-model-id gpt-4o --api-key YOUR_OPENAI_KEY

3. Batch attack HarmBench (AWS Bedrock):

   turbofuzzllm attack --target-model-id us.anthropic.claude-3-5-sonnet-20241022-v2:0 --max-queries 1000

Results appear under output/<date>/*/.

🎯 Key Features

• High Success Rate: Achieves >98% Attack Success Rate (ASR) on GPT-4o, GPT-4 Turbo, and other leading LLMs
• Efficient: 3x fewer queries and 2x more successful templates compared to previous methods
• Generalizable: >90% ASR on unseen harmful questions
• Practical: Easy-to-use CLI with statistics, search visualization, and logging
• Defensive Applications: Generated data improves model safety (74% safer after fine-tuning)

🔧 Method Overview

TurboFuzzLLM performs black-box mutation-based fuzzing to iteratively generate new adversarial red teaming templates. Key innovations include:

1. Expanded Mutation Space: New mutation operations including refusal suppression
2. Reinforcement Learning: Feedback-guided prioritized search
3. Intelligent Heuristics: Efficient exploration with fewer LLM queries
4. Template-Based Approach: Templates can be combined with any harmful question for scalable attacks

🔄 Architecture and Data Flow

High-Level Architecture

TurboFuzzLLM performs black-box mutation-based fuzzing to generate adversarial prompt templates for jailbreaking LLMs. It uses reinforcement learning to prioritize effective mutations.

Key Components

1. Fuzzer Core (fuzzer/core.py):

   • TurboFuzzLLMFuzzer: Main orchestrator class.
   • Manages questions, templates, mutations, evaluations, and statistics.

2. Models (llm/):

   • TargetModel: The LLM being attacked (e.g., GPT-4, Claude).
   • MutatorModel: LLM used for generating mutations (e.g., for paraphrasing templates).
   • JudgeModel: Determines if a response is "jailbroken" (vulnerable to the attack).

3. Mutation System (fuzzer/mutators.py, fuzzer/mutator_selection.py):

   • Various mutation operators: ExpandBefore, FewShots, Rephrase, Crossover, etc.
   • Selection policies: QLearning, UCB, Random, RoundRobin, MCTS, EXP3.

4. Template System (fuzzer/template.py):

   • Template class: Represents adversarial prompt templates.
   • Tracks ASR (Attack Success Rate), jailbreaks, parent/child relationships.

5. Template Selection (fuzzer/template_selection.py):

   • Policies for selecting which template to mutate next (reinforcement learning-based).

Data Flow

1. Initialization: Load initial templates, questions, and configure models.

2. Warmup Phase: Evaluate initial templates on subset of questions.

3. Mutation Loop:

   • Select template using selection policy (e.g., QLearning).
   • Select mutation using mutation policy.
   • Apply mutation to generate new template.
   • Evaluate new template on remaining questions.
   • Update selection/mutation policies based on results.
   • Repeat until stopping criteria (query limit, all questions jailbroken).

4. Evaluation: For each template-question pair:

   • Synthesize prompt (replace placeholder in template with question).
   • Query target model.
   • Judge response for vulnerability.
   • Track statistics and jailbreaks.

5. Output: Generate CSV files, logs, statistics, and visualization of template evolution tree.

📊 Results

🛡️ Applications

1. Vulnerability Identification: Discover prompt-based attack vectors in LLMs
2. Countermeasure Development:
   • Improve in-built LLM safeguards
   • Create external guardrails
3. Adversarial Training: Generate high-quality (attack prompt, harmful response) pairs for safety fine-tuning

⚙️ Configuration

Execution Modes

TurboFuzzLLM supports 4 operational modes:

Command Line Interface

Get help for any mode:

python3 src/__main__.py <mode> --help

Key Parameters

• Models:

  • --target-model-id: LLM to attack (e.g., us.anthropic.claude-3-5-sonnet-20241022-v2:0 for Bedrock, gpt-4o for OpenAI)
  • --mutator-model-id: LLM for mutations (default: gpt-4o)
  • --judge-model-id: LLM for judging success (default: gpt-4o)

• Query and Template Limits:

  • --max-queries: Maximum API calls (default varies by mode, e.g., 100 for answer, 4000 for attack)
  • --max-templates: Limit initial templates (default: 20 for answer, -1 for others)

• Selection Policies:

  • --template-selector: Template selection (ql, ucb, mcts, exp3, rand, rr; default: ql)
  • --mutation-selector: Mutation selection (ql, rand, rr; default: ql)

• Files and Datasets:

  • --templates-path: Path to initial templates CSV
  • --questions-path: Path to questions CSV (e.g., HarmBench dataset)

• Other:

  • --seed: Random seed for reproducibility (default: 0)
  • --num-threads: Threads for parallel evaluation (default: 1)
  • --api-key: API key for non-Bedrock models

Usage Examples

• Before runnign the following commands, please download a seed harm question set or build one of your own, e.g.,

  python3 scripts/get_questions_harmbench_text_standard.py \
  --output configuration/datasets/questions/harmbench/harmbench_behaviors_text_standard_all.csv

Interactive Mode

Test a single question:

python3 src/__main__.py answer --target-model-id gpt-4o --api-key YOUR_OPENAI_KEY

Batch Attack Mode

Attack multiple questions with defaults:

turbofuzzllm attack --target-model-id us.anthropic.claude-3-5-sonnet-20241022-v2:0 --max-queries 1000

Uses local HF models

turbofuzzllm attack \
  --target-model-id HuggingFaceH4/zephyr-7b-beta \
  --mutator-model-id HuggingFaceH4/zephyr-7b-beta \
  --judge-model-id cais/HarmBench-Llama-2-13b-cls \
  --judge-tokenizer cais/HarmBench-Llama-2-13b-cls \
  --max-queries 100

• Att: Please Install accelerate to enable device_map="auto" placement (pip install accelerate). Without it, local HF models fall back to CPU.

• Pleaes use the following command of smaller HFmodels if you have local compute limits. Att: These are minimal/demo-friendly models; they won’t give meaningful jailbreak results—use only for plumbing tests.

  turbofuzzllm attack \
  --target-model-id hf-internal-testing/tiny-random-GPT2LMHeadModel \
  --mutator-model-id hf-internal-testing/tiny-random-GPT2LMHeadModel \
  --judge-model-id cardiffnlp/twitter-roberta-base-offensive \
  --judge-tokenizer cardiffnlp/twitter-roberta-base-offensive \
  --max-queries 20 

customize the seed questions with your own, e.g.,

turbofuzzllm attack \
  --target-model-id HuggingFaceH4/zephyr-7b-beta \
  --mutator-model-id HuggingFaceH4/zephyr-7b-beta \
  --judge-model-id cais/HarmBench-Llama-2-13b-cls \
  --judge-tokenizer cais/HarmBench-Llama-2-13b-cls \
  --questions-path /path/to/your/questions.csv \
  --max-queries 10

Evaluate Templates

Test existing templates on a dataset:

turbofuzzllm evaluate --templates-path configuration/datasets/prompts/prompt_list.csv --questions-path configuration/datasets/questions/jailbreakbench/harmful-behaviors.csv

Legacy Mode

Run GPTFuzzer baseline:

turbofuzzllm legacy --template-selector mcts --mutation-selector rand --max-queries 4000

Common Questions

• What access do I need? API keys for OpenAI (--api-key) and AWS creds for Bedrock/SageMaker. Local HF models work for some judges/targets.
• When do I enable network deployments? Only when you pass --allow-endpoint-deploy; otherwise SageMaker/Bedrock setup is skipped for safety.
• How many queries? Defaults: 100 (answer), 4000 (attack/legacy). Increase for deeper search, respecting rate limits.
• How do I choose selectors? ql is the default; try ucb, mcts, or exp3 for exploration-heavy runs.
• Can I bring my own data? Yes—use --questions-path and --templates-path for custom CSVs.

🤖 Supported Models

• Targets and Mutators: OpenAI chat models (gpt-*, o1-*) via --api-key; AWS Bedrock foundations (e.g., Claude, Mistral) via AWS creds; SageMaker JumpStart/Hugging Face endpoints for hosted inference (requires --allow-endpoint-deploy); local Hugging Face models (Gemma/Zephyr-style IDs) for offline trials.
• Judges: GPTJudge (OpenAI), BedrockJudge, RoBERTaJudge (HF sequence classifier), HarmBenchJudge (HF causal LM), and SageMaker-hosted HarmBench/Llama Guard judges. Configure with --judge-model-id and --judge-tokenizer where applicable.
• Guardrails and validation: Bedrock guardrails supported through --guardrail-id/--guardrail-version. Remote endpoint validation/deployment is blocked by default; pass --allow-endpoint-deploy when you explicitly want SageMaker or Bedrock setup to run.

🧑‍💻 Development

• Install for development: pip install -e .
• Network/cost safety: by default the CLI avoids creating SageMaker endpoints or Bedrock calls; use --allow-endpoint-deploy only when you are ready to permit those operations.

Testing

• Fast check: pytest test/test_cli_and_utils.py
• Broader suite (may need creds/mocking): pytest
• Coverage: pytest --cov=turbofuzzllm

📁 Codebase Structure

TurboFuzzLLM/
├── .gitignore
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE
├── NOTICE
├── pyproject.toml
├── README.md
├── requirements.txt
├── setup.py
├── configuration/
│   ├── datasets/
│   │   ├── prompts/
│   │   │   ├── no_template.csv
│   │   │   └── README.md
│   │   └── questions/
│   │       ├── harmbench/
│   │       │   └── README.md
│   │       └── jailbreakbench/
│   │           └── README.md
├── scripts/
│   ├── fine_tune_data.py
│   ├── generate_stats.sh
│   ├── get_questions_harmbench_text_standard.py
│   ├── get_questions_jailbreakbench.py
│   ├── get_templates_gptfuzzer.py
│   ├── plots.py
│   ├── stats_to_csv.py
├── src/
│   └── turbofuzzllm/
│       ├── __init__.py
│       ├── __main__.py
│       ├── cli.py
│       ├── py.typed
│       ├── fuzzer/
│       │   ├── __init__.py
│       │   ├── core.py
│       │   ├── mutator_selection.py
│       │   ├── mutators.py
│       │   ├── question.py
│       │   ├── template.py
│       │   ├── template_selection.py
│       └── llm/
│       │   ├── __init__.py
│       │   ├── bedrock_judge.py
│       │   ├── bedrock_model.py
│       │   ├── gpt_judge.py
│       │   ├── harmbench_judge.py
│       │   ├── judge_model.py
│       │   ├── local_model.py
│       │   ├── model_utils.py
│       │   ├── model.py
│       │   ├── mutator_model.py
│       │   ├── openai_model.py
│       │   ├── roberta_judge.py
│       │   ├── sagemaker_huggingface_model.py
│       │   ├── sagemaker_jumpstart_model.py
│       │   ├── sm_harmbench_judge.py
│       │   ├── sm_llama_guard_judge.py
│       │   ├── target_model.py
│       └── utils/
│           ├── __init__.py
│           ├── cli_utils.py
│           ├── draw.py
│           ├── logging.py
│           ├── priority_queue.py
│           ├── prompt.py
│           ├── statistics.py
│           ├── timer.py
└── test/
    ├── conftest.py
    ├── test_cli.py
    ├── test_cli_and_utils.py
    ├── test_mutators.py
    ├── test_question.py
    ├── test_selectors.py
    ├── test_template.py
    ├── test_turbofuzzllm.py
    └── test_utils.py

📂 Understanding Output

Each run creates an output folder with the following structure:

output/<date>/<mode>_<target-model-id>_<start-time>/
├── templates.csv      # Summary of each template used
├── mutators.csv       # Performance metrics for each mutator
├── queries.csv        # Details of each LLM query
├── stats.txt          # Key metrics summary
├── details.log        # Detailed execution log
└── template_tree.dot  # Visualization of mutant search space

Output Files Description

• templates.csv: Contains all generated templates with their success rates
• mutators.csv: Performance analysis of different mutation operations
• queries.csv: Complete record of LLM interactions
• stats.txt: High-level metrics including ASR, query count, and timing
• details.log: Verbose logging for debugging
• template_tree.dot: Graphviz visualization of the template evolution tree

🔧 Troubleshooting

Common Issues

API Key Errors:

• Ensure your API keys are correctly set in environment variables or via --api-key parameter
• For Bedrock models, verify AWS CLI is configured with aws configure
• Check that your API keys have sufficient permissions/quota

Installation Problems:

• Module not found errors: Ensure all dependencies are installed with pip install -e .
• Python version issues: Requires Python 3.8+. Check with python --version
• AWS/conda issues: If pip fails, try installing in a conda environment
• protobuf/grpc deprecation warnings: Upgrade the dependencies in your venv with pip install -U protobuf grpcio
• Local Hugging Face models: Install accelerate to enable device_map="auto" placement (pip install accelerate). Without it, models fall back to CPU.

Runtime Errors:

• Model not supported: Check if your target model ID is supported. Use --help to see examples
• Rate limiting: If hitting API rate limits, try reducing --num-threads or adding delays
• Out of memory: For large templates/questions, increase system RAM or reduce batch sizes

Performance Issues:

• Slow execution: Increase --num-threads if you have multi-core CPU (default is 1)
• Too many API calls: Monitor query count with output files. Set appropriate --max-queries limits

Output Problems:

• Missing files: Check file permissions in the output directory
• Empty results: Verify templates and questions CSV files are properly formatted
• Garble output: Check log files for specific error messages

Getting Help

If you encounter issues not covered here:

1. Check the --help output for each mode
2. Review the example commands in this README
3. Open an issue on the GitHub repository with your error logs

👥 Meet the Team

• Aman Goel* (Contact: [email protected])
• Xian Carrie Wu
• Zhe Wang
• Dmitriy Bespalov
• Yanjun (Jane) Qi

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.

Citation

If you find this useful in your research, please consider citing:

@inproceedings{goel2025turbofuzzllm,
  title={TurboFuzzLLM: Turbocharging Mutation-based Fuzzing for Effectively Jailbreaking Large Language Models in Practice},
  author={Goel, Aman and Wu, Xian and Wang, Daisy Zhe and Bespalov, Dmitriy and Qi, Yanjun},
  booktitle={Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track)},
  pages={523--534},
  year={2025}
}