🌊 Aqua-TTS: GPT-SoVITS Low-Latency Inference Runtime on GPU

Built for low-latency voice conversation with your LoRA characters

中文 | English

Aqua-TTS is a GPU-optimized inference runtime purpose-built for real-time voice conversation — specifically, low-latency streaming TTS with your own GPT-SoVITS v3 LoRA character voices. It does not replace model weights — it replaces the execution strategy: static KV cache buffers, bucketed CUDA Graph capture/replay, and pre-compiled BigVGAN CUDA kernels. On an RTX 4070 Ti SUPER, this reaches 440–470 it/s T2S throughput and reduces model-side first-audio latency from 1.0–3.6 s to ~0.26–0.40 s. In the full streaming player pipeline, practical first-audio latency is typically 0.4–0.7 s depending on chunk length, audio device startup, cache state, and scheduling overhead — see Highlights for the full comparison.

Highlights

_{Latency definitions: TTFP benchmark = model-side first audio latency under warm-cache (table below). E2E first-audio = full pipeline including audio buffer and playback startup, typically 0.4–0.7 s in practice. Cold start = init + model load + first inference, dominated by BigVGAN CUDA kernel compilation (~2 min on first run, then cached).}

TTFP benchmark: NVIDIA GeForce RTX 4070 Ti SUPER (16 GB), PyTorch 2.5.1+cu121, float16, warm cache, static KV cache enabled, bucketed CUDA Graph pre-captured. Aqua-TTS numbers are median of 5 runs. Official baseline uses the same GPT checkpoint; TTFP for non-streaming baselines coincides with full utterance completion. See benchmarks/README.md for full methodology and raw repeats.

https://github.com/user-attachments/assets/581cef5f-f8ce-4570-81ae-a6c092698223

Features

• Static KV cache — pre-allocated scatter buffers eliminate per-step torch.cat overhead
• Bucketed CUDA Graph — 13 pre-captured graphs across 6 bucket sizes, no warmup jitter
• Pre-compiled BigVGAN — NVIDIA CUDA kernel auto-loaded from pre-built .pyd, with torch fallback
• Optional FlashAttention2 KV cache — experimental flash_attn_with_kvcache T2S path, disabled by default
• Streaming API — generator-based infer_stream() with early first-audio yield
• Built-in presets — fast / balanced / quality generation presets; full / minimal / lazy / off CUDA Graph presets
• Voice registry — map voice names to reference audio + prompt, with JSON persistence
• HTTP server — lightweight FastAPI server with streaming TTS endpoint, voice management, and health check
• Source install — pip install -e ".[runtime]" → from aquatts import TTSInferencer

Scope notice — Aqua-TTS is a self-contained runtime for GPT-SoVITS v3. It is not a plugin and does not track upstream changes. The techniques here — static KV cache, bucketed CUDA Graph, pre-compiled BigVGAN kernel — are documented in TECHNICAL.md and designed to be portable. If you need v4 support, aquatts/modeling/ and aquatts/_vendor/ are the right starting points for adaptation.

Known limitations — Windows + CUDA is the primary tested path. Linux passes unit tests but GPU-dependent paths (CUDA Graph, BigVGAN kernel) have not been validated on Linux hardware. macOS is not supported. TTFP varies with GPU model, audio device, chunk size, and model weights — numbers in this README are measured on an RTX 4070 Ti SUPER with specific v3 LoRA weights and should not be treated as universal. Only GPT-SoVITS v3 is supported.

Supported Languages

Aqua-TTS inherits GPT-SoVITS v3's language support. Pass the code to text_language / prompt_language — reference audio and target text can use different languages.

How it works

Aqua-TTS vendors overrides for two GPT-SoVITS files inside aquatts/_vendor/ using Python namespace packages (pkgutil.extend_path). When you import aquatts, the package automatically configures sys.path so the vendored files take precedence over the main GPT-SoVITS repo. Set GPT_SOVITS_HOME to point at your GPT-SoVITS installation — Aqua-TTS handles the rest:

aqua-tts/
├── aquatts/                       # Pure Python package (pip-installable)
│   ├── __init__.py                # sys.path configuration + lazy exports
│   ├── inferencer.py              # TTSInferencer — main entry point
│   ├── server.py                  # FastAPI HTTP server
│   ├── voice_registry.py          # Voice name → audio path mapping
│   ├── modeling/
│   │   ├── t2s_streaming.py       # T2SBlockWithStaticCache, CUDA Graph patch
│   │   └── t2s_flash_attn.py      # Optional FlashAttention2 KV-cache patch
│   ├── bigvgan/
│   │   ├── cuda/                  # Standalone CUDA kernel loader + sources
│   │   └── torch/                 # Pure-PyTorch fallback (resample, filter, act)
│   ├── inference/
│   │   ├── streaming.py           # Audio post-processing (fade, chunk finalization)
│   │   ├── params.py              # SoVITS parameter presets
│   │   └── presets.py             # Named presets (generation + CUDA Graph)
│   └── _vendor/
│       └── GPT_SoVITS/            # Vendored overrides (namespace packages)
│           ├── AR/models/
│           │   └── t2s_model.py   # Static KV + CUDA Graph T2S decoder
│           └── BigVGAN/alias_free_activation/cuda/
│               ├── load.py        # Pre-compiled kernel loader (MSVC auto-discovery)
│               ├── activation1d.py  # Fused anti-alias activation
│               └── *.cpp, *.cu, *.h  # NVIDIA BigVGAN CUDA kernel sources
├── benchmarks/                    # TTFP, T2S comparison, BigVGAN raw benchmarks
├── examples/                      # basic_usage.py, streaming_inference.py
└── tests/                         # Unit tests

See TECHNICAL.md for deep technical documentation.

Requirements

Linux CI (Ubuntu) passes for all unit tests. GPU-dependent paths (CUDA Graph, BigVGAN kernel) have not been tested on Linux hardware. macOS is not supported — Aqua-TTS requires CUDA.

On 6 GB cards, use cuda_graph_preset="lazy" or "off" to reduce VRAM pressure from pre-captured graphs.

Installation

1. Install GPT-SoVITS

You need a working GPT-SoVITS v3 installation. Aqua-TTS imports from it.

git clone https://github.com/RVC-Boss/GPT-SoVITS.git
cd GPT-SoVITS
pip install -r requirements.txt

Aqua-TTS has been tested against GPT-SoVITS v3 (2025-04-01 release).

2. Install PyTorch

Install PyTorch matching your CUDA version before installing Aqua-TTS — it is not included in the package dependencies so you can pick the right CUDA wheel:

# Example for CUDA 12.1 — see https://pytorch.org/get-started/locally/ for other versions
pip install torch --index-url https://download.pytorch.org/whl/cu121

Aqua-TTS has been tested with PyTorch 2.5.1+cu121 on an RTX 4070 Ti SUPER.

3. Install Aqua-TTS

git clone https://github.com/Lucas1479/Aqua-TTS.git
cd Aqua-TTS

# Core + runtime dependencies
pip install -e ".[runtime]"

# Or with HTTP server support
pip install -e ".[runtime,server]"

PyPI package is planned but not yet published.

Extras:

• [runtime] — soundfile, librosa, peft (needed by TTSInferencer)
• [server] — FastAPI + uvicorn (includes [runtime] automatically)

4. Configure GPT-SoVITS path

Set the GPT_SOVITS_HOME environment variable to your GPT-SoVITS repo root:

# Windows (PowerShell)
$env:GPT_SOVITS_HOME = "C:\path\to\GPT-SoVITS"

# Linux / macOS
export GPT_SOVITS_HOME=/path/to/GPT-SoVITS

5. Download pretrained models

Aqua-TTS needs BigVGAN v2 pretrained weights from Hugging Face. Download them directly into your GPT-SoVITS repo with huggingface_hub:

pip install huggingface_hub
python -c "
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id='nvidia/bigvgan_v2_24khz_100band_256x',
    local_dir='GPT_SoVITS/pretrained_models/models--nvidia--bigvgan_v2_24khz_100band_256x',
)
"

The models--nvidia--bigvgan_v2_24khz_100band_256x directory name is the HuggingFace Hub cache format — use it exactly as shown.

6. Prepare model weights

• GPT weights (T2S): A Text2SemanticLightningModule checkpoint (e.g., s1v3.ckpt or self-trained xxx-e15.ckpt).
• SoVITS weights (vocoder): A SynthesizerTrn checkpoint with optional LoRA (e.g., xxx_e2_s174_l32.pth).
• Reference audio: A 3-10 second 24 kHz WAV file with known transcript.

Quick sanity check

Once GPT_SOVITS_HOME is set and models are in place, verify the package loads correctly:

python -c "from aquatts import TTSInferencer; print('aquatts OK')"

If this prints aquatts OK, the install is good. If it errors, check that GPT_SOVITS_HOME points to the right directory and all pretrained models are downloaded.

Usage

Python API

import os
os.environ["GPT_SOVITS_HOME"] = "/path/to/GPT-SoVITS"
os.environ["ENABLE_CUDA_GRAPH"] = "1"

from aquatts import TTSInferencer

tts = TTSInferencer(
    device="cuda",
    gpt_path="GPT_weights_v3/s1v3.ckpt",
    sovits_path="SoVITS_weights_v3/your_model.pth",
    cuda_graph_preset="full",  # "full", "minimal", "lazy", or "off"
)

# Streaming generation — yields (sample_rate, audio_chunk, text) tuples
for sr, chunk, text in tts.infer_stream(
    text="こんにちは、世界！",
    ref_audio_path="reference audio/ref_audio.wav",
    prompt_text="こんにちは。今日はいい天気ですね。",
    text_language="日文",
    prompt_language="日文",
    preset="fast",  # "fast", "balanced", or "quality"
):
    # chunk is float32 numpy array at `sr` Hz
    # text is the text segment being spoken
    pass

See examples/basic_usage.py and examples/streaming_inference.py for runnable examples.

Presets

Two layers of presets control quality/speed trade-offs:

Generation presets (per-request, via infer_stream(preset=...)):

CUDA Graph presets (system-level, via TTSInferencer(cuda_graph_preset=...)):

Optional FlashAttention2 T2S path (experimental, disabled by default):

Aqua-TTS can replace the q_len=1 T2S static-KV attention step with
flash_attn_with_kvcache. This is an opt-in path because the useful valid
mode attends over the true KV length rather than the full zero-padded bucket,
so output can differ slightly from the default SDPA bucket path.

Enable it for long first-sentence or long-form T2S throughput experiments:

export AQUATTS_T2S_FLASH_ATTN=1
export AQUATTS_T2S_FLASH_ATTN_MODE=valid  # valid | bucket

or from Python:

tts = TTSInferencer(..., use_flash_attn=True, flash_attn_mode="valid")

Keep it off for short demo lines, early-cut first-sentence demos, or strict
regression comparisons. On short 448/512-bucket shapes the extra path is often
within noise or slightly slower; it becomes more interesting on longer 768+
bucket decode.

Directional AB test on the shared Aqua/Amadeus v3 T2S path
(RTX 4070 Ti SUPER, PyTorch 2.5.1+cu124, flash-attn 2.7.0.post2,
CUDA Graph pre-captured, BigVGAN CUDA kernel cached):

Treat these as guidance rather than headline numbers: FlashAttention2 helps
most when attention bucket length is the bottleneck, while Aqua's default
CUDA-Graph SDPA path is already very fast for short conversational turns.

from aquatts import apply_preset, list_presets

print(list_presets())  # ["balanced", "fast", "quality"]
params = apply_preset("fast", overrides={"top_k": 5})

HTTP Server

Start a lightweight inference server:

python -m aquatts.server \
    --gpt-model GPT_weights_v3/s1v3.ckpt \
    --sovits-model SoVITS_weights_v3/model.pth \
    --cuda-graph-preset full \
    --host 127.0.0.1 --port 8000

# With authentication (required when binding to a non-loopback address)
python -m aquatts.server ... --host 0.0.0.0 --api-key mysecrettoken
# or: export AQUA_API_KEY=mysecrettoken

Endpoints:

# Streaming TTS — returns raw float32 PCM (24kHz, mono, little-endian)
curl -X POST "http://127.0.0.1:8000/tts?text=Hello&voice=alice" --output - \
  | play -t raw -r 24k -e floating-point -b 32 -c 1 -

# Download WAV file
curl -X POST "http://127.0.0.1:8000/tts/file?text=Hello&voice=alice" -o output.wav

Voice Registry

Manage multiple characters/voices without repeating paths:

from aquatts import Voice, VoiceRegistry

registry = VoiceRegistry(json_path="./voices.json")

registry.add(Voice(
    name="alice",
    ref_audio_path="./voices/alice_ref.wav",
    prompt_text="こんにちは。今日はいい天気ですね。",
    prompt_language="日文",
))

voice = registry.get("alice")
for sr, chunk, text in tts.infer_stream(
    text="Hello world",
    ref_audio_path=voice.ref_audio_path,
    prompt_text=voice.prompt_text,
    prompt_language=voice.prompt_language,
):
    pass

On the HTTP server, pass voice instead of ref_audio_path:

curl -X POST "http://127.0.0.1:8000/tts?text=Hello&voice=alice"

For production deployments, pass an explicit registry path:

python -m aquatts.server ... --voice-registry /data/voices.json
# or
export AQUA_VOICE_JSON=/data/voices.json

Configuration

Benchmarks

# T2S comparison (official vs official+CUDA Graph vs Aqua-TTS)
# Replace with your own GPT checkpoint (s1v3.ckpt or a self-trained .ckpt)
python benchmarks/t2s_comparison_bench.py --gpt-model GPT_weights_v3/xxx-e15.ckpt

# TTFP benchmark (streaming end-to-end)
python benchmarks/aqua_ttfp.py \
    --gpt-model GPT_weights_v3/xxx-e15.ckpt \
    --sovits-model SoVITS_weights_v3/xxx_e2_s174_l32.pth \
    --ref-audio "reference audio/ref_audio.wav" \
    --ref-text "transcript of reference audio"

# BigVGAN raw kernel timing
python benchmarks/bigvgan_raw_bench.py

See benchmarks/README.md for full methodology and results.

Speaker Demo

pip install -e ".[playback]"
python examples/play_ete.py --gpt-sovits-home /path/to/GPT-SoVITS-v3lora

The demo plays three Kurisu-style Japanese utterances (short, medium, long) through PyAudio and prints TTFP plus live T2S throughput for each scripted line. Pass --show-total to also print audio duration, wall time, and RTF.
Playback demos default to the low-latency live profile (--top-p 1 --speed 1.1 --sample-steps 4 --how-to-cut 按标点符号切) with queue-based playback, chunk merging, short fades, and padding to reduce chunk-boundary artifacts. To inspect whole-utterance T2S throughput without punctuation splitting, pass --how-to-cut 不切.

For recording an interactive demo, load the voice once and type any Japanese text:

python examples/live_talk.py --gpt-sovits-home /path/to/GPT-SoVITS-v3lora

Acknowledgements

Aqua-TTS was inspired by GENIE-TTS, which demonstrated that a focused, self-contained inference runtime could meaningfully close the latency gap in GPT-SoVITS. That framing — optimise the runtime, not the model — shaped the direction of this project.

License

MIT — see LICENSE.

Third-party code:

• GPT-SoVITS: vendored aquatts/_vendor/GPT_SoVITS/AR/models/t2s_model.py is based on GPT-SoVITS (MIT) — see NOTICE.
• NVIDIA BigVGAN: CUDA kernel sources under Apache 2.0 — see NOTICE.
• alias-free-torch: aquatts/bigvgan/torch/ adapted under Apache 2.0 — see NOTICE.

Development

git clone https://github.com/Lucas1479/Aqua-TTS.git
cd Aqua-TTS
pip install -e ".[runtime]"
pip install -r requirements-dev.txt

export GPT_SOVITS_HOME=/path/to/GPT-SoVITS
python -m pytest tests/ -v

See CONTRIBUTING.md for contribution guidelines and CHANGELOG.md for release history.