LAION MOSS Local-Transformer 1.5 — Voice Acting (4.55B, 48 kHz)

The public home of LAION's expressive voice-acting model built on the MOSS-TTS-Local-Transformer-v1.5
architecture: fantasy characters (orc / dragon / fairy / goblin), shouting, whispering, and vocal
bursts (laughs, gasps, sighs, giggles, moans) driven by natural-language director instructions,
in English and German, at native 48 kHz.

⚠️ Which checkpoint to use — read this first

	repo	use it for
✅ CURRENT — v2	laion/moss-tts-local-transformer-4.55b-voice-acting-v2	everything new. This is the base that all published LoRA adapters are trained against — emotion LoRAs, character LoRAs and the vocal-burst adapters. An adapter trained on v2 will not behave correctly on the v1 checkpoint.
previous	laion/moss-tts-local-transformer-4.55b-voice-acting	reproducing older results on this page. Merged, Apache-2.0, no adapter handling needed.

Most of the demo grids linked below were generated with v2. If you are pairing this
model with any LoRA from LAION, use v2.

🎛️ Vocal-burst LoRA adapters (64 classes):
laion/vocal-burst-lora-adapters
— one adapter per non-verbal vocal burst (chuckle, gasp, sob, whistle, moan, …), each
confirmed by a listening audit against real recordings and a no-LoRA baseline.
🎧 Listen to all 77 classes side by side →
(real recordings vs no-LoRA baseline vs top-3 per rank, with all scores)

🎬 Casting sessions — 9 agents, 3 rounds each:
🎧 projects.laion.ai/moss-acting-casting →
— ~30 s performances with a dramatic arc, each written as 2–4 parts, generated best-of-16 and
assembled as a whole. Every take lists the agent's intended arc, the exact GENERAL/SCRIPT
prompt per part, all LoRAs and merge doses, the sampling, the voice reference used, and both
the agent's own score and a listening model's.
(Hosted in its own repo — this repo's Pages
build fails now that it is past ~500 MB.)

🇩🇪 German Mediathek HQ LoRA adapters (3):
laion/moss-mediathek-hq-lora
— rank 16/32/64 trained on 43,612 segments / 185 h of high-quality expressive German
public-broadcast speech (an emotion half at EmoNet > 2.5, class-balanced, plus an equal quality
half ranked by vocal-burst blend + genuineness). Take r64_epoch3.
🎧 7 emotions × German/English, with and without the emotion adapter →
It is a German adapter: English output runs long (10–22 s for a 13-word line vs 3.7–4.8 s
for German). Use it for German; for English, control length explicitly.

📣 Sports-commentator LoRA adapters (3):
laion/moss-sports-commentator-lora
— the three best cells of a 13-cell rank × epoch sweep, trained on 820 filtered English
generations + 468 real German Mediathek commentary segments.
🎧 Listening evaluation →
Read the headline before using them: the un-adapted base model (1.750/2) is statistically
indistinguishable from real human commentary (1.775/2), so the task is near-saturated and
no cell survives multiple-comparison correction. The adapters raise consistency (97 % vs
75 % of clips at the top rating), not capability.

🏆🎭 THE FINAL SHOWCASE — 13 Character Voices, Directed by Prompt

The capstone of the character voice-acting study: the round-5 best-of-64 champions for all 13 voices
(zombie · ork · goblin · dragon · evil-ghost · mouse · fairy · ranting · pain-scream · ASMR man/woman · grieving woman/man),
each with its exact reusable prompt, listenable champion audio + full VoiceNet profile, the complete 13-principle
prompting playbook distilled from all five rounds (with a copy-paste template and an effect→words lookup), and the
round-1→5 improvement arc — all zero-shot, no reference audio.
→ projects.laion.ai/.../voices.html

🎧 Hear it: production best-of-64 grid
— top-3 takes per group across 40 emotions with full quality scores, generated with this model.

🗣️ Cross-lingual voice cloning from Japanese references:
German & English from Japanese anime reference audio
— 20 distinct Japanese reference speakers (verified: mean pairwise ECAPA cosine 0.105), prompted
in German and English across four emotions. Identity does NOT transfer: 0.188 mean ECAPA over
all 20 references / 1,280 generations, against a 0.105 unrelated-speaker floor and 1.000
self-similarity. (An earlier version of this page quoted 0.416 — that came from a single
reference measured before the full scoring job had run.) What does carry is pitch: r = 0.591
between reference and generation median F0, though compressed toward the model's own register
(182 Hz → 157 Hz; 7 of 20 references above 200 Hz vs 1 of 20 generations). Treat this as a way to
mine new character voices, not as a cloning method. The other measured finding is that the
emotion-LoRA merge dose destroys speaker identity monotonically — 0.62 at dose 0, 0.50 at
dose 1.0, −0.03 at dose 1.5, i.e. below the unrelated-speaker floor. Keep emotion merges
≤ 0.5 when reference fidelity matters. Per-emotion pages:
neutral ·
angry ·
sad ·
amused.

🎧 NEW — 25 originals vs their 3 best reinterpretations
— side-by-side players: the source recording next to the three highest-reward re-performances of the
same script out of 64 candidates, stratified across 25 distinct edge-case categories. Each clip
shows its reward decomposition (emotion similarity / target strength / burst blend / genuineness),
WER, detected vocal-burst spans, ASR text and the full prompt. Dataset:
laion/dramabox-reinterpretations-top3.

🎭 Reinterpretations: original vs top-3 side-by-side
— 30 full DramaBox performances, each voice-cloned and re-performed 64×; the 3 best takes (ranked by
(blend + genuineness + 1.25× 42-dim Empathic-Insight emotion-profile similarity) × invWER) next to the
original recording. Full 2,953-group dataset: laion/moss-local-dramabox-full-reinterpretations-64.

🏆 Top-3-by-reward previews (reward = (blend + genuineness + target-emotion EI) × invWER):
v3 grid
— 40 groups (one per emotion), top 3 of 64 takes each, voice-cloned from a unique reference speaker with
double-length scripts (dataset) ·
v4 “emorant” grid
— maximum-intensity delivery prompting, 40 groups from the first emotions of the (still running) 512k-clip run
(dataset).

📚 Read it: full experiment overview
— every temperature/scaling, best-of-N, reward-variant (V1–V6), VoiceCLAP-small blend-scorer, FAIR-eval, character/emotion and voice-cloning study we ran with this model, with results and links to the live grids.

🚀 Serving on SGLang (A100) — setup, traps & speed: for-dummies guide
— exact sgl-omni install + serve command, the full /v1/audio/speech API (reference-less / voice-clone / streaming), every startup trap with its fix, and a measured tuning chapter: ~0.74 s TTFA, ~4.5× single-stream, and ~35× realtime (≈35 audio-hours per A100-hour) at concurrency 96 for offline dataset generation.

🎭 Character archetype prompt study: evolving director instructions (no reference audio)
— an evolutionary search for the natural-language performance instruction that best embodies each of 10 character/style archetypes (zombie, orc, fairy, mouse, dragon, goblin, ASMR man/woman, ranting, pain-scream), scored by a VoiceNet 57-dim reward (blend + genuineness + 1.5·character). ~60 listenable takes with per-clip voice profiles, the full reward method, and a consolidated lessons playbook (describe acoustics not labels, balance beats char-maxing, one director-burst max, escalation phrasing).

🎭🔥 Round 2 — emotion-aware reward: directing character voices with emotion
— the same 10 archetypes re-optimized with an EmoNet (Empathic-Insight-Voice-Plus) emotion score added to the reward (0.5·blend + 0.5·genu + 2.0·char_vn + 1.5·char_emo; blend/genu halved, character pushed harder, prompts more extreme). Shows the round-1 → round-2 jump per archetype (e.g. the ranting fix: char 0.65 → 0.90 with words that still land; goblin gains Amusement 3.5 / Malice 2.1; zombie gains Helplessness/Fatigue/Pain) and a new emotion-layer lessons playbook.

🎭🎚️ Round 3 — resonance · pitch · dominance: directing character voices with body
— the same 10 archetypes re-optimized once more, now exploiting three further families of VoiceNet dimensions: resonance (chest/throat/nasal/head), pitch & register (how high/low, how wide the pitch swings) and dominance (commanding vs meek). Prompts learn to name a hollow chest groan, violent pitch swings, low flat menace or meek and powerless. Shows the round-1 → 2 → 3 char progression, the newly-rewarded dim values per take, and a resonance/pitch/dominance lessons playbook (e.g. the rant's dominance framing lifts char to 0.88; zombie 0.75 → 0.81; goblin 0.66 → 0.74).

🎭🎯 Round 4 — weighted dims & emotions, + 3 new voices: tuning the reward and building new characters
— introduces per-dimension (vn_w) and per-emotion (emo_w) reward weights so the search chases the two or three signals that define a character. Four archetypes are re-tuned (goblin made more evil — Malevolence ×2.5, char_emo 0.28 → 0.67; mouse pushed to an extreme high pitch — RANG ×2.5, char 0.76 → 0.87; pain-scream with real screams written into the text — "Aaah! … Aaargh!" — and Arousal ×2.5, char_emo 0.81 → 0.94; ranting with arousal up-weighted), and three brand-new voices are built from scratch: a breathy evil ghost (airy hollow whispered menace), a crying woman and a crying man (grief split by a single gender phrase; Sadness ×2.5, char_emo 0.92 / 0.89). Per-take weighted-dim values, firing emotions, and a weighting-behaviour lessons playbook.

🎭🏆 Round 5 — the FINAL showcase (all 13 voices): 13 Character Voices, Directed by Prompt
— the definitive capstone page. The round-5 best-of-64 champions for every one of the 13 characters, each with its
exact reusable director instruction, champion audio + a freshly recomputed 57-dim VoiceNet profile and procedural caption,
its char / char_emo / blend / genu / invWER scores, signature dimensions and firing emotions. Adds a complete "how to prompt any
character voice" playbook (13 principles, reusable template, effect→words lookup, per-character round-5 recipes), the full
round-1→5 improvement arc table, and the compute/method footer. Round 5 introduced per-character blend/genu weights on top of
the vn_w/emo_w levers and pushed every voice to its extreme (ork hatred char_emo 0.85→0.93; grieving man overt sobbing 0.89→0.99;
goblin caught cleaner and more malevolent; ASMR-man serenity 0.67→0.88). Everything zero-shot, no reference audio.

🎭🎬 Character groups — best-of-64 grid (live snapshot): one champion instruction × many spoken lines
— a browsable grid built from the growing laion/moss-character-voices-bestof64 dataset. Each group holds a character's fixed champion instruction paired with a different Gemma-authored spoken text; the model samples 64 takes per group, reward-ranked within the group, and each card plays the top-3 of 64 (① Champion / ② Runner-up / ③ Third) with its reward / char / char_emo / blend / genu / invWER. This is a snapshot of the first characters generated so far (dragon + the two ASMR voices, EN + DE); more creatures, the intense voices and the grief voices land as generation continues.

📘 Practical cheat-sheet: PROMPTING_GUIDE.md — a for-dummies, per-archetype guide distilling all three rounds: copy-paste champion instructions, a short recipe for each character (which acoustic + resonance + pitch + dominance + emotion words, which bursts, what to avoid), a general principles list, and a reusable template.

🧩 Fine-tuning prompt-format proposal: sentence-level caption format
— a design document for a fine-tuning format that gives both a global speaker/scene description (in instruction) and per-sentence delivery control interleaved inline in text as (caption) words [pause Xs] …, using MOSS's verified inline markup ([pause Xs], ${token:N}). The per-sentence (captions) are derived automatically from audio via the four LAION scoring models (Empathic-Insight 42 emotions · VoiceNet 57 dims · genuineness · vocal-burst blend) and a deviation-from-baseline (z-score) recipe — with real captions produced by the existing voicenet_v3grid/caption.py proof-of-concept, worked examples, and design recommendations.

🎬 DramaBox best-of-4 TTS benchmark: vanilla two-scene CUT TO: benchmark
— 4 reward-ranked seed variants per two-scene CUT TO: prompt with listenable takes, the best-of-k quality/compute trade-off, measured timing, and the full k=1..32 seed-scaling walltime table. (Now lives in the Voice-Acting-Pipeline repo.)

Everything in this repository is Apache-2.0 (code) / CC-BY-4.0 (docs & sample pages).

The two LAION MOSS voice-acting models

In our side-by-side listening and scoring, this 4.55B model gives subjectively better results —
higher bandwidth out of the box, cleaner word accuracy, comparable emotional range — while being
smaller and faster. Use the 8B sibling when you need its broader multilingual coverage.

Training data

DramaBox-style voice-acting distillation, Gemini TTS distillation (audio-matched fixed prompts with
MOSS RVQ tokens), German Emolia emotional speech, and podcast snippets from publicly available
podcasts on the web. Vocal-burst / strong-emotion filtered; rank-256 LoRA (α=256) over all attention
and MLP projections on top of a full DramaBox fine-tune, then merged into the released weights.

Quick start

pip install "transformers>=4.45" torch torchaudio safetensors huggingface_hub

import torch, torchaudio
from transformers import AutoProcessor, AutoModel

REPO = "laion/moss-tts-local-transformer-4.55b-voice-acting"
DEVICE = "cuda"

proc = AutoProcessor.from_pretrained(REPO, trust_remote_code=True,
                                     codec_path="OpenMOSS-Team/MOSS-Audio-Tokenizer-v2")
proc.audio_tokenizer = proc.audio_tokenizer.to(DEVICE).eval()
model = AutoModel.from_pretrained(REPO, trust_remote_code=True,
                                  dtype=torch.bfloat16,
                                  attn_implementation="sdpa").to(DEVICE).eval()

msg = proc.build_user_message(
    text="You dare enter my domain? Then you shall not leave alive!",
    instruction="As a mighty orc warrior with a deep, guttural, growling voice, "
                "bellow these words with brutal force.",
    language="English",
    tokens=120,                       # ≈ words × 6 codec frames (12.5 Hz) — pacing dial
)
batch = proc([[msg]], mode="generation")
with torch.no_grad():
    out = model.generate(input_ids=batch["input_ids"].to(DEVICE),
                         attention_mask=batch["attention_mask"].to(DEVICE),
                         max_new_tokens=600, do_sample=True,
                         audio_temperature=0.8, audio_top_p=0.95,
                         audio_top_k=25, audio_repetition_penalty=1.1)
wav = proc.decode(out)[0].audio_codes_list[0]     # (channels, T) @ 48 kHz
torchaudio.save("orc.wav", wav.cpu().float(), 48000)

Prompt format (DramaBox style): instruction = the performance description / stage directions
(vocal bursts as director notes like (gasping)); text = the words to speak.
Voice cloning: pass reference=[codes] from proc.encode_audios_from_path([...], n_vq=12); an
empty instruction gives the highest speaker similarity (measured).

Fast inference — what actually matters (all measured on A100-80GB)

1. Batch by repetition. Repeat the prompt's input_ids n× and call generate once.
   Batch 64 → 0.87 s per ~15 s clip; batch 128 → 0.83 s. Best-of-N groups get this for free
   (batch = group ⇒ identical lengths ⇒ zero padding waste).
2. SDPA + bf16. Flash-attention 2.x is incompatible with this checkpoint's remote-code
   attention (reshape mismatch in the packed-QKV path) — SDPA is the supported fast path.
3. Token budget = pacing dial. tokens ≈ words × 6 (12.5 Hz frames ≈ 2.1 words/s) avoids
   mid-sentence cut-offs on slow emotional delivery; smaller budgets force a rushed pace.
   max_new_tokens ≈ 2.2 × tokens + 300.
4. Keep the output raw. The model emits native 48 kHz; restoration/enhancement passes
   (e.g. 48→16→48 kHz vocoders) audibly soften it.
5. Score in the same process if you rank takes: batched bf16 ASR (Parakeet, batch 32 →
   0.0097 s/clip, 134× vs naive) + one VoiceCLAP encode reused for several quality heads. A fused
   generate→score worker sustains ~1.1–1.3 s/clip/GPU end-to-end.

Full write-up with every measurement: docs/fast_inference_insights.html
(rendered).
Bulk-generation recipe (prompt sources, rewards, throughput planning): SYNTHETIC_DATA_RECIPE.md.

Going faster: SGLang, and streaming

Update — MossTTSLocal now has an SGLang path. SGLang-Omni
ships a moss_tts_local model, so this model can be served today behind an OpenAI-compatible
POST /v1/audio/speech endpoint with continuous batching, a staged pipeline and CUDA graphs.
We ran it on JUPITER (JSC — aarch64 Grace + GH200 120 GB, Slurm, no Docker) and measured
27.7× realtime at concurrency 32 and 31.2× at concurrency 128 on one GH200, versus
18.8× realtime for our tuned transformers batched-generate() baseline on the same
GPU — 1.47× to 1.66× faster, saturating around concurrency 96–128.
→ sglang-omni-jupiter/ — full runbook: the working environment,
the five install/launch failure modes with symptom → cause → fix, the Slurm launcher and
benchmark client, an Apptainer definition (Docker is not runnable on HPC), and a
troubleshooting table keyed by exact error string.
Caveat, stated up front: only ×realtime is comparable between the two systems — ms/clip is not,
because the two measurements used different clip lengths. And we did not reproduce the
"~35× on A100" figure quoted below.

The upstream OpenMOSS MOSS-TTS repository ships an
SGLang backend for the MossTTSDelay architecture —
a fused TTS+codec serving path with continuous batching, reported ~3× generation throughput.
That applies directly to our 8B delay sibling.

For this MossTTSLocal model there is no official SGLang integration yet, but the port is
well-scoped: the OpenMOSS fork already contains moss_tts_delay.py / moss_tts_delay_with_codec.py
model files to use as templates; the local-transformer head predicts the 12 RVQ codebooks per frame
with a small per-frame transformer on top of the backbone, which fits SGLang's model interface —
the backbone gets continuous batching + radix caching, the local head runs per decoded frame.
Expected gains mirror the delay backend (~2–3× over HF generate).

Streaming: the local-transformer family is OpenMOSS's streaming-oriented design — audio
frames are complete as they are emitted (12 codebooks per 80 ms frame), so playback can start
after the first few frames: decode incrementally with chunked MOSS-Audio-Tokenizer-v2 calls
(e.g. 12–25-frame windows ≈ 1–2 s of audio with a 1-frame overlap) instead of waiting for the
full sequence. For fully interactive agents, see also OpenMOSS's dedicated low-latency models
(MOSS-TTS-Realtime, ~180 ms TTFB, and the 100M MOSS-TTS-Nano with CPU streaming at 48 kHz) —
this repo's model targets quality-first batch and near-real-time use.

Evaluation corpus

40 emotions × 100 prompts × 64 takes (256k clips) generated with this model, scored with
Parakeet WER, VoiceCLAP blend/genuineness and Empathic-Insight-Plus emotion heads:
laion/moss-local-voice-acting-64x100
— browse the best takes in the audio grid.

Links

• Model: laion/moss-tts-local-transformer-4.55b-voice-acting
• 8B sibling: laion/moss-tts-v1.5-8b-voice-acting
• Upstream architecture & tooling: OpenMOSS/MOSS-TTS · OpenMOSS/sglang
• Serving on HPC: sglang-omni-jupiter/ — SGLang-Omni on JUPITER (aarch64 / GH200), runbook + benchmark · upstream sgl-project/sglang-omni
• Audio tokenizer: OpenMOSS-Team/MOSS-Audio-Tokenizer-v2

License

Code: Apache-2.0. Documentation and sample pages: CC-BY-4.0.
Model weights: Apache-2.0 (see the model card).