CrisperWhisper.cpp

Native C++/CUDA inference for
Nyra Labs CrisperWhisper 2.0:
controllable speech-to-text with verbatim and intended output, word-preserving
long-audio continuation, and no Python dependency at runtime.

The project uses whisper.cpp and
ggml for optimized CPU/CUDA execution. Python is never needed at runtime and
is only needed when converting an original safetensors checkpoint yourself;
ready-to-run GGML models are available from the link above.

[!NOTE]
This is an unofficial community implementation. It is not affiliated with,
maintained by, or endorsed by Nyra Health or Nyra Labs.

[!IMPORTANT]
This repository's C++ code is MIT licensed, but that does not change the
model license. The CrisperWhisper 2.0 weights, checkpoints, parameters,
configuration files, tokenizers/lexica, and outputs generated by the
models are governed by the
Nyra Health Non-Commercial Research License.
They are limited to non-commercial use unless you obtain a separate
commercial license from Nyra. Pro models are commercially licensed and
gated; large_pro requires manual access approval.

At a glance

Downloads

GitHub Releases
contains the native runtime archives; model weights are downloaded separately
from Hugging Face.

The Linux CUDA ZIP bundles the CUDA 12.8 Runtime, cuBLAS, cuBLASLt, and
nvJitLink user-space libraries. It needs a compatible NVIDIA driver but does
not require the CUDA Toolkit, Docker, or virtualization. No model file is
duplicated inside a GitHub archive.

RTX 5090 benchmark

Measured locally with the large FP16 model and matching verbatim decoding
settings. Lower time and RTF are better.

“Native faster” is calculated as (Python time / native time - 1) × 100.

The 55-second input repeats the bundled samples/jfk.wav
from whisper.cpp; it is a reproducible timing workload, not a natural
long-form quality sample. All rows had exact normalized transcript agreement.
See BENCHMARKS.md for model-load times, RTF, methodology,
caveats, and raw measurements.

Quick start

Windows

# One-time model download and conversion.
.\scripts\setup-model.ps1

# Native CUDA build. Use -Cpu for a CPU-only build.
.\scripts\build.ps1

# Transcribe.
.\build\bin\crisper-whisper.exe `
  --model .\models\ggml-crisperwhisper-large-f16.bin `
  --file .\samples\jfk.wav `
  --mode verbatim `
  --language en

Windows source-build details are in CPP.md.

Linux

# One-time model download and conversion.
./scripts/setup-model.sh

# Auto-select CUDA when nvcc + an NVIDIA GPU are available.
./scripts/build.sh

# Transcribe.
./build/bin/crisper-whisper \
  --model ./models/ggml-crisperwhisper-large-f16.bin \
  --file ./samples/jfk.wav \
  --mode verbatim \
  --language en

Distribution packages, CUDA setup, manual CMake commands, and troubleshooting
are in BUILD_LINUX.md.

Common CLI examples

Verbatim output keeps fillers, repetitions, false starts, and vocal events:

crisper-whisper -m MODEL.bin -f meeting.wav --mode verbatim -l en

Intended output produces the cleaner version the speaker meant:

crisper-whisper -m MODEL.bin -f meeting.wav --mode intended -l en

Machine-readable output:

crisper-whisper -m MODEL.bin -f meeting.mp3 --json

Optional supervised word timestamps:

crisper-whisper -m MODEL.bin -f meeting.mp3 --word-timestamps --json

The normal path keeps GGML Flash Attention enabled. Word timestamps lazily
load a separate alignment context and run one additional teacher-forced
no-Flash pass because fused Flash Attention does not expose the
decoder-to-audio probability matrix. If the flag is omitted, there is no
alignment pass or timestamp-related model-memory cost.

Add audio-grounded disfluencies to a trusted clean transcript:

crisper-whisper -m MODEL.bin -f meeting.wav \
  --verbatimize "the intended transcript"

Show every option:

crisper-whisper --help

<details open>
<summary><strong>What CrisperWhisper 2.0 does</strong></summary>

CrisperWhisper makes transcript style an explicit prompt-controlled choice.
The same audio can be decoded as:

• Verbatim: what was actually said, including fillers, repetitions,
  stutters, corrections, cut-offs, and vocal events.
• Intended: the clean, readable text the speaker meant.

The model is multilingual across most languages supported by Whisper. The
native CLI accepts the usual Whisper language codes such as en, de, fr,
es, ja, and zh.

Nyra's original resources:

• CrisperWhisper repository
• CrisperWhisper 2.0 large model
• Research paper
• Release post
• Verbatim speech benchmark

</details>
<details>
<summary><strong>CLI reference</strong></summary>

Required arguments

Recognition

Long audio

Hardware and output

Example with long-audio tuning:

crisper-whisper \
  -m models/ggml-crisperwhisper-large-f16.bin \
  -f interview.flac \
  --mode verbatim \
  --language en \
  --chunk-seconds 30 \
  --stride-seconds 26 \
  --context-words 12 \
  --drop-words 2 \
  --json

</details>

<details>
<summary><strong>Model download and conversion</strong></summary>

Available CrisperWhisper 2.0 checkpoints

Ready-to-run F16 GGML conversions of the four public checkpoints are hosted
together at
drbaph/CrisperWhisper2.0-GGML.
Download the .bin you want into models/; you do not need Python or the
conversion step when using one of these files.

The standard small, medium, turbo, and large models remain subject to Nyra's
Non-Commercial Research License.
Model files are downloaded from Hugging Face during setup and are never
included in this GitHub repository.

The default setup scripts download
nyralabs/CrisperWhisper2.0_large and produce:

models/
  ggml-crisperwhisper-large-f16.bin

The Hugging Face download is approximately 3.1 GB. The converted F16 model is
approximately 2.9 GB.

Windows:

.\scripts\setup-model.ps1

Linux:

./scripts/setup-model.sh

Choose another open checkpoint:

.\scripts\setup-model.ps1 `
  -Model "nyralabs/CrisperWhisper2.0_medium" `
  -Output "models/ggml-crisperwhisper-medium-f16.bin"

./scripts/setup-model.sh \
  --model nyralabs/CrisperWhisper2.0_medium \
  --output models/ggml-crisperwhisper-medium-f16.bin

The converter streams safetensors directly and requires only NumPy and
huggingface_hub; it does not install PyTorch or Transformers.

</details>

<details>
<summary><strong>Why is the converted model a <code>.bin</code> file?</strong></summary>

ggml is the tensor format and runtime family, not a required filename
extension. Traditional whisper.cpp models use names such as
ggml-model-f16.bin: the ggml- prefix describes the serialization and
.bin identifies the binary container.

This project writes whisper.cpp's supported legacy GGML model container rather
than GGUF because that loader accepts the exact Whisper tensor layout and full
51,896-token CrisperWhisper vocabulary we need. The file begins with GGML's
format magic and structured hyperparameters, vocabulary, and named tensors; it
is not an unstructured weight dump.

GGUF could be added later if whisper.cpp's Whisper loader and conversion path
make it useful, but changing the extension alone would not convert the format
or improve inference.

</details>

<details>
<summary><strong>Windows build</strong></summary>

Known-good requirements:

CUDA:

.\scripts\build.ps1

CPU:

.\scripts\build.ps1 -Cpu

Backend and CPU tuning are independent. CUDA builds retain a CPU fallback
using the selected CPU profile.

# Self-contained CPU builds.
.\scripts\build.ps1 -Cpu -CpuProfile portable -BuildDirectory build-cpu-portable
.\scripts\build.ps1 -Cpu -CpuProfile balance  -BuildDirectory build-cpu-balance
.\scripts\build.ps1 -Cpu -CpuProfile fast     -BuildDirectory build-cpu-fast

# CUDA plus a balanced AVX2 CPU fallback.
.\scripts\build.ps1 -CpuProfile balance -BuildDirectory build-cuda

The wrapper loads the Visual Studio 2022 x64 compiler environment and uses
Visual Studio's bundled Ninja. This allows CMake to drive nvcc without
requiring CUDA's optional Visual Studio extension.

The default release target is 86;89;120, producing one binary for RTX 3090,
RTX 4090, and RTX 5090-class GPUs. Use -CudaArchitectures to make a narrower
custom build. CUDA 12.8
was the first toolkit with sm_120 compiler support; with an older toolkit,
target only the architecture it supports.

# Examples for a smaller GPU-specific build.
.\scripts\build.ps1 -CudaArchitectures "86"  # RTX 3090
.\scripts\build.ps1 -CudaArchitectures "89"  # RTX 4090
.\scripts\build.ps1 -CudaArchitectures "120" # RTX 5090 (CUDA 12.8+)

</details>

<details>
<summary><strong>Linux build</strong></summary>

CPU or automatic CUDA detection:

./scripts/build.sh

Force a backend:

./scripts/build.sh --cuda
./scripts/build.sh --cpu

Choose the CPU compatibility profile for either backend:

./scripts/build.sh --cpu  --cpu-profile portable --build-dir build-cpu-portable
./scripts/build.sh --cpu  --cpu-profile balance  --build-dir build-cpu-balance
./scripts/build.sh --cpu  --cpu-profile fast     --build-dir build-cpu-fast
./scripts/build.sh --cuda --cpu-profile balance  --build-dir build-cuda

Install the resulting CLI into the configured CMake prefix:

cmake --install build --config Release

See BUILD_LINUX.md for Ubuntu/Debian, Fedora, Arch,
CUDA-toolkit, compiler-compatibility, and troubleshooting instructions.

</details>

<details>
<summary><strong>Long-audio continuation</strong></summary>

Whisper-family encoders process at most 30 seconds at once. CrisperWhisper was
trained with a conditional-continuation prompt:

[mode tags] <ctx> confirmed words from the previous window <ectx>

The native runtime follows that design:

1. Decode a 30-second window.
2. Confirm its safe words, leaving a small boundary tail to be re-covered.
3. Advance 26 seconds, creating a 4-second overlap.
4. Put the last 12 confirmed words in the next decoder prompt.
5. Continue until the end of the recording.

This does not naively concatenate overlapping model output:

• With --word-timestamps, the 26-second stride is an ownership cutoff.
  Words beginning in a chunk's trailing overlap are emitted by the next
  window, not both windows.
• A word straddling the cutoff is deduplicated only when its normalized text
  and absolute time both match the preceding word. Repeated speech elsewhere
  is never removed merely because its text matches an earlier passage.
• Emitted word times are clamped and monotonized, so start <= end and a
  later word never begins before the previous emitted word ends.
• Without word timestamps, the fixed boundary tail remains as a fallback and
  a normalized text seam matcher removes exact repeated sequences.

The bundled 55-second repeated-JFK regression returns all five repetitions
(110 timed words across two chunks), with zero invalid timestamps, zero
overlapping adjacent word pairs, zero adjacent text/timestamp duplicates, and
no missing chunk-boundary passage.

</details>

<details>
<summary><strong>Why stock whisper.cpp conversion is not enough</strong></summary>

CrisperWhisper 2.0 uses the Whisper encoder/decoder architecture, but appends
32 vocabulary entries after Whisper's timestamp tokens:

• 15 vocal-event tokens
• 5 verbatim-strength tokens
• 5 intended-strength tokens
• verbatimize start/end markers
• continuation-context start/end markers
• hotword start/end markers

Stock whisper.cpp currently infers several control-token positions from total
vocabulary size. With these appended tokens, its high-level decoder would
select incorrect task/timestamp IDs.

This project fixes the compatibility boundary without maintaining a fork of
ggml:

1. The converter serializes the complete Hugging Face vocabulary.
2. The C++ runtime discovers control IDs from their actual token strings.
3. It uses whisper.cpp's low-level encoder/decoder API.
4. Mode material is placed before the standard Whisper decoder prefix, matching
   CrisperWhisper training.

</details>

<details>
<summary><strong>C++ API</strong></summary>

#include <crisperwhisper.h>
#include <iostream>

int main() {
    crisperwhisper::Model model(
        "models/ggml-crisperwhisper-large-f16.bin"
    );

    crisperwhisper::TranscriptionOptions options;
    options.mode = crisperwhisper::Mode::Verbatim;
    options.language = "en";

    const auto result = model.transcribe_file("meeting.wav", options);
    std::cout << result.text << '\n';
}

The public API is declared in
include/crisperwhisper.h. It accepts either an
audio file or caller-provided mono 16 kHz float PCM.

</details>

<details>
<summary><strong>Architecture and project layout</strong></summary>

Audio file
  -> native decode/resample to mono 16 kHz
    -> Whisper log-mel frontend
      -> ggml encoder on CPU/CUDA
        -> CrisperWhisper custom decoder prompt
          -> native greedy autoregressive decode
            -> continuation stitching
              -> text or JSON

include/crisperwhisper.h        Public C++ API
src/crisperwhisper.cpp          Prompting, inference, long audio
src/audio.cpp                   Native audio decode/resample
src/word_timing.cpp             Supervised cross-attention word alignment
src/main.cpp                    CLI
tools/convert_hf_to_ggml.py     One-time safetensors converter
scripts/build.ps1               Windows build wrapper
scripts/build.sh                Linux build wrapper
scripts/package-windows.ps1     Self-contained Windows release ZIPs
scripts/setup-model.ps1         Windows model setup
scripts/setup-model.sh          Linux model setup
tests/cpp/                      Native tests
.github/workflows/native.yml    Windows/Linux CPU build matrix
.github/workflows/release.yml   Tagged Windows/Linux CPU ZIP releases

whisper.cpp is fetched at configure time and pinned to a known commit for
reproducible model-format compatibility.

</details>

<details>
<summary><strong>Current parity and roadmap</strong></summary>

Implemented natively:

• Standard CrisperWhisper 2.0 inference
• Verbatim and intended prompt control
• Verbatimize prompt construction
• Hotword prompt construction
• Long-audio continuation
• Default token suppression
• F16 ggml conversion
• CUDA and CPU backends
• Native audio decode/resample
• JSON results and per-chunk metadata
• Opt-in supervised cross-attention word timestamps
• Timestamp-aware overlap ownership and seam deduplication

Still to port from Nyra's Python package:

• Forced alignment
• Dual verbatim/intended batching
• Speculative decoding with a smaller draft checkpoint
• Full looping-hallucination repair
• Coverage-gated temperature fallback
• Early-EOT recovery

The README does not claim feature parity for items in the second list.

</details>

<details>
<summary><strong>Verification</strong></summary>

Native unit tests:

ctest --test-dir build -C Release --output-on-failure

ctest --test-dir build --output-on-failure

The repository includes a six-job x86-64 CPU matrix covering portable,
balanced, and fast profiles on both Windows and ubuntu-22.04, plus
a native ubuntu-22.04-arm portable ARM64 job. ARM64 release builds download
the Turbo GGML model and run the 55-second, five-repetition timestamped JFK
regression before packaging. Linux release jobs reject binaries requiring
newer than GLIBC 2.35 and statically link the GNU C++ runtime. The ARM64 ZIP
has also been validated on an Orange Pi 5 Pro running Ubuntu 22.04. Windows
ARM64 and ARM CUDA/Jetson are not supported in v1.1.1. The Windows and Linux
CUDA ZIPs are built for compute capabilities 8.6, 8.9, and 12.0. Hosted CI
verifies compilation, unit tests, dependency bundling, ABI compatibility, and
CLI startup; runtime inference should also be tested on the intended GPU
because the hosted runners do not provide an NVIDIA GPU.

A functional model test should verify:

• Model vocabulary size is 51896.
• All custom mode/context/hotword markers load.
• Verbatim and intended prompts produce distinct output on the same audio.
• Short audio completes without timestamp/control tokens in output.
• Long audio returns ordered chunk metadata with continuation context.
• --json output parses as valid JSON.

</details>

<details>
<summary><strong>Troubleshooting</strong></summary>

model vocabulary is missing required CrisperWhisper token

The model was probably converted with the stock whisper.cpp converter.
Reconvert it with:

tools/convert_hf_to_ggml.py

Windows says No CUDA toolset found

Use scripts/build.ps1, which configures the supported Visual Studio compiler
environment and Ninja automatically. Confirm nvcc --version works.

nvidia-smi is not on PATH

Update/install the NVIDIA driver, or build CPU-only:

.\scripts\build.ps1 -Cpu

Linux cannot find nvcc

Install the CUDA toolkit or build CPU-only:

./scripts/build.sh --cpu

The first model setup is slow

The default checkpoint download is multi-gigabyte. Conversion then reads BF16
safetensors and writes a new F16 ggml model. Later launches use that converted
file directly.

Commercial deployment

The MIT code license does not override the checkpoint license. Contact Nyra
for commercial model licensing.

</details>

Licensing and credits

• Native port and repository inference code: MIT
• whisper.cpp and ggml: MIT
• Miniaudio and stb_vorbis: included through whisper.cpp under their upstream
  licenses
• CrisperWhisper 2.0 standard models and their outputs:
  Nyra Health Non-Commercial Research License
• CrisperWhisper 2.0 Pro models: gated, commercial license required

Credits:

• Nyra Health / Nyra Labs for
  CrisperWhisper and the open checkpoints
• ggml-org for whisper.cpp/ggml
• 0xShug0/audio.cpp as a reference for
  structuring native multi-platform audio inference projects