Nemotron-Labs-3-Puzzle-75B on DGX Spark

Serve NVIDIA Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 on a single NVIDIA DGX Spark (GB10) node using vLLM 0.24 inside Docker. The launcher applies the aarch64 / SM12.1 workarounds needed for stable NVFP4 MoE + Mamba inference on Blackwell Spark hardware.

Why this repo

This is a minimal, Spark-tuned Docker recipe. start.sh handles:

• Resolving weights from the local Hugging Face cache (offline serving)
• Materializing symlinked custom model code (.py, .json, .jinja) so trust_remote_code loads correctly
• aarch64 NCCL and FlashInfer patches inside the container
• CUDA 13 / sm_121a environment variables for GB10
• Tool calling (qwen3_coder) and reasoning output (nemotron_v3)

A related layout with native start.sh / stop.sh, vendored ./model weights, and additional tuning options is maintained separately as Nemotron-Labs-3-Puzzle-75B (same Spark tuning lineage).

Requirements

Hardware

• 1× NVIDIA DGX Spark (GB10) — ~121 GB unified memory
• ~75 GB free disk — ~50 GB model weights + ~21 GB Docker image + cache headroom

Software

Verify GPU access:

docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi

Quick start

1. Download the model

pip install -U huggingface_hub
hf download nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4

Weights land under ~/.cache/huggingface/hub/models--nvidia--NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4/.

2. Pull the vLLM image

docker pull vllm/vllm-openai:v0.24.0

3. Clone and launch

git clone https://github.com/MiaAI-Lab/Nemotron-Labs-3-Puzzle-75B-DGX-Spark.git
cd Nemotron-Labs-3-Puzzle-75B-DGX-Spark
chmod +x start.sh
./start.sh

start.sh stops any prior containers, launches vLLM in Docker, then streams container logs until the API responds on /v1/models. When the server is ready, log following stops and you get your shell back.

Warmup can take several minutes on first boot while FlashInfer kernels compile.

4. Stop

docker stop nemo-75b-vllm
docker rm nemo-75b-vllm

Re-running ./start.sh performs the same cleanup automatically.

Configuration

All knobs are environment variables. Override them inline when calling start.sh:

SEQS=4 MAXLEN=131072 MOE_BACKEND=FLASHINFER_CUTLASS ./start.sh

Fixed in start.sh (edit the script to change):

Context and concurrency tuning

API examples

Chat completion

curl http://localhost:8888/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Nemotron-75b-Puzzle",
    "messages": [{"role": "user", "content": "Explain hybrid MoE in one paragraph."}],
    "max_tokens": 512
  }'

Python client

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8888/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="Nemotron-75b-Puzzle",
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=256,
)
print(response.choices[0].message.content)

For coding agents, NVIDIA recommends passing extra_body={"chat_template_kwargs": {"force_nonempty_content": True}} on chat requests.

Harness settings (Pi agent, thinking on)

Recommended settings for ~/.pi/agent/models.json, matching the served model name from start.sh (Nemotron-75b-Puzzle).

Add under a provider pointing at this server (for example "vLLM Local" with baseUrl: "http://localhost:8888/v1"):

{
  "id": "Nemotron-75b-Puzzle",
  "name": "Nemotron Labs 3 Puzzle 75B A9B NVFP4",
  "reasoning": true,
  "input": ["text"],
  "contextWindow": 262144,
  "maxTokens": 32768,
  "params": {
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 20,
    "min_p": 0,
    "presence_penalty": 0,
    "repetition_penalty": 1
  },
  "compat": {
    "supportsDeveloperRole": false,
    "requiresReasoningContentOnAssistantMessages": false,
    "thinkingFormat": "nemotron_v3"
  }
}

Summary:

For coding agents over the raw API, also pass extra_body={"chat_template_kwargs": {"force_nonempty_content": true}} so tool-use turns keep non-empty assistant content.

Troubleshooting

What start.sh does

┌─────────────────────────────────────────────────────────────┐
│ 1. Resolve HF cache snapshot for Puzzle-75B NVFP4           │
│ 2. Materialize symlinked model code files in snapshot         │
│ 3. Remove prior containers (nemo-75b-vllm, puzzle_a4q, etc.)  │
│ 4. docker run vllm/vllm-openai:v0.24.0                      │
│    ├─ NCCL lib symlink (aarch64)                              │
│    ├─ FlashInfer all-reduce + cuda_ipc patches              │
│    └─ vllm serve with MoE, Mamba, MTP, tool/reasoning APIs    │
└─────────────────────────────────────────────────────────────┘

Project layout

.
├── start.sh    # Docker launcher for DGX Spark
└── README.md

References

• Model card — NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4
• Tech report — Compressing Hybrid MoE LLMs
• Nemotron 3 Super technical report
• vLLM

License

Model weights and use are governed by the OpenMDW License Agreement v1.1. See the model README for benchmarks, safety information, and full documentation.

Deployment scripts in this repository are provided as-is for reproducibility on DGX Spark hardware.