Qwen3.6-35B-A3B-NVFP4 — DGX Spark

A vLLM deployment for Unsloth Qwen3.6-35B-A3B-NVFP4 on NVIDIA DGX Spark (GB10) — mixed FP8/NVFP4 MoE with FlashInfer CUTLASS NVFP4 experts, Triton FP8 experts, FlashInfer B12X linear GEMM, FP8 KV cache, and MTP speculative decoding.

Key Features

Prerequisites

Quick Start

1. Clone the Repository

git clone https://github.com/MiaAI-Lab/Unsloth-Qwen3.6-35b-NVFP4-DGX-Spark.git
cd Unsloth-Qwen3.6-35b-NVFP4-DGX-Spark

2. (Optional) Set HuggingFace Token

export HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

3. Start the Server

./start.sh

This will:

1. Check for Docker and curl on PATH
2. Create the HuggingFace cache directory
3. Remove any stale container with the same name
4. Launch the container with --gpus all, --shm-size=32g, --ulimit memlock=-1, --cap-add IPC_LOCK
5. Stream container logs to your terminal
6. Poll /v1/models until the server is ready
7. Print the OpenAI base URL

Expected output (first startup may take 10–20 minutes — torch.compile + warmup; subsequent restarts are ~10–20 seconds):

Model unsloth/Qwen3.6-35B-A3B-NVFP4 is already cached in ...
Starting vLLM container for unsloth/Qwen3.6-35B-A3B-NVFP4
Image: ghcr.io/miaai-lab/unsloth-qwen3.6-35b-nvfp4-fast-dgx-spark:latest
Listening on 0.0.0.0:8888
Spawning container Qwen35-35b-a3b-nvfp4 ...
Waiting for HTTP readiness at http://127.0.0.1:8888/v1/models
--- container logs ---
...
vLLM is ready!
OpenAI base URL: http://0.0.0.0:8888/v1

4. Test It

# Quick health check
curl http://0.0.0.0:8888/v1/models | jq

# Chat completion
curl -s http://0.0.0.0:8888/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "unsloth/Qwen3.6-35B-A3B-NVFP4",
    "messages": [{"role": "user", "content": "What is 19 × 23?"}],
    "temperature": 0,
    "max_tokens": 100
  }' | jq

5. Stop the Server

./stop.sh

Configuration

All options in start.sh. Key variables:

vLLM Flags

Recommended Client Configuration for Coding

The server exposes an OpenAI-compatible API. Configure your client with these settings:

Provider

{
  "baseUrl": "http://localhost:8888/v1",
  "api": "openai-completions",
  "apiKey": "dummy",
  "compat": {
    "supportsDeveloperRole": false,
    "supportsReasoningEffort": false,
    "maxTokensField": "max_tokens"
  }
}

Model Entry

{
  "id": "unsloth/Qwen3.6-35B-A3B-NVFP4",
  "name": "Unsloth Qwen3.6 35B A3B NVFP4",
  "reasoning": true,
  "input": ["text", "image"],
  "contextWindow": 262144,
  "maxTokens": 32000,
  "params": {
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 20,
    "min_p": 0,
    "presence_penalty": 0,
    "repetition_penalty": 1,
    "chat_template_kwargs": {
      "enable_thinking": true,
      "preserve_thinking": true
    }
  }
}

The model field in API requests must match the server's --served-model-name (defaults to the HuggingFace model ID).

Qwen's recommended parameters: For non-coding / general tasks (thinking mode), use temperature=1.0, top_p=0.95, top_k=20, min_p=0.0. The temperature=0.6 above is suited for precise coding tasks. See the Unsloth Qwen3.6 guide for details.

Docker Details

Performance Notes

• 80 % GPU memory allocated (--gpu-memory-utilization 0.80). Model loads ~22 GiB.
• 24 concurrent sequences (--max-num-seqs 24) based on measured KV cache capacity of ~5.9M FP8 tokens at 262K context.
• 32768 batched tokens (--max-num-batched-tokens 32768) for efficient batching.
• Speculative decoding (MTP, 2 tokens) improves decode speed; acceptance rate ~86 %.
• FlashInfer B12X linear kernel provides native SM121 NVFP4 execution.
• FP8 KV cache balances memory efficiency and accuracy.
• Decode speed at c1: ~80 tok/s on GB10. Higher concurrency increases total throughput.
• TTFT (Time to First Token): P50 103ms, P95 107ms at c1 with 200-token prompt, 50-token generation. Minimal variance.

Troubleshooting

License

• Model weights: See Unsloth model card for licensing
• This codebase: MIT License

Resources

• vLLM Documentation
• Qwen3.6 on HuggingFace
• FlashInfer
• OpenAI API Reference