DeepSeek V4 Flash – Dual DGX Spark (1M Context)

Deploy DeepSeek-V4-Flash — a Mixture-of-Experts (MoE) reasoning model — across two NVIDIA DGX Spark nodes with 1 million token context length, InfiniBand interconnect, and FP8 KV-cache.

Overview

This repository provides a ready-to-run docker-compose.yml and helper scripts that launch a vLLM inference server serving DeepSeek V4 Flash. The setup uses:

• 2 × DGX Spark nodes (spark1 = head, spark2 = worker)
• Tensor parallelism (TP) across both GPUs
• InfiniBand (NCCL over IB) for inter-node communication
• FP8 KV-cache (--kv-cache-dtype fp8) for memory efficiency
• Multi-Token Prediction (MTP) speculative decoding (2 draft tokens)
• Prefix caching and FlashInfer autotune
• Tool calling and reasoning support (DeepSeek V4 parsers)

Default generation config: The docker-compose.yml sets --override-generation-config '{"temperature":0.6,"top_p":0.95}' — tuned for coding tasks. Remove this line to use DeepSeek V4 Flash's default sampling parameters.

Requirements

Hardware

Software

• Docker with docker compose plugin (v2.24+)
• Passwordless SSH from spark1 → spark2
• NVIDIA Container Toolkit (nvidia-ctk) — installed by default on DGX Spark
• InfiniBand drivers (ibdev2netdev, ibstat) — installed by default
• Git and curl

Quick Start

1. Clone the repo on both nodes

git clone https://github.com/MiaAI-Lab/DeepSeek-V4-Flash-Dual-DGX-Spark-1M-Context.git
cd DeepSeek-V4-Flash-Dual-DGX-Spark-1M-Context

Run this on both spark1 (head) and spark2 (worker).

2. Download model weights on both nodes

Pre-download the model weights into the HuggingFace cache so the container doesn't download them on first run:

hf download deepseek-ai/DeepSeek-V4-Flash

Run this on both spark1 (head) and spark2 (worker). This requires the huggingface_hub CLI (pip install huggingface_hub).

3. Configure environment

cp .env.example .env

Edit .env to match your cluster:

Finding your InfiniBand interface:

ibdev2netdev -v

Pick the rocep* device connected to your IB link.

Finding your socket interface:

ip addr show | grep 169.254

Pick the interface with the link-local IP used for NCCL out-of-band communication.

4. Set NODE_RANK on each node

On spark1 (head):

NODE_RANK=0

On spark2 (worker):

NODE_RANK=1
HEADLESS=1

Update .env accordingly on each node.

5. Start the server

From spark1 only:

./start-deepseek-v4-flash.sh

This script:

1. SSHs into spark2 and starts the container there
2. Starts the container on spark1
3. Polls http://127.0.0.1:8000/v1/models until the API is ready (up to ~20 minutes)

6. Stop the server

From spark1 only:

./stop-deepseek-v4-flash.sh

API Usage

Once running, the OpenAI-compatible vLLM API is available on both nodes at http://localhost:8000.

Chat completion (with reasoning)

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v4-flash",
    "messages": [
      {"role": "user", "content": "What is 47 × 89?"}
    ],
    "temperature": 0.0
  }'

Streaming

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v4-flash",
    "messages": [
      {"role": "user", "content": "Write a haiku about distributed inference"}
    ],
    "stream": true
  }'

Tool calling

The server is configured with --tool-call-parser deepseek_v4 and --enable-auto-tool-choice for function-calling tasks.

Architecture

┌─────────────────────────┐       InfiniBand        ┌─────────────────────────┐
│     spark1 (head)       │◄───────────────────────►│    spark2 (worker)      │
│  NODE_RANK=0            │   NCCL (IB / sockets)    │  NODE_RANK=1            │
│  ┌───────────────────┐  │                          │  ┌───────────────────┐  │
│  │ vLLM container    │  │                          │  │ vLLM container    │  │
│  │ TP rank 0 (GPU 0) │  │                          │  │ TP rank 1 (GPU 1) │  │
│  │ Port 8000 (API)   │  │                          │  │ Port 8000 (API)   │  │
│  └───────────────────┘  │                          │  └───────────────────┘  │
└─────────────────────────┘                          └─────────────────────────┘

Key vLLM parameters

Docker Images

The compose file uses aidendle94/sparkrun-vllm-ds4-gb10:production-ready — a custom vLLM build optimized for DGX Spark (GB10, CUDA arch 12.1a, FlashInfer for Hopper).

If you need to rebuild or use a different image, update the image: field in docker-compose.yml.

Files

Troubleshooting

NCCL / InfiniBand

• Verify IB link state: ibstat (should show LinkUp: true)
• Test passwordless SSH: ssh <WORKER_HOST> hostname
• Check NCCL debug logs (set NCCL_DEBUG=INFO in compose env)

Container fails to start

docker compose logs

CUDA out of memory

Reduce --gpu-memory-utilization (e.g., 0.75) or --max-model-len.

Timeout waiting for API

The model loads from HuggingFace on first run. Check logs:

docker logs deepseek-v4-flash-vllm-1

License

This repository's code is provided under the MIT License. The DeepSeek V4 Flash model weights are subject to DeepSeek's license.