Gemma 4 31B IT — NVFP4

Serve Google Gemma 4 31B IT in 4-bit NVFP4 quantization via vLLM with an OpenAI-compatible endpoint, Multi-Token Prediction speculative decoding, tool calling, and thinking/reasoning support.

Automatically downloads and caches both models from Hugging Face — just set your token and run.

Models

The draft model runs 4 lightweight decoder layers with Q-only attention, sharing KV cache with the target. It produces up to 4 draft tokens per step for speculative decoding — accelerating generation without sacrificing output quality.

Quick Start

Prerequisites

• Docker with NVIDIA Container Toolkit (nvidia-ctk completed)
• NVIDIA GPU (tested on a DGX Spark — GB10 GPU with 128 GB unified memory)
• Hugging Face token with access to both gated models:
  • https://huggingface.co/nvidia/Gemma-4-31B-IT-NVFP4
  • https://huggingface.co/google/gemma-4-31B-it-assistant

Start the server

HF_TOKEN=hf_your_token_here ./start.sh

The script downloads both models on first run (cached in ~/.cache/huggingface), launches the vLLM container, and waits for it to be ready.

Use the API

curl http://localhost:8888/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/Gemma-4-31B-IT-NVFP4",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Stop the server

./stop.sh

Features

🧠 Thinking / Reasoning

Enabled by default. The model outputs its reasoning in a <channel>thought block before the final response:

<|turn|>model
<|channel|>thought
The user is asking about...
<channel|>Here's the answer...
<turn|>

Disable per-request by passing chat_template_kwargs: {"enable_thinking": false} in your API call.

🛠️ Tool Calling

Native tool calling via the gemma4 parser. Everything is pre-configured — just pass tools in your request:

curl http://localhost:8888/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/Gemma-4-31B-IT-NVFP4",
    "messages": [{"role": "user", "content": "What'\''s the weather in Paris?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": { "type": "string", "description": "City name" }
          },
          "required": ["location"]
        }
      }
    }]
  }'

🖼️ Multi-modal (Image + Video)

The model supports images and video through a shared vision encoder. Audio has a token ID defined but the NVFP4 quantized weights do not include an audio encoder (audio_config: null).

Image example:

{
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}},
        {"type": "text", "text": "What's in this image?"}
      ]
    }
  ]
}

Video example:

{
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}},
        {"type": "text", "text": "Summarize this video"}
      ]
    }
  ]
}

Limits: up to 4 images or 1 video (or 1 audio if supported) per prompt.

⚡ MTP Speculative Decoding

Multi-Token Prediction generates 4 draft tokens per step using the lightweight assistant model. Most effective for:

• Batched workloads
• Long-context generation
• System-prompt-heavy applications (prefix caching also active)

Recommended Client Settings

As recommended by Google for Gemma 4:

Pass per-request or set as defaults in your client.

Concurrency & Performance

Tested on an NVIDIA DGX Spark (GB10 GPU, 128 GB unified memory).

Limits

KV cache at 0.70 gpu-memory-utilization

Realistic throughput

Tuning for higher concurrency

The --gpu-memory-utilization of 0.70 leaves headroom. You can increase concurrency by raising it — more KV cache blocks fit, which allows more concurrent sequences at the cost of less free GPU memory:

# In start.sh, for more concurrency:
--gpu-memory-utilization 0.95
--max-num-seqs 4
--max-num-batched-tokens 16384

Monitor with nvidia-smi to ensure you don't OOM.

Configuration

Environment variables

Key vLLM flags

Per-request overrides

Clients can override the template kwargs per-request:

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "preserve_thinking": false
  }
}

Files

.
├── start.sh               # Launch vLLM container (auto-downloads models)
├── stop.sh                # Gracefully stop the container
├── chat_template.jinja    # Gemma 4 canonical chat template (390 lines)
├── README.md              # This file
├── .gitignore             # Ignores runtime artifacts
├── .vllm.log              # Container logs (git-ignored)
├── .vllm.pid              # Container PID (git-ignored)
└── .cache/                # Triton cache (git-ignored)

Python Client Examples

OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8888/v1",
    api_key="not-needed",
)

response = client.chat.completions.create(
    model="nvidia/Gemma-4-31B-IT-NVFP4",
    messages=[{"role": "user", "content": "Write a quick sort in Python"}],
    temperature=0.2,
)

print(response.choices[0].message.content)

With tool calling

tools = [
    {
        "type": "function",
        "function": {
            "name": "run_code",
            "description": "Execute Python code and return stdout",
            "parameters": {
                "type": "object",
                "properties": {
                    "code": {"type": "string"}
                },
                "required": ["code"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="nvidia/Gemma-4-31B-IT-NVFP4",
    messages=[{"role": "user", "content": "Calculate 42 * 37"}],
    tools=tools,
)

Troubleshooting

License

The model weights are governed by the Gemma License. This repository's scripts and configuration are provided under the MIT License.

References

• Gemma 4 technical report
• NVIDIA ModelOpt NVFP4
• vLLM documentation
• Hugging Face: nvidia/Gemma-4-31B-IT-NVFP4