NVRC - NVIDIA Runtime Container Init

A minimal init system (PID 1) for ephemeral NVIDIA GPU-enabled VMs running
under Kata Containers. NVRC sets up GPU drivers, configures hardware, spawns
NVIDIA management daemons, and hands off to kata-agent for container
orchestration.

Design Philosophy

Fail Fast, Fail Hard: NVRC is designed for ephemeral confidential VMs where
any configuration failure should immediately terminate the VM. There are no
recovery mechanisms—if GPU initialization fails, the VM powers off. This
"panic-on-failure" approach ensures:

• Security: No undefined states in confidential computing environments
• Simplicity: No complex error recovery logic to audit
• Clarity: If it's running, it's configured correctly

Architecture

┌────────────────────────────────────────────────────────────────┐
│                        NVRC (PID 1)                            │
│                                                                │
│  1. Set panic hook (power off VM on panic)                    │
│  2. Mount filesystems (/proc, /dev, /sys, /dev/shm)           │
│  3. Initialize kernel message logging                          │
│  4. Start syslog daemon                                        │
│  5. Parse kernel parameters (/proc/cmdline)                    │
│                                                                │
│  ┌──────────────────────────────────────────────────────────┐ │
│  │            Mode Selection (nvrc.mode)                    │ │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐     │ │
│  │  │GPU (default)│  │  CPU Mode   │  │NVSwitch-NVL4│ ... │ │
│  │  │• nvidia.ko  │  │• Skip GPU   │  │(H100/H200)  │     │ │
│  │  │• nvidia-uvm │  │             │  │• nvidia.ko  │     │ │
│  │  │• Lock clocks│  │             │  │• fabric-mgr │     │ │
│  │  │• Lock memory│  │             │  │• Health chk │     │ │
│  │  │• Power limit│  │             │  │             │     │ │
│  │  │• Daemons    │  │             │  │             │     │ │
│  │  │• CDI spec   │  │             │  │             │     │ │
│  │  │• SRS config │  │             │  │             │     │ │
│  │  │• Health chk │  │             │  │             │     │ │
│  │  └─────────────┘  └─────────────┘  └─────────────┘     │ │
│  └──────────────────────────────────────────────────────────┘ │
│                                                                │
│  6. Remount / as read-only (security hardening)                │
│  7. Disable kernel module loading (lockdown)                   │
│  8. Fork kata-agent (handoff control)                          │
│  9. Poll syslog forever (keep PID 1 alive)                     │
└────────────────────────────────────────────────────────────────┘

Kernel Parameters

NVRC is configured entirely via kernel command-line parameters (no config
files). This is critical for minimal init environments where userspace
configuration doesn't exist yet.

Core Parameters

GPU Configuration

Daemon Control

Example Configurations

Minimal GPU setup (defaults):

nvrc.mode=gpu

CPU-only mode:

nvrc.mode=cpu

NVSwitch NVL4 mode (Service VM for HGX H100/H200/H800 - NVLink 4.0):

nvrc.mode=nvswitch-nvl4

NVSwitch NVL5 mode (Service VM for HGX B200/B300/B100 - NVLink 5.0):

nvrc.mode=nvswitch-nvl5

GPU with locked clocks for benchmarking:

nvrc.mode=gpu nvrc.smi.lgc=1500 nvrc.smi.lmc=5001 nvrc.smi.pl=300

GPU with DCGM monitoring:

nvrc.mode=gpu nvrc.dcgm=on nvrc.log=info

Multi-GPU with NVLink:

nvrc.mode=gpu nvrc.fm.mode=0 nvrc.log=debug

Build

NVRC is compiled as a statically-linked musl binary for minimal dependencies:

# x86_64
cargo build --release --target x86_64-unknown-linux-musl

# aarch64
cargo build --release --target aarch64-unknown-linux-musl

Build configuration in .cargo/config.toml enables aggressive size
optimization and static linking.

Testing

# Unit tests (requires root for some tests)
cargo test

# Coverage (requires llvm-cov and root)
cargo llvm-cov --all-features --workspace

# Fuzzing
cargo +nightly fuzz run kernel_params
cargo +nightly fuzz run mount_parsing

# Static analysis
cargo clippy --all-features -- -D warnings
cargo audit
cargo deny check

Security Model

NVRC operates with a defense-in-depth security model appropriate for
confidential computing:

1. Minimal Attack Surface: 9 direct dependencies, statically linked
2. Fail-Fast: Panic hook powers off VM on any panic (no undefined states)
3. Read-Only Root: Filesystem becomes read-only after initialization
4. Module Lockdown: Kernel module loading disabled after GPU setup
5. OOM Protection: kata-agent protected with OOM score adjustment (-997)
6. Static Linking: No dynamic library dependencies to compromise
7. SLSA L3: Build provenance and Sigstore artifact signing

Why Panic Instead of Recover?

In traditional long-running systems, recovering from errors is valuable. In
ephemeral confidential VMs:

• VM lifetime is seconds/minutes: Restarting is faster than debugging
  partial failures
• Confidential computing requires integrity: Undefined states could leak
  secrets
• Orchestrator handles retries: Kubernetes/Kata will reschedule the pod
• Simpler audit surface: No complex recovery logic to verify

Troubleshooting

VM powers off immediately

Check kernel logs for panic messages. Common causes:

• Missing NVIDIA drivers in container image
• Invalid kernel parameters (check /proc/cmdline)
• Daemon startup failures (check logs with nvrc.log=debug)

GPU not available in container

• Verify nvrc.mode=gpu (default, but check explicitly)
• Check that GPU is passed through to VM
• Ensure nvidia kernel modules are present
• Verify CDI spec generation succeeded

DCGM/Fabric Manager not starting

• Enable debug logging: nvrc.log=debug
• Check that binaries exist in container image
• Verify configuration files are present (/etc/dcgm-exporter/, /usr/share/nvidia/nvswitch/)

Contributing

See CONTRIBUTING.md for DCO sign-off requirements.

Verification

See VERIFY.md for instructions on verifying release artifacts
with Sigstore.

License

Apache-2.0 - Copyright (c) NVIDIA CORPORATION