Promote Sign in Sign up

Repository

llm-d-workload-variant-autoscaler

Variant optimization autoscaler for distributed inference workloads

llm-d
Repository

README

Workload-Variant-Autoscaler (WVA)

Go Report Card
License
FOSSA Status

The Workload Variant Autoscaler (WVA) is a Kubernetes-based global autoscaler for inference model servers serving LLMs. WVA works alongside the standard Kubernetes HPA and external autoscalers like KEDA to drive the scale subresource of inference deployments. The high-level details of the algorithms are documented here. It determines optimal replica counts for a given request traffic load by considering constraints such as GPU availability, energy budget, and performance budget (latency/throughput).

What is a Variant?

WVA introduces the concept of variants — multiple model servers in an InferencePool that all serve the same base model but differ in hardware configuration (e.g., GPU type), serving configuration (e.g., tensor parallelism, max batch size, quantization), or both.

Use cases include:

  • P/D disaggregation: prefill is one variant, decode is another — variant = role in a disaggregated pipeline.
  • batch-gateway: variants distinguish batch vs. interactive workloads sharing the same pool.
  • Autoscaler: a costed serving configuration the autoscaler chooses among.

Key Features

  • Intelligent Autoscaling: Optimizes replica count by observing the current state of the system
  • Cost Optimization: Minimizes infrastructure costs by picking the correct accelerator variant

Documentation

See the architecture and autoscaling design docs for high-level algorithm details.

See the docs directory for design docs, developer guide, and more.

How It Works

Prerequisites: deploy llm-d infrastructure (model servers) and create an HPA or KEDA object targeting each deployment.

WVA then:

  1. Continuously monitors request rates and server performance via Prometheus metrics
  2. Capacity model obtains KV cache utilization and queue depth to determine desired replica counts
  3. Actuator emits optimization metrics to Prometheus
  4. External autoscaler (HPA/KEDA) reads the metrics and scales the deployment accordingly

Example

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llama-8b-autoscaler
  namespace: llm-inference
  annotations:
    llm-d.ai/managed: "true"  # Opt-in to WVA management
    llm-d.ai/variant-cost: "10.0"  # Optional, defaults to "10.0"
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llama-8b
  # minReplicas: 0  # scale to zero - alpha feature
  maxReplicas: 2
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Pods
        value: 10
        periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 60
      policies:
      - type: Pods
        value: 10
        periodSeconds: 15
  metrics:
  - type: External
    external:
      metric:
        name: wva_desired_replicas
        selector:
          matchLabels:
            variant_name: llama-8b
            exported_namespace: llm-inference
      target:
        type: AverageValue
        averageValue: "1"

More examples in config/samples/hpa/ and config/samples/keda/.

Upgrading

Upgrading to v0.9.0 — V2 saturation analyzer is now the default

Behavioral change. The default saturation analyzer changes from V1
(percentage/spare-capacity-based) to V2 (token/capacity-based). The shipped
default entry in the saturation ConfigMap now includes an analyzers: section,
which selects V2. No code change or image rebuild is involved — analyzer selection
is driven entirely by config.

V2 may produce different scaling decisions than V1 for the same workload. Review
your dashboards and alert thresholds after upgrading.

Staying on V1 (opt-out). Remove the analyzers: section (and the V2-only
scaleUpThreshold / scaleDownBoundary fields) from the default entry of your
saturation ConfigMap. The remaining kvCacheThreshold, queueLengthThreshold,
kvSpareTrigger, and queueSpareTrigger fields drive V1:

data:
  default: |
    kvCacheThreshold: 0.80
    queueLengthThreshold: 5
    kvSpareTrigger: 0.1
    queueSpareTrigger: 3

Apply with kubectl apply -f deploy/configmap-saturation-scaling.yaml; the change
takes effect immediately (the controller watches the ConfigMap).

V1 is deprecated and scheduled for removal in a future release. See the
saturation scaling configuration guide
for threshold ownership (which fields each analyzer reads) and migration details.

Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

Join the llm-d autoscaling community meetings to get involved.

License

Apache 2.0 - see LICENSE for details.

FOSSA Status

References