Platform Metrics

• Triton metrics such as request rate, failures, queue depth, and latency
• vLLM metrics such as scheduler state, KV cache pressure, and token throughput
• GPU metrics from the DCGM exporter
• Node metrics from node_exporter

​

Before you enable them

observability:
  monitoring: true

​

Enable platform metrics

observability:
  monitoring: true
  platformMetrics:
    enabled: true
    generationIntervalMs: 2000
    gpu:
      enabled: true
    inference:
      counterLatencies: true
      histogramLatencies: true
      summaryLatencies: true
      summaryQuantiles: ""

​

Configuration options

​

What you get

​

Triton

• nv_inference_request_success
• nv_inference_request_failure
• nv_inference_pending_request_count
• nv_inference_request_duration_us
• nv_inference_compute_infer_duration_us

​

vLLM

• vllm_llms_v1:*
• vllm_embeddings_v1:*

• vllm_llms_v1:num_requests_running
• vllm_llms_v1:kv_cache_usage_perc
• vllm_llms_v1:time_to_first_token_seconds_bucket
• vllm_llms_v1:generation_tokens_total

​

GPU

• nv_gpu_utilization
• nv_gpu_memory_used_bytes
• nv_gpu_power_usage

​

Node

• node_cpu_seconds_total
• node_memory_MemAvailable_bytes
• node_filesystem_avail_bytes

​

Next step

• inspect them in the in-cluster Grafana stack
• forward them to your own backend through Metrics Destinations
• use the reference External Grafana Dashboard