feat(llm-inference): Phase 7 — Prometheus monitoring + Grafana dashboard

- Phase 7 task file: monitoring.yml
  - node_exporter (port 9100) via apt, systemd managed
  - nvidia_gpu_exporter v1.2.2 (port 9835) — GPU util, VRAM, temp, power
  - Patches kube-prometheus additionalScrapeConfigs secret with 3 new jobs:
    node-astro-orbiter, gpu-astro-orbiter, vllm-astro-orbiter
  - Deploys Grafana dashboard ConfigMap via kubectl apply

- Grafana dashboard (11 panels):
  - Row 1: GPU util %, VRAM used, GPU temp gauge
  - Row 2: GPU power draw, vLLM token throughput, request queue depth
  - Row 3: vLLM e2e latency p50/p95/p99, KV cache utilization %
  - Row 4: System CPU %, memory, root disk gauge

- defaults/main.yml: llm_gpu_exporter_version, llm_gpu_exporter_port
- handlers/main.yml: restart nvidia-gpu-exporter
This commit is contained in:
Hermes Agent service account
2026-08-03 11:53:36 -05:00
parent dda6b91330
commit 423891001c
5 changed files with 504 additions and 0 deletions

View File

@@ -22,3 +22,7 @@ llm_serve_port: 8000
llm_serve_host: "0.0.0.0"
llm_gpu_memory_utilization: "0.90"
llm_max_model_len: 8192
# Monitoring
llm_gpu_exporter_version: "1.2.2"
llm_gpu_exporter_port: 9835