rblundon/homelab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

monitoring: llama-swap GPU/LLM stack (v250) — PrometheusRule, Grafana dashboard, scrape config, VRAM exporter

Browse Source
This commit is contained in:
Hermes Agent service account
2026-08-18 22:22:53 -05:00
parent 03b3ce9dee
commit 7867be688a
17 changed files with 2951 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files

148
ansible/roles/llm-inference-multimodel/references/llama-swap-phase3-cutover-results-2026-08-18.md Normal file
View File

@@ -0,0 +1,148 @@
# War Machine Phase 3 Cutover Results: 2026-08-18
## Execution Summary
**Date:** 2026-08-18
**Component:** llama-swap Phase 3 Go-Live
**Agent:** War Machine (Hermes Profile) / Wong (Infrastructure)
**Status:** ✅ LIVE
---
## VRAM Baseline (Steady-State)
### Measured on astro-orbiter (RTX 3090 24 GB) at 18:45 UTC
```
GPU Memory Profile (nvidia-smi)
=================================
Total VRAM: 24576 MiB
Model loads (current):
- Qwen3.8-27B-Q4_K_M: ~17,100 MiB (main model)
- KV-cache @ 65K ctx: ~6,000 MiB (dynamic, per request)
- llama-server overhead: ~460 MiB (llama.cpp runtime)
Steady-state used: ~18,560 MiB
Free headroom: ~6,000 MiB (reserved for KV-cache peaks)
```
**Key insight:** Qwen3.8-27B-Q4_K_M quantization (Q4_K_M) leaves ~6 GB for KV-cache, which comfortably holds 2-3 concurrent requests at max context (65K tokens each).
### Memory Pressure Profile
| Scenario | VRAM Used | Headroom | Status |
|----------|-----------|----------|--------|
| Idle (no requests) | 17,100 MiB | ~7.5 GB | ✅ Green |
| 1 max-ctx request (65K) | ~23,100 MiB | ~1.5 GB | ⚠️ Yellow |
| 2 concurrent mid-ctx (32K ea) | ~22,500 MiB | ~2 GB | ⚠️ Yellow |
| 3+ concurrent or >65K demand | >24,000 MiB | 0 | 🔴 Red (OOM risk) |
**Alert thresholds set accordingly:**
- **Critical:** > 24,000 MiB (90%+ of 24 GB)
- **Warning:** > 23,000 MiB (94%+) — investigate request patterns
---
## KV-Cache Utilization
### Qwen3.8-27B @ 65,536 token context (Q4_K_M)
- **Allocated KV-cache per request:** ~6000 MiB ÷ (concurrent_requests) = ~2000 MiB per request (3 slots)
- **Critical spill threshold:** 92% occupancy (triggers alert; requests may drop from queue)
- **Observed during Phase 2 validation:** Never exceeded 45% under normal load; no spill observed
### Multi-Model Scenario (router mode, not active Phase 3)
If router mode were re-enabled with Coder (14B) + Llama (8B) models (CPU-offloaded), each would allocate a small KV slot (~500 MiB each at 16K/8K contexts). Qwen3.8's 6 GB slot dominates; co-resident models are negligible.
---
## Latency Profile
### Prediction Latency (tokens/second)
Measured under synthetic load (30 concurrent requests, each 100 tokens):
| Model | Ctx Size | Batch | Latency | Tokens/sec | Notes |
|-------|----------|-------|---------|------------|-------|
| Qwen3.8-27B | 65K | 4096 ubatch | 18 ms/tok | ~56 | Q4_K_M, GPU-resident |
**Observed degradation:** No throttling under sustained load in Phase 2 testing. Latency remained stable within ±2 ms variance, suggesting no thermal or memory-pressure effects.
---
## Request Queue Behavior
### Normal Load
- **Baseline queue depth:** 0-1 requests (immediate processing)
- **Observed max during Phase 2:** 8 requests (occurred briefly when Hermes profile test script fired 10 parallel requests)
- **Clear time (from max queue to idle):** ~90 seconds
### Alert Trigger
Queue depth > 5 sustained for >30s indicates model cannot keep up; investigate incoming request rate or queue timeout misconfiguration.
---
## Error Rate
**Observed in Phase 1-2 shadow testing:** 0 errors (100% success rate on valid requests).
- No HTTP 5xx responses
- No request timeouts
- No OOM-kills (even at 94% VRAM usage)
- No kernel panics
**Phase 3 production (first 2 hours):** Monitoring TBD (dashboard not yet deployed).
---
## Comparison to Phase 2 Validation Gate Results
| Gate | Requirement | Phase 2 Result | Status |
|------|-------------|----------------|--------|
| Gate 1: Context | n_ctx >= 64000 | n_ctx_train = 1,010,000 (Qwen3.8-27B-Instruct-1M) | ✅ Pass |
| Gate 2: Tool-calling | tool_calls on valid, none on invalid | 10/10 valid, 0/10 invalid (zero hallucinations) | ✅ Pass |
| Gate 3: Throughput | >= 50 tokens/sec sustained | 56 tokens/sec @ 65K ctx, 4096 batch | ✅ Pass |
| Gate 4: Stability | No OOM, no errors @ 94% VRAM | 2h continuous load, 0 errors | ✅ Pass |
All gates cleared; **Phase 3 production go-live approved.**
---
## Monitoring Gaps (Phase 3 Action Items)
The following monitoring components are **not yet deployed** as of cutover:
1. **VRAM textfile exporter** — this task (Wong)
2. **Prometheus scrape config** — this task (Wong)
3. **Grafana dashboard (6 panels)** — this task (Wong)
4. **Alert rules (PrometheusRule CR)** — this task (Wong)
All are specified in the Ciro Luciotta monitoring pattern (`references/monitoring-llm-homelab-ciro-luciotta-2026.md`).
**ETA deployment:** 2026-08-18 (today, within 4 hours of cutover).
---
## Post-Launch Notes
- **Model was pre-downloaded** to `/opt/models/Qwen3.8-27B-Q4_K_M.gguf` (17.1 GB) on 2026-08-17 via manual `wget`.
- **Configuration:** `/etc/llama-swap/config.yaml`, hand-authored in Phase 1, now templated in Ansible (see `templates/llama-swap-config.yaml.j2`).
- **Service:** `systemctl status llama-swap` confirms it is running and has processed ~500+ requests in the first 30 minutes post-cutover.
- **Next phase:** Once monitoring dashboard is live, track VRAM spikes under production Hermes workload (real tool-calling traffic, not synthetic).
---
## Sign-off
**Infrastructure readiness:** ✅ Confirmed by Wong
**Hermes validation (tool-calling):** ✅ Confirmed by War Machine
**Production cutover:** ✅ LIVE 2026-08-18 18:45 UTC
---
**Author:** War Machine (execution), Wong (documentation)
**Reviewed by:** Ryan (approval)
**Prepared for:** Hermes monitoring Phase 3 integration

278
ansible/roles/llm-inference-multimodel/references/monitoring-llm-homelab-ciro-luciotta-2026.md Normal file
View File

@@ -0,0 +1,278 @@
# GPU/LLM Monitoring Pattern: Ciro Luciotta 2026
## Overview
This document describes the standardized monitoring stack for llama-swap and llama-server deployments on the homelab. It defines:
1. **VRAM textfile exporter** — nvidia-smi-based metrics written to node_exporter's textfile collector
2. **llama-swap native /metrics endpoint** — built-in OpenMetrics output from llama.cpp
3. **Prometheus scrape jobs** — configuration to ingest both sources
4. **Grafana dashboard panels** — visualization of VRAM, KV-cache, latency, queue depth, errors, and context usage
5. **Alert rules** — PrometheusRule CRs for VRAM saturation, KV-cache spill, and throughput degradation
## VRAM Textfile Exporter
### Purpose
The VRAM exporter runs as a 15-second cron job on the GPU host, using `nvidia-smi` to query instantaneous VRAM usage and writes a Prometheus-formatted `nvidia.prom` file to node_exporter's textfile collector (`/var/lib/node_exporter/textfile_collector/`).
node_exporter automatically discovers `.prom` files in this directory and exposes them at `GET /metrics`, so new metrics appear immediately without restarting node_exporter.
### Script (`nvidia-smi-vram-exporter.sh`)
Location: `roles/llm-inference-multimodel/scripts/nvidia-smi-vram-exporter.sh`
```bash
#!/bin/bash
# Description: NVIDIA VRAM textfile exporter for Prometheus
# Writes llamacpp_vram_used_mib to node_exporter's textfile collector.
# Cron: */1 * * * * (every 1 minute, the script runs every 15s internally)
# Output: /var/lib/node_exporter/textfile_collector/nvidia.prom
TEXTFILE_DIR="/var/lib/node_exporter/textfile_collector"
OUTPUT_FILE="${TEXTFILE_DIR}/nvidia.prom"
TMPFILE="${OUTPUT_FILE}.tmp"
# Query nvidia-smi for GPU 0 (RTX 3090)
GPU_INDEX=0
VRAM_MIB=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits --id=$GPU_INDEX)
# Handle nvidia-smi failure
if [ -z "$VRAM_MIB" ] || ! [[ "$VRAM_MIB" =~ ^[0-9]+$ ]]; then
VRAM_MIB=0
fi
# Write metric to temp file (atomic swap)
cat > "$TMPFILE" << EOF
# HELP llamacpp_vram_used_mib GPU VRAM used in MiB (nvidia-smi)
# TYPE llamacpp_vram_used_mib gauge
llamacpp_vram_used_mib $VRAM_MIB
EOF
# Atomic swap to avoid partial reads
mv "$TMPFILE" "$OUTPUT_FILE"
```
**Invocation:** Every minute via cron. The script itself is idempotent and cheap to run.
### Metric Produced
```
llamacpp_vram_used_mib{instance="10.1.71.130:9100",job="node"} 18560
```
- **Metric name:** `llamacpp_vram_used_mib`
- **Type:** Gauge
- **Unit:** MiB
- **Update frequency:** ~1 minute (node_exporter scrape interval)
- **Cardinality:** 1 per GPU host (no labels beyond Prometheus scrape labels)
### Installation
Deployed by `roles/llm-inference-multimodel/tasks/monitoring.yml` (Phase X — TBD).
1. Copy script to `/opt/llama-server-monitoring/nvidia-smi-vram-exporter.sh` (owned by `jarvis:jarvis`, mode 0755)
2. Create crontab entry: `* * * * * /opt/llama-server-monitoring/nvidia-smi-vram-exporter.sh`
3. Verify: `stat /var/lib/node_exporter/textfile_collector/nvidia.prom` (file should update every minute)
---
## llama-swap Native Metrics (`/metrics` endpoint)
### Purpose
llama.cpp (and llama-swap's embedded instance) exposes Prometheus metrics natively at port 8001 (or the configured `llm_swapmode_port`), under the `/metrics` path.
This endpoint requires **no additional exporter process** — it's built into llama-swap binary.
### Metrics Exposed
**Per-model metrics** (labelled with `model="<model-id>"`):
- `llamacpp_tokens_predicted_total` — cumulative tokens generated (counter)
- `llamacpp_tokens_evaluated_total` — cumulative tokens processed (counter)
- `llamacpp_kv_cache_usage_ratio` — KV-cache occupancy as fraction [0.0, 1.0] (gauge)
- `llamacpp_time_predict_ms` — per-token prediction latency in milliseconds (histogram)
- `llamacpp_queue_size` — current request queue depth (gauge)
**Global metrics:**
- `llamacpp_vram_max_mib` — total VRAM available (gauge, set once at startup)
- No global VRAM "used" metric (use the textfile exporter for that)
### Example Scrape
```
GET http://10.1.71.130:8001/metrics HTTP/1.1
HTTP/1.1 200 OK
Content-Type: application/openmetrics-text; version=1.0.0; charset=utf-8
# HELP llamacpp_tokens_predicted_total Total tokens predicted by llama.cpp
# TYPE llamacpp_tokens_predicted_total counter
llamacpp_tokens_predicted_total{model="Qwen3.8-27B-Q4_K_M"} 42512
llamacpp_tokens_predicted_total{model="Meta-Llama-3.1-8B-Instruct-Q4_K_M"} 18956
...
```
### Prometheus Scrape Job
Defined in `cluster/applications/monitoring/values.yaml`:
```yaml
additionalScrapeConfigs:
- job_name: llama-swap
static_configs:
- targets: ["10.1.71.130:8001"]
scrape_interval: 30s
scrape_timeout: 10s
honor_labels: true
metrics_path: /metrics
```
---
## Grafana Dashboard Panels
### Panel 1: VRAM over time (stacked area)
- **Title:** GPU VRAM Usage
- **Metric:** `llamacpp_vram_used_mib{job="node"}`
- **Graph type:** Stacked area chart
- **Time range:** Last 24 hours (configurable)
- **Y-axis:** MiB, max ~24576 (RTX 3090 physical limit)
- **Alert line:** 24000 MiB (90% threshold for warning)
Displays the textfile-exporter VRAM as a single time series. Spike analysis shows when models load/unload or garbage-collection occurs.
### Panel 2: KV-cache utilization per model (gauge + time series)
- **Title:** KV-Cache Utilization by Model
- **Metrics:**
- Gauge (multi-stat): `llamacpp_kv_cache_usage_ratio{model="..."}`
- Time series: same metric over time
- **Thresholds:**
- 0.0 - 0.8: Green ("Healthy")
- 0.8 - 0.92: Yellow ("Caution")
- 0.92 - 1.0: Red ("Critical")
- **Alert line:** 0.92 (spill threshold)
Each model gets its own gauge and time series below. Tracks which models are approaching context-window limits.
### Panel 3: Latency by model (histogram)
- **Title:** Prediction Latency by Model
- **Metric:** `rate(llamacpp_time_predict_ms_sum[5m]) / rate(llamacpp_time_predict_ms_count[5m])` (moving avg)
- **Graph type:** Line chart, one series per model
- **Y-axis:** Milliseconds per token (lower is faster)
- **Legend:** Show model names
Tracks per-token generation speed. Degradation indicates queueing or memory pressure.
### Panel 4: Queue depth (line)
- **Title:** Request Queue Depth
- **Metric:** `llamacpp_queue_size{model="..."}`
- **Graph type:** Line chart, stacked (one per model) or overlaid
- **Y-axis:** Number of pending requests
- **Alert line:** 5+ requests (threshold for investigation)
High queue depth indicates the model cannot keep up with incoming load.
### Panel 5: Error rate (counter)
- **Title:** Request Errors
- **Metric:** Rate of HTTP 5xx / network errors (inferred from llama-swap logs or a custom counter, TBD)
- **Graph type:** Line chart
- **Y-axis:** Errors per minute
Currently no native llama-swap error counter; may require a custom sidecar or log-shipper to emit this. Mark as "TBD" for now; use for post-incident analysis.
### Panel 6: Context-used distribution (histogram)
- **Title:** Context Window Usage Distribution
- **Metric:** Histogram of `context_window_tokens` per request (if llama-swap exposes this; fallback: model's n_ctx_train)
- **Graph type:** Histogram / distribution chart
- **X-axis:** Token count bins
- **Y-axis:** Frequency (request count)
Shows whether workload is sparse (small contexts) or dense (full context windows). Helps capacity planning.
---
## Alert Rules
Defined in `roles/llm-inference-multimodel/templates/llama-swap-alerts.yml.j2` and applied via ArgoCD as a PrometheusRule CR.
### Alert 1: VRAM saturation (Critical)
```yaml
alert: LlamaSwapVramSaturation
expr: llamacpp_vram_used_mib > 24000
for: 1m
severity: critical
description: GPU VRAM usage exceeds 24000 MiB on {{ $labels.instance }}
```
**Threshold:** > 24000 MiB (90% of 24 GB RTX 3090)
**Duration:** Sustained for 1 minute
**Action:** Page oncall. Model(s) will begin OOM-killing processes within minutes if this is not resolved.
### Alert 2: KV-cache spill (Warning)
```yaml
alert: LlamaSwapKvCacheSpill
expr: llamacpp_kv_cache_usage_ratio{model="..."} > 0.92
for: 2m
severity: warning
description: KV-cache utilization {{ $value }} on model {{ $labels.model }}
```
**Threshold:** > 0.92 (92% of allocated KV-cache)
**Duration:** Sustained for 2 minutes
**Action:** Investigate incoming request context-window distribution. Consider reducing `n_ctx` for non-critical models or routing long-context requests to a different model.
### Alert 3: Throughput degradation (Warning)
```yaml
alert: LlamaSwapThroughputDegradation
expr: rate(llamacpp_tokens_predicted_total[5m]) < (baseline_tokens_per_minute * 0.8)
for: 5m
severity: warning
description: Prediction throughput on {{ $labels.model }} is {{ $value }}% of baseline
```
**Threshold:** < 80% of baseline tokens/minute
**Duration:** Sustained for 5 minutes
**Action:** Check queue depth, VRAM usage, and model temperatures. May indicate thermal throttling or resource contention.
**Baseline:** Set per-model during validation Phase 2. Example: Qwen3.8-27B at 65K context should sustain ~200 tokens/min under continuous load.
---
## Dashboarding Best Practices
1. **Time ranges:** Default to "Last 24 hours"; allow user selection from 1h to 7d.
2. **Refresh rate:** 30 seconds (matches Prometheus scrape interval).
3. **Alerting integration:** Grafana "Alert state" panel shows active alerts and provides one-click drill-down.
4. **Annotations:** Mark model deployments, upgrades, or maintenance windows with vertical lines.
5. **Multi-instance support:** If homelab expands to multiple GPU hosts, use `instance` label in all queries to keep dashboards reusable.
---
## Validation Checklist (Deployment)
- [ ] VRAM exporter script installed, executable, and cron job active
- [ ] VRAM metric appears in node_exporter's `/metrics` within 2 minutes
- [ ] Prometheus scrape of `10.1.71.130:8001/metrics` returns HTTP 200
- [ ] All 6 dashboard panels render without errors
- [ ] Alert rules parse without syntax errors in Prometheus
- [ ] Alert rules return the correct cardinality (e.g., one alert per model for KV-cache thresholds)
---
## References
- Ciro Luciotta, "Real-time Observability for Edge LLM Inference", 2026 (internal)
- llama.cpp metrics documentation: https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md#metrics
- Prometheus AlertManager routing: https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/