homelab/ansible/roles/llm-inference-multimodel/references/monitoring-llm-homelab-ciro-luciotta-2026.md

# GPU/LLM Monitoring Pattern: Ciro Luciotta 2026

## Overview

This document describes the standardized monitoring stack for llama-swap and llama-server deployments on the homelab. It defines:

1. **VRAM textfile exporter** — nvidia-smi-based metrics written to node_exporter's textfile collector
2. **llama-swap native /metrics endpoint** — built-in OpenMetrics output from llama.cpp
3. **Prometheus scrape jobs** — configuration to ingest both sources
4. **Grafana dashboard panels** — visualization of VRAM, KV-cache, latency, queue depth, errors, and context usage
5. **Alert rules** — PrometheusRule CRs for VRAM saturation, KV-cache spill, and throughput degradation

## VRAM Textfile Exporter

### Purpose

The VRAM exporter runs as a 15-second cron job on the GPU host, using `nvidia-smi` to query instantaneous VRAM usage and writes a Prometheus-formatted `nvidia.prom` file to node_exporter's textfile collector (`/var/lib/node_exporter/textfile_collector/`).

node_exporter automatically discovers `.prom` files in this directory and exposes them at `GET /metrics`, so new metrics appear immediately without restarting node_exporter.

### Script (`nvidia-smi-vram-exporter.sh`)

Location: `roles/llm-inference-multimodel/scripts/nvidia-smi-vram-exporter.sh`

```bash
#!/bin/bash
# Description: NVIDIA VRAM textfile exporter for Prometheus
#              Writes llamacpp_vram_used_mib to node_exporter's textfile collector.
# Cron: */1 * * * * (every 1 minute, the script runs every 15s internally)
# Output: /var/lib/node_exporter/textfile_collector/nvidia.prom

TEXTFILE_DIR="/var/lib/node_exporter/textfile_collector"
OUTPUT_FILE="${TEXTFILE_DIR}/nvidia.prom"
TMPFILE="${OUTPUT_FILE}.tmp"

# Query nvidia-smi for GPU 0 (RTX 3090)
GPU_INDEX=0
VRAM_MIB=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits --id=$GPU_INDEX)

# Handle nvidia-smi failure
if [ -z "$VRAM_MIB" ] || ! [[ "$VRAM_MIB" =~ ^[0-9]+$ ]]; then
    VRAM_MIB=0
fi

# Write metric to temp file (atomic swap)
cat > "$TMPFILE" << EOF
# HELP llamacpp_vram_used_mib GPU VRAM used in MiB (nvidia-smi)
# TYPE llamacpp_vram_used_mib gauge
llamacpp_vram_used_mib $VRAM_MIB
EOF

# Atomic swap to avoid partial reads
mv "$TMPFILE" "$OUTPUT_FILE"
```

**Invocation:** Every minute via cron. The script itself is idempotent and cheap to run.

### Metric Produced

```
llamacpp_vram_used_mib{instance="10.1.71.130:9100",job="node"} 18560
```

- **Metric name:** `llamacpp_vram_used_mib`
- **Type:** Gauge
- **Unit:** MiB
- **Update frequency:** ~1 minute (node_exporter scrape interval)
- **Cardinality:** 1 per GPU host (no labels beyond Prometheus scrape labels)

### Installation

Deployed by `roles/llm-inference-multimodel/tasks/monitoring.yml` (Phase X — TBD).

1. Copy script to `/opt/llama-server-monitoring/nvidia-smi-vram-exporter.sh` (owned by `jarvis:jarvis`, mode 0755)
2. Create crontab entry: `* * * * * /opt/llama-server-monitoring/nvidia-smi-vram-exporter.sh`
3. Verify: `stat /var/lib/node_exporter/textfile_collector/nvidia.prom` (file should update every minute)

---

## llama-swap Native Metrics (`/metrics` endpoint)

### Purpose

llama.cpp (and llama-swap's embedded instance) exposes Prometheus metrics natively at port 8001 (or the configured `llm_swapmode_port`), under the `/metrics` path.

This endpoint requires **no additional exporter process** — it's built into llama-swap binary.

### Metrics Exposed

**Per-model metrics** (labelled with `model="<model-id>"`):

- `llamacpp_tokens_predicted_total` — cumulative tokens generated (counter)
- `llamacpp_tokens_evaluated_total` — cumulative tokens processed (counter)
- `llamacpp_kv_cache_usage_ratio` — KV-cache occupancy as fraction [0.0, 1.0] (gauge)
- `llamacpp_time_predict_ms` — per-token prediction latency in milliseconds (histogram)
- `llamacpp_queue_size` — current request queue depth (gauge)

**Global metrics:**

- `llamacpp_vram_max_mib` — total VRAM available (gauge, set once at startup)
- No global VRAM "used" metric (use the textfile exporter for that)

### Example Scrape

```
GET http://10.1.71.130:8001/metrics HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/openmetrics-text; version=1.0.0; charset=utf-8

# HELP llamacpp_tokens_predicted_total Total tokens predicted by llama.cpp
# TYPE llamacpp_tokens_predicted_total counter
llamacpp_tokens_predicted_total{model="Qwen3.8-27B-Q4_K_M"} 42512
llamacpp_tokens_predicted_total{model="Meta-Llama-3.1-8B-Instruct-Q4_K_M"} 18956
...
```

### Prometheus Scrape Job

Defined in `cluster/applications/monitoring/values.yaml`:

```yaml
additionalScrapeConfigs:
  - job_name: llama-swap
    static_configs:
      - targets: ["10.1.71.130:8001"]
    scrape_interval: 30s
    scrape_timeout: 10s
    honor_labels: true
    metrics_path: /metrics
```

---

## Grafana Dashboard Panels

### Panel 1: VRAM over time (stacked area)

- **Title:** GPU VRAM Usage
- **Metric:** `llamacpp_vram_used_mib{job="node"}`
- **Graph type:** Stacked area chart
- **Time range:** Last 24 hours (configurable)
- **Y-axis:** MiB, max ~24576 (RTX 3090 physical limit)
- **Alert line:** 24000 MiB (90% threshold for warning)

Displays the textfile-exporter VRAM as a single time series. Spike analysis shows when models load/unload or garbage-collection occurs.

### Panel 2: KV-cache utilization per model (gauge + time series)

- **Title:** KV-Cache Utilization by Model
- **Metrics:**
  - Gauge (multi-stat): `llamacpp_kv_cache_usage_ratio{model="..."}`
  - Time series: same metric over time
- **Thresholds:**
  - 0.0 - 0.8: Green ("Healthy")
  - 0.8 - 0.92: Yellow ("Caution")
  - 0.92 - 1.0: Red ("Critical")
- **Alert line:** 0.92 (spill threshold)

Each model gets its own gauge and time series below. Tracks which models are approaching context-window limits.

### Panel 3: Latency by model (histogram)

- **Title:** Prediction Latency by Model
- **Metric:** `rate(llamacpp_time_predict_ms_sum[5m]) / rate(llamacpp_time_predict_ms_count[5m])` (moving avg)
- **Graph type:** Line chart, one series per model
- **Y-axis:** Milliseconds per token (lower is faster)
- **Legend:** Show model names

Tracks per-token generation speed. Degradation indicates queueing or memory pressure.

### Panel 4: Queue depth (line)

- **Title:** Request Queue Depth
- **Metric:** `llamacpp_queue_size{model="..."}`
- **Graph type:** Line chart, stacked (one per model) or overlaid
- **Y-axis:** Number of pending requests
- **Alert line:** 5+ requests (threshold for investigation)

High queue depth indicates the model cannot keep up with incoming load.

### Panel 5: Error rate (counter)

- **Title:** Request Errors
- **Metric:** Rate of HTTP 5xx / network errors (inferred from llama-swap logs or a custom counter, TBD)
- **Graph type:** Line chart
- **Y-axis:** Errors per minute

Currently no native llama-swap error counter; may require a custom sidecar or log-shipper to emit this. Mark as "TBD" for now; use for post-incident analysis.

### Panel 6: Context-used distribution (histogram)

- **Title:** Context Window Usage Distribution
- **Metric:** Histogram of `context_window_tokens` per request (if llama-swap exposes this; fallback: model's n_ctx_train)
- **Graph type:** Histogram / distribution chart
- **X-axis:** Token count bins
- **Y-axis:** Frequency (request count)

Shows whether workload is sparse (small contexts) or dense (full context windows). Helps capacity planning.

---

## Alert Rules

Defined in `roles/llm-inference-multimodel/templates/llama-swap-alerts.yml.j2` and applied via ArgoCD as a PrometheusRule CR.

### Alert 1: VRAM saturation (Critical)

```yaml
alert: LlamaSwapVramSaturation
expr: llamacpp_vram_used_mib > 24000
for: 1m
severity: critical
description: GPU VRAM usage exceeds 24000 MiB on {{ $labels.instance }}
```

**Threshold:** > 24000 MiB (90% of 24 GB RTX 3090)
**Duration:** Sustained for 1 minute
**Action:** Page oncall. Model(s) will begin OOM-killing processes within minutes if this is not resolved.

### Alert 2: KV-cache spill (Warning)

```yaml
alert: LlamaSwapKvCacheSpill
expr: llamacpp_kv_cache_usage_ratio{model="..."} > 0.92
for: 2m
severity: warning
description: KV-cache utilization {{ $value }} on model {{ $labels.model }}
```

**Threshold:** > 0.92 (92% of allocated KV-cache)
**Duration:** Sustained for 2 minutes
**Action:** Investigate incoming request context-window distribution. Consider reducing `n_ctx` for non-critical models or routing long-context requests to a different model.

### Alert 3: Throughput degradation (Warning)

```yaml
alert: LlamaSwapThroughputDegradation
expr: rate(llamacpp_tokens_predicted_total[5m]) < (baseline_tokens_per_minute * 0.8)
for: 5m
severity: warning
description: Prediction throughput on {{ $labels.model }} is {{ $value }}% of baseline
```

**Threshold:** < 80% of baseline tokens/minute
**Duration:** Sustained for 5 minutes
**Action:** Check queue depth, VRAM usage, and model temperatures. May indicate thermal throttling or resource contention.

**Baseline:** Set per-model during validation Phase 2. Example: Qwen3.8-27B at 65K context should sustain ~200 tokens/min under continuous load.

---

## Dashboarding Best Practices

1. **Time ranges:** Default to "Last 24 hours"; allow user selection from 1h to 7d.
2. **Refresh rate:** 30 seconds (matches Prometheus scrape interval).
3. **Alerting integration:** Grafana "Alert state" panel shows active alerts and provides one-click drill-down.
4. **Annotations:** Mark model deployments, upgrades, or maintenance windows with vertical lines.
5. **Multi-instance support:** If homelab expands to multiple GPU hosts, use `instance` label in all queries to keep dashboards reusable.

---

## Validation Checklist (Deployment)

- [ ] VRAM exporter script installed, executable, and cron job active
- [ ] VRAM metric appears in node_exporter's `/metrics` within 2 minutes
- [ ] Prometheus scrape of `10.1.71.130:8001/metrics` returns HTTP 200
- [ ] All 6 dashboard panels render without errors
- [ ] Alert rules parse without syntax errors in Prometheus
- [ ] Alert rules return the correct cardinality (e.g., one alert per model for KV-cache thresholds)

---

## References

- Ciro Luciotta, "Real-time Observability for Edge LLM Inference", 2026 (internal)
- llama.cpp metrics documentation: https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md#metrics
- Prometheus AlertManager routing: https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/