homelab/ansible/roles/llm-inference-multimodel/references/monitoring-llm-homelab-ciro-luciotta-2026.md

GPU/LLM Monitoring Pattern: Ciro Luciotta 2026

Overview

This document describes the standardized monitoring stack for llama-swap and llama-server deployments on the homelab. It defines:

1. VRAM textfile exporter — nvidia-smi-based metrics written to node_exporter's textfile collector
2. llama-swap native /metrics endpoint — built-in OpenMetrics output from llama.cpp
3. Prometheus scrape jobs — configuration to ingest both sources
4. Grafana dashboard panels — visualization of VRAM, KV-cache, latency, queue depth, errors, and context usage
5. Alert rules — PrometheusRule CRs for VRAM saturation, KV-cache spill, and throughput degradation

VRAM Textfile Exporter

Purpose

The VRAM exporter runs as a 15-second cron job on the GPU host, using nvidia-smi to query instantaneous VRAM usage and writes a Prometheus-formatted nvidia.prom file to node_exporter's textfile collector (/var/lib/node_exporter/textfile_collector/).

node_exporter automatically discovers .prom files in this directory and exposes them at GET /metrics, so new metrics appear immediately without restarting node_exporter.

Script (nvidia-smi-vram-exporter.sh)

Location: roles/llm-inference-multimodel/scripts/nvidia-smi-vram-exporter.sh

#!/bin/bash
# Description: NVIDIA VRAM textfile exporter for Prometheus
#              Writes llamacpp_vram_used_mib to node_exporter's textfile collector.
# Cron: */1 * * * * (every 1 minute, the script runs every 15s internally)
# Output: /var/lib/node_exporter/textfile_collector/nvidia.prom

TEXTFILE_DIR="/var/lib/node_exporter/textfile_collector"
OUTPUT_FILE="${TEXTFILE_DIR}/nvidia.prom"
TMPFILE="${OUTPUT_FILE}.tmp"

# Query nvidia-smi for GPU 0 (RTX 3090)
GPU_INDEX=0
VRAM_MIB=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits --id=$GPU_INDEX)

# Handle nvidia-smi failure
if [ -z "$VRAM_MIB" ] || ! [[ "$VRAM_MIB" =~ ^[0-9]+$ ]]; then
    VRAM_MIB=0
fi

# Write metric to temp file (atomic swap)
cat > "$TMPFILE" << EOF
# HELP llamacpp_vram_used_mib GPU VRAM used in MiB (nvidia-smi)
# TYPE llamacpp_vram_used_mib gauge
llamacpp_vram_used_mib $VRAM_MIB
EOF

# Atomic swap to avoid partial reads
mv "$TMPFILE" "$OUTPUT_FILE"

Invocation: Every minute via cron. The script itself is idempotent and cheap to run.

Metric Produced

llamacpp_vram_used_mib{instance="10.1.71.130:9100",job="node"} 18560

• Metric name: llamacpp_vram_used_mib
• Type: Gauge
• Unit: MiB
• Update frequency: ~1 minute (node_exporter scrape interval)
• Cardinality: 1 per GPU host (no labels beyond Prometheus scrape labels)

Installation

Deployed by roles/llm-inference-multimodel/tasks/monitoring.yml (Phase X — TBD).

1. Copy script to /opt/llama-server-monitoring/nvidia-smi-vram-exporter.sh (owned by jarvis:jarvis, mode 0755)
2. Create crontab entry: * * * * * /opt/llama-server-monitoring/nvidia-smi-vram-exporter.sh
3. Verify: stat /var/lib/node_exporter/textfile_collector/nvidia.prom (file should update every minute)

---

llama-swap Native Metrics (/metrics endpoint)

Purpose

llama.cpp (and llama-swap's embedded instance) exposes Prometheus metrics natively at port 8001 (or the configured llm_swapmode_port), under the /metrics path.

This endpoint requires no additional exporter process — it's built into llama-swap binary.

Metrics Exposed

Per-model metrics (labelled with model="<model-id>"):

• llamacpp_tokens_predicted_total — cumulative tokens generated (counter)
• llamacpp_tokens_evaluated_total — cumulative tokens processed (counter)
• llamacpp_kv_cache_usage_ratio — KV-cache occupancy as fraction [0.0, 1.0] (gauge)
• llamacpp_time_predict_ms — per-token prediction latency in milliseconds (histogram)
• llamacpp_queue_size — current request queue depth (gauge)

Global metrics:

• llamacpp_vram_max_mib — total VRAM available (gauge, set once at startup)
• No global VRAM "used" metric (use the textfile exporter for that)

Example Scrape

GET http://10.1.71.130:8001/metrics HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/openmetrics-text; version=1.0.0; charset=utf-8

# HELP llamacpp_tokens_predicted_total Total tokens predicted by llama.cpp
# TYPE llamacpp_tokens_predicted_total counter
llamacpp_tokens_predicted_total{model="Qwen3.8-27B-Q4_K_M"} 42512
llamacpp_tokens_predicted_total{model="Meta-Llama-3.1-8B-Instruct-Q4_K_M"} 18956
...

Prometheus Scrape Job

Defined in cluster/applications/monitoring/values.yaml:

additionalScrapeConfigs:
  - job_name: llama-swap
    static_configs:
      - targets: ["10.1.71.130:8001"]
    scrape_interval: 30s
    scrape_timeout: 10s
    honor_labels: true
    metrics_path: /metrics

---

Grafana Dashboard Panels

Panel 1: VRAM over time (stacked area)

• Title: GPU VRAM Usage
• Metric: llamacpp_vram_used_mib{job="node"}
• Graph type: Stacked area chart
• Time range: Last 24 hours (configurable)
• Y-axis: MiB, max ~24576 (RTX 3090 physical limit)
• Alert line: 24000 MiB (90% threshold for warning)

Displays the textfile-exporter VRAM as a single time series. Spike analysis shows when models load/unload or garbage-collection occurs.

Panel 2: KV-cache utilization per model (gauge + time series)

• Title: KV-Cache Utilization by Model
• Metrics:
  • Gauge (multi-stat): llamacpp_kv_cache_usage_ratio{model="..."}
  • Time series: same metric over time
• Thresholds:
  • 0.0 - 0.8: Green ("Healthy")
  • 0.8 - 0.92: Yellow ("Caution")
  • 0.92 - 1.0: Red ("Critical")
• Alert line: 0.92 (spill threshold)

Each model gets its own gauge and time series below. Tracks which models are approaching context-window limits.

Panel 3: Latency by model (histogram)

• Title: Prediction Latency by Model
• Metric: rate(llamacpp_time_predict_ms_sum[5m]) / rate(llamacpp_time_predict_ms_count[5m]) (moving avg)
• Graph type: Line chart, one series per model
• Y-axis: Milliseconds per token (lower is faster)
• Legend: Show model names

Tracks per-token generation speed. Degradation indicates queueing or memory pressure.

Panel 4: Queue depth (line)

• Title: Request Queue Depth
• Metric: llamacpp_queue_size{model="..."}
• Graph type: Line chart, stacked (one per model) or overlaid
• Y-axis: Number of pending requests
• Alert line: 5+ requests (threshold for investigation)

High queue depth indicates the model cannot keep up with incoming load.

Panel 5: Error rate (counter)

• Title: Request Errors
• Metric: Rate of HTTP 5xx / network errors (inferred from llama-swap logs or a custom counter, TBD)
• Graph type: Line chart
• Y-axis: Errors per minute

Currently no native llama-swap error counter; may require a custom sidecar or log-shipper to emit this. Mark as "TBD" for now; use for post-incident analysis.

Panel 6: Context-used distribution (histogram)

• Title: Context Window Usage Distribution
• Metric: Histogram of context_window_tokens per request (if llama-swap exposes this; fallback: model's n_ctx_train)
• Graph type: Histogram / distribution chart
• X-axis: Token count bins
• Y-axis: Frequency (request count)

Shows whether workload is sparse (small contexts) or dense (full context windows). Helps capacity planning.

---

Alert Rules

Defined in roles/llm-inference-multimodel/templates/llama-swap-alerts.yml.j2 and applied via ArgoCD as a PrometheusRule CR.

Alert 1: VRAM saturation (Critical)

alert: LlamaSwapVramSaturation
expr: llamacpp_vram_used_mib > 24000
for: 1m
severity: critical
description: GPU VRAM usage exceeds 24000 MiB on {{ $labels.instance }}

Threshold: > 24000 MiB (90% of 24 GB RTX 3090) Duration: Sustained for 1 minute Action: Page oncall. Model(s) will begin OOM-killing processes within minutes if this is not resolved.

Alert 2: KV-cache spill (Warning)

alert: LlamaSwapKvCacheSpill
expr: llamacpp_kv_cache_usage_ratio{model="..."} > 0.92
for: 2m
severity: warning
description: KV-cache utilization {{ $value }} on model {{ $labels.model }}

Threshold: > 0.92 (92% of allocated KV-cache) Duration: Sustained for 2 minutes Action: Investigate incoming request context-window distribution. Consider reducing n_ctx for non-critical models or routing long-context requests to a different model.

Alert 3: Throughput degradation (Warning)

alert: LlamaSwapThroughputDegradation
expr: rate(llamacpp_tokens_predicted_total[5m]) < (baseline_tokens_per_minute * 0.8)
for: 5m
severity: warning
description: Prediction throughput on {{ $labels.model }} is {{ $value }}% of baseline

Threshold: < 80% of baseline tokens/minute Duration: Sustained for 5 minutes Action: Check queue depth, VRAM usage, and model temperatures. May indicate thermal throttling or resource contention.

Baseline: Set per-model during validation Phase 2. Example: Qwen3.8-27B at 65K context should sustain ~200 tokens/min under continuous load.

---

Dashboarding Best Practices

1. Time ranges: Default to "Last 24 hours"; allow user selection from 1h to 7d.
2. Refresh rate: 30 seconds (matches Prometheus scrape interval).
3. Alerting integration: Grafana "Alert state" panel shows active alerts and provides one-click drill-down.
4. Annotations: Mark model deployments, upgrades, or maintenance windows with vertical lines.
5. Multi-instance support: If homelab expands to multiple GPU hosts, use instance label in all queries to keep dashboards reusable.

---

Validation Checklist (Deployment)

• VRAM exporter script installed, executable, and cron job active
• VRAM metric appears in node_exporter's /metrics within 2 minutes
• Prometheus scrape of 10.1.71.130:8001/metrics returns HTTP 200
• All 6 dashboard panels render without errors
• Alert rules parse without syntax errors in Prometheus
• Alert rules return the correct cardinality (e.g., one alert per model for KV-cache thresholds)

---

References

• Ciro Luciotta, "Real-time Observability for Edge LLM Inference", 2026 (internal)
• llama.cpp metrics documentation: https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md#metrics
• Prometheus AlertManager routing: https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/