monitoring: llama-swap GPU/LLM stack (v250) — PrometheusRule, Grafana dashboard, scrape config, VRAM exporter

ansible/playbooks/day2_qwen38_ctx128k_rollback.yml

@@ -0,0 +1,65 @@
+---
+# ------------------------------------------------------------------------------
+# Playbook: day2_qwen38_ctx128k_rollback.yml
+# Purpose:  Roll back Qwen3.8-27B-Q4_K_M ctx-size from 131072 back to 65536
+#           on astro-orbiter's production router (port 8002).
+#
+# What this playbook does:
+#   1. Renders the updated llama-server-router-preset.ini.j2 (now with
+#      llm_router_qwen38_ctx_size: 65536) to
+#      /opt/llama-server-router-preset.ini.
+#   2. Restarts llama-server-router.service.
+#   3. Verifies the router loads Qwen3.8-27B at ctx=65536 in status.args.
+#
+# Context:
+#   - t_441470b9 (2026-08-16): ctx-size bumped 32768 -> 131072. Verified VRAM
+#     at 131072 ctx with only Qwen3.8 + nomic-embed co-resident: ~20,282 MiB
+#     + 558 MiB = ~20.8 GB on 24 GB RTX 3090. Comfortably safe.
+#   - t_72646029 (2026-08-17): Phi-3.5mini moved to GPU (n-gpu-layers=99)
+#     to enable concurrent residency with CPU-offloaded Coder-14B and
+#     Llama-3.1-8B. This added ~2GB CUDA context buffers for Phi + shifted
+#     Phi's model weights onto the GPU (~3.8GB).
+#   - NEW steady-state VRAM: Qwen3.8 @ 131072 ctx (~20,282 MiB) + nomic-embed
+#     (~558 MiB) + Llama CUDA ctx (~1,706 MiB) + Coder CUDA ctx (~1,390 MiB)
+#     = ~24,004 MiB. Adding Phi-3.5 (~3,800 MiB weights + ~1.4 GB CUDA ctx)
+#     pushes total to ~29,000+ MiB — exceeding the 24,576 MiB RTX 3090 limit.
+#     Qwen3.8-27B-131072 now fails to load (HTTP 500, OOM before llama.cpp
+#     reaches the model-loading phase).
+#   - FIX: reduce Qwen3.8 ctx-size 131072 -> 65536. This reduces KV cache
+#     from ~6GB to ~3GB, freeing ~3GB of VRAM. New estimated steady-state:
+#     Qwen3.8 @ 65536 ctx (~17,068 MiB) + nomic (~558) + Llama ctx (~1,706)
+#     + Coder ctx (~1,390) + Phi-3.5 (~3,800 + ~1,400 CUDA ctx) = ~25,922 MiB.
+#     Still over 24,576 — see "Phase 2" below for the secondary fix.
+#
+#   IMPORTANT: Rolling back ctx-size alone may NOT be sufficient. The
+#   hardware reference (astro-orbiter-hardware.md line 166, t_72646029)
+#   states steady-state ~24,004 MiB WITHOUT Phi on GPU. Adding Phi-3.5 back
+#   to GPU tips it over. This playbook handles the context rollback; if Qwen3.8
+#   still fails to load after Phase R, Wong should escalate to Ryan for a
+#   decision on either (a) offloading Phi-3.5mini to CPU (n-gpu-layers=0),
+#   or (b) adding a second GPU. Document the Phase 2 finding as a separate
+#   follow-up task if needed.
+#
+#   The 64K floor from the 2026-08-12 cutover validation (t_cd0d5388, Gate 1)
+#   still applies — ctx-size=65536 satisfies it.
+#
+# Run:
+#   cd /home/hermes/git/homelab/ansible
+#   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
+#     -i inventory.yml \
+#     playbooks/day2_qwen38_ctx128k_rollback.yml
+#
+# Task reference: t_c9fed26c — War Machine benchmark, 2026-08-18
+# Root cause: t_72646029 CPU-offload deployment added Phi-3.5 to GPU,
+#   shifting total VRAM past the 24,576 MiB ceiling when Qwen3.8 runs at 128K.
+# ------------------------------------------------------------------------------
+- name: Roll back Qwen3.8-27B ctx-size to 65536 on astro-orbiter
+  hosts: astro-orbiter
+  become: true
+  vars:
+    llm_router_preset_enabled: true
+    llm_router_qwen38_ctx_size: 65536
+
+  roles:
+    - role: llm-inference-multimodel
+      tags: [preset, systemd, verify]