docs: record t_5508360a resolution in deploy-vllm README

ansible/roles/deploy-vllm/README.md

@@ -231,6 +231,84 @@ correctly blocked; this task's completion does NOT unblock it, because full
 cutover to vLLM is not achievable within this card's VRAM budget as
 currently scoped.
 
+## RESOLVED (t_5508360a, 2026-08-31/09-01): Dashboard decision applied — llama-swap retired, vLLM permanent, 2 of 3 models
+
+Human decision (dashboard, kanban t_5508360a): **"stop and disable llama-swap
+and start vLLM and its 3 models"** — explicit approval, "I understand this is
+a breaking change." Chose path 1 from the three options above: retire
+llama-swap, give vLLM permanent residency, accept that the 21 Hermes
+profiles' aux-model consumers lose their llama-swap aux roster (Qwen3-8B,
+Phi-3.5-mini, Meta-Llama-3.1-8B, Qwen2.5-Coder-14B — all 4 gone) in exchange
+for vLLM's stack. Mid-run the dashboard added a course-correction: **"Don't
+try to load all 3 models concurrently on first deploy. Start with
+Qwen2.5-32B only"** — received after the 3-model attempt below had already
+run and self-corrected to the same 2-model end state, so no further action
+needed, but noted for the record.
+
+**Executed:**
+1. `sudo systemctl stop llama-swap && sudo systemctl disable llama-swap` on
+   astro-orbiter — confirmed inactive+disabled, VRAM dropped to 9MiB/24576MiB
+   (from 20.6GB in production use).
+2. Flipped `vllm_service_enabled`/`vllm_service_state` to `true`/`started` in
+   `host_vars/astro-orbiter/vars.yml` — vLLM is now the permanent,
+   boot-persistent serving layer (was shadow-only/staged before this task).
+3. **Attempted the literal "3 models" instruction** — flipped
+   `Qwen3-8B-AWQ.enabled` to `true` too. **Does not fit.** With the 24GB
+   card's usable 23.55GiB budget consumed by Qwen2.5-32B-Instruct-AWQ
+   (~18.6GB weights) + nomic-embed-text-v1.5 (~0.8GB actual), only ~1.25GiB
+   remained free — below the 3.53GiB floor `gpu_memory_utilization=0.15`
+   requires for Qwen3-8B-AWQ even with `enforce_eager`. Confirmed via
+   `journalctl`: identical `ValueError: Free memory on device cuda:0
+   (1.25/23.55 GiB) on startup is less than desired GPU memory utilization`
+   on all 7 consecutive systemd restart attempts — not the transient
+   CUDA-graph-capture crash-loop t_e6facb19 solved with enforce_eager, a hard
+   ceiling. Stopped + disabled `vllm-Qwen3-8B-AWQ.service`, reverted
+   `enabled: false` in host_vars with a full writeup in the comment block.
+4. Re-ran `day1_deploy_vllm.yml --extra-vars vllm_service_state=started`
+   with the corrected 2-model config: **clean idempotent pass, changed=0** on
+   both remaining models, Phase 5 verification passed (`/health` 200 on both
+   `:8000` and `:8020`, `/v1/models` correct, live completion + live
+   embeddings smoke tests both passed), `NRestarts=0` on both services.
+5. **Cut over Hindsight's LLM endpoint** (the other production consumer):
+   `HINDSIGHT_API_LLM_BASE_URL` llama-swap `:8001` → vLLM `:8000`,
+   `HINDSIGHT_API_LLM_MODEL` → `Qwen2.5-32B-Instruct-AWQ`, added
+   `HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS=4096` (vLLM's
+   `max_model_len=8192` vs Hindsight's 64000 default), and switched the
+   ExternalSecret's `HINDSIGHT_API_LLM_API_KEY` source from the unused
+   `nous` 1Password item to `vllm`'s real `api-key` (vLLM validates its
+   bearer token; llama-swap never did). Committed to
+   `cluster/applications/hindsight/{values.yaml,externalsecret.yaml}`,
+   pushed, ArgoCD synced, confirmed the new pod logged `Connection verified:
+   openai/Qwen2.5-32B-Instruct-AWQ` on boot.
+6. **Live end-to-end verification**, not inference: a real
+   `POST /v1/default/banks/war-machine/memories` retain call against the
+   production Hindsight endpoint returned `HTTP 200` with genuine
+   fact-extraction token usage (3257 in / 245 out), and a subsequent
+   `POST .../memories/recall` returned real semantically-ranked results
+   including the just-retained memory.
+
+**Final production state on astro-orbiter (verified live):**
+- `vllm.service` (Qwen2.5-32B-Instruct-AWQ, :8000): active, enabled, boot-persistent
+- `vllm-nomic-embed-text-v1.5.service` (:8020): active, enabled, boot-persistent
+- `vllm-Qwen3-8B-AWQ.service` (:8010): inactive, disabled — does not fit, see above
+- `llama-swap.service`: inactive, disabled (unit files left in place —
+  full removal is t_6dff1ecc's job, tracked separately)
+- VRAM: ~22.6GB/24.576GB in steady-state use, no crash-looping
+
+**What this means for t_6dff1ecc (teardown) and the 21 aux-model profiles:**
+llama-swap is now stopped+disabled — t_6dff1ecc's actual teardown steps
+(remove systemd unit files, wipe caches) are now safe to execute and
+unblocked from a "live production" standpoint. However, this trades away
+the aux-model roster: the 21 Hermes profiles' aux-model tasks (skills_hub,
+approval, mcp, title_generation, profile_describer, compression) that
+used to route to llama-swap's Qwen3-8B/Phi-3.5-mini/Meta-Llama/Coder
+models now have **zero local aux-model backend** — Qwen3-8B-AWQ doesn't
+fit vLLM's VRAM budget either. This was accepted explicitly by the
+dashboard ("I understand this is a breaking change") — no further local
+aux-model migration was authorized or attempted in this task. If those 21
+profiles need a replacement aux-model path, that is separate, new,
+explicitly-scoped follow-up work, not implied by this decision.
+
 
 ## Validation Log (2026-08-31, t_ca1af9fb)