diff --git a/ansible/roles/deploy-vllm/README.md b/ansible/roles/deploy-vllm/README.md index ae1efd1..a66d1b2 100644 --- a/ansible/roles/deploy-vllm/README.md +++ b/ansible/roles/deploy-vllm/README.md @@ -231,6 +231,84 @@ correctly blocked; this task's completion does NOT unblock it, because full cutover to vLLM is not achievable within this card's VRAM budget as currently scoped. +## RESOLVED (t_5508360a, 2026-08-31/09-01): Dashboard decision applied — llama-swap retired, vLLM permanent, 2 of 3 models + +Human decision (dashboard, kanban t_5508360a): **"stop and disable llama-swap +and start vLLM and its 3 models"** — explicit approval, "I understand this is +a breaking change." Chose path 1 from the three options above: retire +llama-swap, give vLLM permanent residency, accept that the 21 Hermes +profiles' aux-model consumers lose their llama-swap aux roster (Qwen3-8B, +Phi-3.5-mini, Meta-Llama-3.1-8B, Qwen2.5-Coder-14B — all 4 gone) in exchange +for vLLM's stack. Mid-run the dashboard added a course-correction: **"Don't +try to load all 3 models concurrently on first deploy. Start with +Qwen2.5-32B only"** — received after the 3-model attempt below had already +run and self-corrected to the same 2-model end state, so no further action +needed, but noted for the record. + +**Executed:** +1. `sudo systemctl stop llama-swap && sudo systemctl disable llama-swap` on + astro-orbiter — confirmed inactive+disabled, VRAM dropped to 9MiB/24576MiB + (from 20.6GB in production use). +2. Flipped `vllm_service_enabled`/`vllm_service_state` to `true`/`started` in + `host_vars/astro-orbiter/vars.yml` — vLLM is now the permanent, + boot-persistent serving layer (was shadow-only/staged before this task). +3. **Attempted the literal "3 models" instruction** — flipped + `Qwen3-8B-AWQ.enabled` to `true` too. **Does not fit.** With the 24GB + card's usable 23.55GiB budget consumed by Qwen2.5-32B-Instruct-AWQ + (~18.6GB weights) + nomic-embed-text-v1.5 (~0.8GB actual), only ~1.25GiB + remained free — below the 3.53GiB floor `gpu_memory_utilization=0.15` + requires for Qwen3-8B-AWQ even with `enforce_eager`. Confirmed via + `journalctl`: identical `ValueError: Free memory on device cuda:0 + (1.25/23.55 GiB) on startup is less than desired GPU memory utilization` + on all 7 consecutive systemd restart attempts — not the transient + CUDA-graph-capture crash-loop t_e6facb19 solved with enforce_eager, a hard + ceiling. Stopped + disabled `vllm-Qwen3-8B-AWQ.service`, reverted + `enabled: false` in host_vars with a full writeup in the comment block. +4. Re-ran `day1_deploy_vllm.yml --extra-vars vllm_service_state=started` + with the corrected 2-model config: **clean idempotent pass, changed=0** on + both remaining models, Phase 5 verification passed (`/health` 200 on both + `:8000` and `:8020`, `/v1/models` correct, live completion + live + embeddings smoke tests both passed), `NRestarts=0` on both services. +5. **Cut over Hindsight's LLM endpoint** (the other production consumer): + `HINDSIGHT_API_LLM_BASE_URL` llama-swap `:8001` → vLLM `:8000`, + `HINDSIGHT_API_LLM_MODEL` → `Qwen2.5-32B-Instruct-AWQ`, added + `HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS=4096` (vLLM's + `max_model_len=8192` vs Hindsight's 64000 default), and switched the + ExternalSecret's `HINDSIGHT_API_LLM_API_KEY` source from the unused + `nous` 1Password item to `vllm`'s real `api-key` (vLLM validates its + bearer token; llama-swap never did). Committed to + `cluster/applications/hindsight/{values.yaml,externalsecret.yaml}`, + pushed, ArgoCD synced, confirmed the new pod logged `Connection verified: + openai/Qwen2.5-32B-Instruct-AWQ` on boot. +6. **Live end-to-end verification**, not inference: a real + `POST /v1/default/banks/war-machine/memories` retain call against the + production Hindsight endpoint returned `HTTP 200` with genuine + fact-extraction token usage (3257 in / 245 out), and a subsequent + `POST .../memories/recall` returned real semantically-ranked results + including the just-retained memory. + +**Final production state on astro-orbiter (verified live):** +- `vllm.service` (Qwen2.5-32B-Instruct-AWQ, :8000): active, enabled, boot-persistent +- `vllm-nomic-embed-text-v1.5.service` (:8020): active, enabled, boot-persistent +- `vllm-Qwen3-8B-AWQ.service` (:8010): inactive, disabled — does not fit, see above +- `llama-swap.service`: inactive, disabled (unit files left in place — + full removal is t_6dff1ecc's job, tracked separately) +- VRAM: ~22.6GB/24.576GB in steady-state use, no crash-looping + +**What this means for t_6dff1ecc (teardown) and the 21 aux-model profiles:** +llama-swap is now stopped+disabled — t_6dff1ecc's actual teardown steps +(remove systemd unit files, wipe caches) are now safe to execute and +unblocked from a "live production" standpoint. However, this trades away +the aux-model roster: the 21 Hermes profiles' aux-model tasks (skills_hub, +approval, mcp, title_generation, profile_describer, compression) that +used to route to llama-swap's Qwen3-8B/Phi-3.5-mini/Meta-Llama/Coder +models now have **zero local aux-model backend** — Qwen3-8B-AWQ doesn't +fit vLLM's VRAM budget either. This was accepted explicitly by the +dashboard ("I understand this is a breaking change") — no further local +aux-model migration was authorized or attempted in this task. If those 21 +profiles need a replacement aux-model path, that is separate, new, +explicitly-scoped follow-up work, not implied by this decision. + ## Validation Log (2026-08-31, t_ca1af9fb)