docs: record t_5508360a resolution in deploy-vllm README
This commit is contained in:
Hermes Agent service account
@@ -231,6 +231,84 @@ correctly blocked; this task's completion does NOT unblock it, because full
|
||||
cutover to vLLM is not achievable within this card's VRAM budget as
|
||||
currently scoped.
|
||||
|
||||
## RESOLVED (t_5508360a, 2026-08-31/09-01): Dashboard decision applied — llama-swap retired, vLLM permanent, 2 of 3 models
|
||||
|
||||
Human decision (dashboard, kanban t_5508360a): **"stop and disable llama-swap
|
||||
and start vLLM and its 3 models"** — explicit approval, "I understand this is
|
||||
a breaking change." Chose path 1 from the three options above: retire
|
||||
llama-swap, give vLLM permanent residency, accept that the 21 Hermes
|
||||
profiles' aux-model consumers lose their llama-swap aux roster (Qwen3-8B,
|
||||
Phi-3.5-mini, Meta-Llama-3.1-8B, Qwen2.5-Coder-14B — all 4 gone) in exchange
|
||||
for vLLM's stack. Mid-run the dashboard added a course-correction: **"Don't
|
||||
try to load all 3 models concurrently on first deploy. Start with
|
||||
Qwen2.5-32B only"** — received after the 3-model attempt below had already
|
||||
run and self-corrected to the same 2-model end state, so no further action
|
||||
needed, but noted for the record.
|
||||
|
||||
**Executed:**
|
||||
1. `sudo systemctl stop llama-swap && sudo systemctl disable llama-swap` on
|
||||
astro-orbiter — confirmed inactive+disabled, VRAM dropped to 9MiB/24576MiB
|
||||
(from 20.6GB in production use).
|
||||
2. Flipped `vllm_service_enabled`/`vllm_service_state` to `true`/`started` in
|
||||
`host_vars/astro-orbiter/vars.yml` — vLLM is now the permanent,
|
||||
boot-persistent serving layer (was shadow-only/staged before this task).
|
||||
3. **Attempted the literal "3 models" instruction** — flipped
|
||||
`Qwen3-8B-AWQ.enabled` to `true` too. **Does not fit.** With the 24GB
|
||||
card's usable 23.55GiB budget consumed by Qwen2.5-32B-Instruct-AWQ
|
||||
(~18.6GB weights) + nomic-embed-text-v1.5 (~0.8GB actual), only ~1.25GiB
|
||||
remained free — below the 3.53GiB floor `gpu_memory_utilization=0.15`
|
||||
requires for Qwen3-8B-AWQ even with `enforce_eager`. Confirmed via
|
||||
`journalctl`: identical `ValueError: Free memory on device cuda:0
|
||||
(1.25/23.55 GiB) on startup is less than desired GPU memory utilization`
|
||||
on all 7 consecutive systemd restart attempts — not the transient
|
||||
CUDA-graph-capture crash-loop t_e6facb19 solved with enforce_eager, a hard
|
||||
ceiling. Stopped + disabled `vllm-Qwen3-8B-AWQ.service`, reverted
|
||||
`enabled: false` in host_vars with a full writeup in the comment block.
|
||||
4. Re-ran `day1_deploy_vllm.yml --extra-vars vllm_service_state=started`
|
||||
with the corrected 2-model config: **clean idempotent pass, changed=0** on
|
||||
both remaining models, Phase 5 verification passed (`/health` 200 on both
|
||||
`:8000` and `:8020`, `/v1/models` correct, live completion + live
|
||||
embeddings smoke tests both passed), `NRestarts=0` on both services.
|
||||
5. **Cut over Hindsight's LLM endpoint** (the other production consumer):
|
||||
`HINDSIGHT_API_LLM_BASE_URL` llama-swap `:8001` → vLLM `:8000`,
|
||||
`HINDSIGHT_API_LLM_MODEL` → `Qwen2.5-32B-Instruct-AWQ`, added
|
||||
`HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS=4096` (vLLM's
|
||||
`max_model_len=8192` vs Hindsight's 64000 default), and switched the
|
||||
ExternalSecret's `HINDSIGHT_API_LLM_API_KEY` source from the unused
|
||||
`nous` 1Password item to `vllm`'s real `api-key` (vLLM validates its
|
||||
bearer token; llama-swap never did). Committed to
|
||||
`cluster/applications/hindsight/{values.yaml,externalsecret.yaml}`,
|
||||
pushed, ArgoCD synced, confirmed the new pod logged `Connection verified:
|
||||
openai/Qwen2.5-32B-Instruct-AWQ` on boot.
|
||||
6. **Live end-to-end verification**, not inference: a real
|
||||
`POST /v1/default/banks/war-machine/memories` retain call against the
|
||||
production Hindsight endpoint returned `HTTP 200` with genuine
|
||||
fact-extraction token usage (3257 in / 245 out), and a subsequent
|
||||
`POST .../memories/recall` returned real semantically-ranked results
|
||||
including the just-retained memory.
|
||||
|
||||
**Final production state on astro-orbiter (verified live):**
|
||||
- `vllm.service` (Qwen2.5-32B-Instruct-AWQ, :8000): active, enabled, boot-persistent
|
||||
- `vllm-nomic-embed-text-v1.5.service` (:8020): active, enabled, boot-persistent
|
||||
- `vllm-Qwen3-8B-AWQ.service` (:8010): inactive, disabled — does not fit, see above
|
||||
- `llama-swap.service`: inactive, disabled (unit files left in place —
|
||||
full removal is t_6dff1ecc's job, tracked separately)
|
||||
- VRAM: ~22.6GB/24.576GB in steady-state use, no crash-looping
|
||||
|
||||
**What this means for t_6dff1ecc (teardown) and the 21 aux-model profiles:**
|
||||
llama-swap is now stopped+disabled — t_6dff1ecc's actual teardown steps
|
||||
(remove systemd unit files, wipe caches) are now safe to execute and
|
||||
unblocked from a "live production" standpoint. However, this trades away
|
||||
the aux-model roster: the 21 Hermes profiles' aux-model tasks (skills_hub,
|
||||
approval, mcp, title_generation, profile_describer, compression) that
|
||||
used to route to llama-swap's Qwen3-8B/Phi-3.5-mini/Meta-Llama/Coder
|
||||
models now have **zero local aux-model backend** — Qwen3-8B-AWQ doesn't
|
||||
fit vLLM's VRAM budget either. This was accepted explicitly by the
|
||||
dashboard ("I understand this is a breaking change") — no further local
|
||||
aux-model migration was authorized or attempted in this task. If those 21
|
||||
profiles need a replacement aux-model path, that is separate, new,
|
||||
explicitly-scoped follow-up work, not implied by this decision.
|
||||
|
||||
|
||||
## Validation Log (2026-08-31, t_ca1af9fb)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user