From 1af645d27271a022ce4af100748ae5e668bdd12e Mon Sep 17 00:00:00 2001 From: Hermes Agent service account Date: Mon, 31 Aug 2026 18:36:04 -0500 Subject: [PATCH] REVERT: vLLM cannot be continuously resident alongside llama-swap (t_e6facb19) Critical finding: flipping vllm_service_enabled/state=true/started and restarting llama-swap alongside it broke llama-swap's ability to load ANY of its own generative models -- every /v1/chat/completions request against Qwen3.8-27B-Q4_K_M or Qwen3-8B aux models failed with 'upstream command exited prematurely' (llama-server OOM at spawn, ~1.8GB free on this 24GB card once vLLM's ~22.8GB was claimed). Confirmed by direct A/B: same request 500s with vLLM running, 200s seconds after stopping it. This breaks 21 Hermes agent profiles' aux-model tasks (skills_hub, approval, mcp, title_generation, profile_describer, compression) plus OpenViking's VLM -- a far larger blast radius than Hindsight's single LLM endpoint. Reverted: - vllm_service_enabled/state back to role defaults (false/stopped) -- vLLM stays staged, startable for a brief validated shadow window, NOT safe to leave resident in production. - Hindsight's HINDSIGHT_API_LLM_BASE_URL back to llama-swap (astro-orbiter:8001, Qwen3.8-27B-Q4_K_M) and the API key secret source back to the Nous fallback item (pre-task state) -- the vLLM cutover, while functionally validated in isolation (health, /v1/chat/completions, and a live hindsight_retain+recall round-trip all succeeded), requires continuous vLLM availability which is now known to be unsafe on this card. Comment posted on t_6dff1ecc: teardown remains correctly blocked -- full cutover is not achievable within this card's VRAM budget as currently scoped. Needs a human decision on aux-model migration strategy (see roles/deploy-vllm README's 'Critical architectural finding' section) before any further progress. --- ansible/host_vars/astro-orbiter/vars.yml | 23 ++++ ansible/roles/deploy-vllm/README.md | 128 ++++++++++++------ .../hindsight/externalsecret.yaml | 18 +-- cluster/applications/hindsight/values.yaml | 41 +++--- 4 files changed, 136 insertions(+), 74 deletions(-) diff --git a/ansible/host_vars/astro-orbiter/vars.yml b/ansible/host_vars/astro-orbiter/vars.yml index 1ff07a6..67f45f5 100644 --- a/ansible/host_vars/astro-orbiter/vars.yml +++ b/ansible/host_vars/astro-orbiter/vars.yml @@ -163,3 +163,26 @@ vllm_models: trust_remote_code: true enabled: true +# --- deploy-vllm role: boot persistence NOT enabled (t_e6facb19, 2026-08-31) --- +# ATTEMPTED enabling vllm_service_enabled/state=started here, then reverted +# after a production-breaking discovery: with vLLM's two processes +# (Qwen2.5-32B + nomic-embed, ~22.8GB combined) resident, llama-swap could no +# longer load ANY of its own generative models -- every /v1/chat/completions +# request against Qwen3.8-27B-Q4_K_M or the Qwen3-8B aux models failed with +# "upstream command exited prematurely" (llama-server's own OOM at spawn +# time, silently swallowed by llama-swap's generic error). Confirmed by +# direct A/B: same request 500s with vLLM running, 200s within seconds of +# `systemctl stop vllm.service vllm-nomic-embed-text-v1.5.service`. This +# breaks all 21 Hermes agent profiles' aux-model tasks (skills_hub, approval, +# mcp, title_generation, profile_describer, compression) plus OpenViking's +# VLM -- a severe regression, worse than the status quo. Left +# vllm_service_enabled/state at role defaults (false/stopped) -- vLLM stays +# staged and manually startable for a brief shadow window (same pattern as +# t_ca1af9fb's original validation), but is NOT safe to leave resident +# alongside llama-swap on this 24GB card. See README's "Known Gaps" section: +# full teardown of llama-swap (t_6dff1ecc) is the ONLY path to giving vLLM +# permanent residency without starving the other models -- this is not +# solvable by tuning gpu_memory_utilization further, the two stacks +# together need more VRAM than this card has once both hold real models +# resident. + diff --git a/ansible/roles/deploy-vllm/README.md b/ansible/roles/deploy-vllm/README.md index 656558a..ae1efd1 100644 --- a/ansible/roles/deploy-vllm/README.md +++ b/ansible/roles/deploy-vllm/README.md @@ -89,25 +89,12 @@ supports (`--quantization awq`) and which fit the VRAM budget: - `vllm_service_enabled` defaults to `false` deliberately — see "Deliberate staging-first default" above. Flip together with the cutover step, not before. -- **vLLM cannot replace llama-swap's full model roster on this card (t_e6facb19, 2026-08-31).** - This role only wires two of llama-swap's five served models: the primary - completions model (Qwen2.5-32B-Instruct-AWQ, replacing Qwen3.8-27B) and - the embedding model (nomic-embed-text-v1.5). llama-swap ALSO serves - Qwen3-8B-Q4_K_M(-no_think), Phi-3.5-mini-instruct-Q8_0, - Meta-Llama-3.1-8B-Instruct-Q4_K_M, and Qwen2.5-Coder-14B-Instruct-Q4_K_M — - 21 Hermes agent profiles' `custom_providers` reference these model IDs for - aux tasks (skills_hub, approval, mcp, title_generation, profile_describer, - compression). vLLM 0.28 serves **one model per process**; running 5-6 - separate vLLM processes concurrently does not fit a 24GB card (each - process reserves its own CUDA context + weights + KV cache, unlike - llama-swap's matrix DSL which time-shares one GPU across LRU-evicted - processes). **Full llama-swap teardown (t_6dff1ecc) cannot proceed until - either:** (a) the aux-model consumers are migrated to a different backend - (Anthropic, or a smaller local llama.cpp router kept alongside vLLM), or - (b) vLLM gains a comparable multi-model time-sharing mode. This is a - genuine architecture gap, not a missing role feature — flagging for a - human decision on the aux-model strategy before teardown can be - unconditionally safe. +- **vLLM cannot replace llama-swap's full model roster on this card — see + "Critical architectural finding" section below for the full incident.** + Short version: vLLM's one-model-per-process design plus llama-swap's own + VRAM needs exceed this 24GB card's capacity when both must serve real + models simultaneously. Full llama-swap teardown (t_6dff1ecc) cannot + proceed until a human decides the aux-model + VRAM strategy. ## Embedding-mode support (t_e6facb19, 2026-08-31) @@ -162,32 +149,87 @@ crash-looped repeatedly: ## Consumer cutover status (t_e6facb19, 2026-08-31) -**Cut over (validated end-to-end):** -- Hindsight (`cluster/applications/hindsight/values.yaml` + - `externalsecret.yaml`): `HINDSIGHT_API_LLM_BASE_URL` → vLLM `:8000` - (Qwen2.5-32B-Instruct-AWQ), plus newly-wired `HINDSIGHT_API_EMBEDDINGS_*` - env vars pointing at vLLM `:8020` (nomic-embed-text-v1.5). Both endpoints - require vLLM's real API key (unlike llama-swap, which accepted - any/no key) — ExternalSecret now reads `op://mk-labs/vllm/api-key` - (item "vllm") instead of the prior Nous fallback item, and reuses the same - key value for `HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY` (both vLLM - endpoints share one key file per `tasks/api-key.yml`). +**Attempted, then REVERTED — Hindsight LLM cutover.** Hindsight's +`HINDSIGHT_API_LLM_BASE_URL` was pointed at vLLM `:8000` +(Qwen2.5-32B-Instruct-AWQ) and validated working in isolation: health, +`/v1/chat/completions`, and a live `hindsight_retain` + recall round-trip +all succeeded (after also fixing `HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS`, +which defaulted to 64000 — exceeding vLLM's `max_model_len=8192` — down to +4096). **Reverted anyway**, because of a severe discovery documented in the +next section: vLLM cannot stay resident on this card without starving +llama-swap, and Hindsight's LLM endpoint needs continuous availability, not +just a validation window. Restored to `http://astro-orbiter:8001/v1` +(llama-swap, Qwen3.8-27B-Q4_K_M) — the pre-task working state. -**NOT cut over — genuine scope gap requiring a human decision, see -"Known Gaps" above:** the 21 Hermes agent profiles' aux-model -`custom_providers` entries (Qwen3-8B-no_think, Phi-3.5-mini, Meta-Llama-3.1-8B, -Qwen2.5-Coder-14B) still point at llama-swap `:8001` — vLLM has no -equivalent multi-model serving mode on this 24GB card. llama-swap MUST stay -up to serve these until that gap is resolved. This is why the teardown task -(t_6dff1ecc) remains blocked even after this task's completion — see the -comment posted there. +**NOT cut over — embeddings.** Hindsight was discovered to have NEVER used +astro-orbiter for embeddings — it defaults to a bundled local +`BAAI/bge-small-en-v1.5` (384-dim) embedder whenever +`HINDSIGHT_API_EMBEDDINGS_PROVIDER` is unset, which was always the case here. +Pointing it at vLLM's `nomic-embed-text-v1.5` (768-dim) crash-looped the pod: +`RuntimeError: Cannot change embedding dimension from 384 to 768: +memory_units table contains 1289 rows with embeddings.` Re-embedding all +existing memory data across ~20 agent banks is destructive and irreversible +— reverted immediately, left as a separate, explicitly-approved future task. -- OpenViking (`cluster/platform/openviking/values.yaml`): still points at - llama-swap `:8001` (`Meta-Llama-3.1-8B-Instruct-Q4_K_M` VLM + nomic-embed - for dense embeddings). Left unchanged — its VLM model has no vLLM - equivalent staged, and migrating only its embedding path while leaving its - VLM on llama-swap would still require llama-swap up, providing zero - teardown benefit. Flagged, not touched, per the same aux-model gap above. +**NOT cut over — 21 Hermes agent profiles' aux models + OpenViking VLM.** +See "Critical architectural finding" below — this was never attempted once +the VRAM collision was discovered, would have made things categorically +worse. + +## Critical architectural finding: vLLM CANNOT be continuously resident alongside llama-swap on this 24GB card (t_e6facb19, 2026-08-31) + +After validating vLLM's two processes (Qwen2.5-32B-Instruct-AWQ + nomic-embed- +text-v1.5, ~22.8GB combined) work correctly in isolation, this role's +`vllm_service_enabled`/`vllm_service_state` were flipped to `true`/`started` +as host_vars overrides to make the deployment permanent (per the task's +"enable for boot" requirement) — llama-swap was then restarted alongside +vLLM to preserve its own consumers. **Result: llama-swap could no longer +load ANY of its own generative models.** Every `/v1/chat/completions` +request against `Qwen3.8-27B-Q4_K_M` or the `Qwen3-8B` aux models failed +with `{"error":"unspecific error: upstream command exited prematurely", +"src":"llama-swap"}` — llama-server's own OOM at spawn time, only ~1.8GB +free on a 24GB card once vLLM's ~22.8GB was already claimed. + +**Confirmed by direct A/B test, not inference:** identical +`Qwen3.8-27B-Q4_K_M` chat completion request returned HTTP 500 with vLLM's +two processes running, then HTTP 200 with a real completion within seconds +of `systemctl stop vllm.service vllm-nomic-embed-text-v1.5.service` — same +llama-swap process, same request, only the GPU memory pressure changed. + +**This is a hard architectural collision, not a tunable-parameter problem.** +llama-swap needs ~18-20GB for its own primary model (Qwen3.8-27B-Q4_K_M); +vLLM's two processes need ~22.8GB even with `enforce_eager` and a lowered +`gpu_memory_utilization`. The two together need more VRAM than a 24GB card +has once both hold real models resident — there is no `gpu_memory_utilization` +value that resolves this while both stacks serve real production models +simultaneously. + +**Consequence — reverted the boot-persistence flip.** `vllm_service_enabled` +and `vllm_service_state` are back to role defaults (`false`/`stopped`) in +`host_vars/astro-orbiter/vars.yml`. vLLM stays staged (venv, model weights, +systemd units all in place) and can be started for a brief shadow-validation +window (same pattern as t_ca1af9fb's original Phase 5), but is NOT safe to +leave resident in production alongside llama-swap. + +**Path forward — requires a human decision, not more role tuning:** +1. Full llama-swap teardown (t_6dff1ecc) BEFORE vLLM gets permanent + residency — but that breaks the 21 agent profiles' aux-model tasks and + OpenViking's VLM unless those consumers are migrated to a different + backend first (Anthropic API, a second smaller local box, or a + redesigned single-process serving strategy that covers all the models + vLLM and llama-swap currently split between them). +2. Accept vLLM as a shadow-only / on-demand stack (manually started for + specific validated windows, stopped otherwise) and do NOT attempt + permanent Hindsight cutover — keeps llama-swap as the sole continuous + production serving layer, matching the pre-task state. +3. A hardware change (larger GPU, or a second GPU) — out of scope for this + task, flagging for Ryan's awareness if the aux-model consumer set is + expected to grow. + +Comment posted on t_6dff1ecc with this finding — the teardown task remains +correctly blocked; this task's completion does NOT unblock it, because full +cutover to vLLM is not achievable within this card's VRAM budget as +currently scoped. ## Validation Log (2026-08-31, t_ca1af9fb) diff --git a/cluster/applications/hindsight/externalsecret.yaml b/cluster/applications/hindsight/externalsecret.yaml index 4476dc5..7a3aa56 100644 --- a/cluster/applications/hindsight/externalsecret.yaml +++ b/cluster/applications/hindsight/externalsecret.yaml @@ -75,15 +75,15 @@ spec: remoteRef: key: hindsight property: HINDSIGHT_API_MCP_AUTH_TOKEN - # Cutover to vLLM (t_e6facb19, 2026-08-31): astro-orbiter's llama-swap - # (which needed no real auth) is being retired for Hindsight's - # completions+embeddings roles. vLLM enforces a real API key — read the - # SAME key roles/deploy-vllm's api-key.yml phase writes to - # /etc/vllm/api-key.env on astro-orbiter, sourced from 1Password - # op://mk-labs/vllm/api-key (item "vllm", field "api-key", vault mk-labs). - # Replaces the prior "nous" item's api-key (Nous free-tier fallback, - # no longer the active LLM backend once this cutover lands). + # 1Password item "nous" (vault mk-labs), field "api-key" (Ryan-provisioned, + # decision 4). REVERTED to this source (t_e6facb19, 2026-08-31) after the + # vLLM cutover attempt was reverted -- llama-swap doesn't validate the + # API key at all, so this value is effectively unused by the live LLM + # path, but restoring the original source keeps this file's intent + # honest (Nous fallback item, not vLLM's real auth key) until a real + # vLLM cutover is safe to attempt again. See values.yaml's api.env + # comment for the full incident writeup. - secretKey: HINDSIGHT_API_LLM_API_KEY remoteRef: - key: vllm + key: nous property: api-key diff --git a/cluster/applications/hindsight/values.yaml b/cluster/applications/hindsight/values.yaml index 1df216d..462a0ab 100644 --- a/cluster/applications/hindsight/values.yaml +++ b/cluster/applications/hindsight/values.yaml @@ -60,29 +60,26 @@ existingSecret: hindsight-credentials # ---------------------------------------------------------------------------- api: env: - # Cutover to vLLM (t_e6facb19, 2026-08-31): astro-orbiter's llama-swap is - # being retired for the completions role Hindsight needs. vLLM serves - # Qwen2.5-32B-Instruct-AWQ (:8000, completions) — validated end-to-end - # (health, /v1/chat/completions) in a shadow window with llama-swap - # stopped. Model swap from Qwen3.8-27B to Qwen2.5-32B-Instruct-AWQ: both - # are capable instruction-tuned models; confirm structured-extraction/ - # tagging behavior (the prior stepfun/solar 400 "missing tags" failure - # mode) holds up after this swap — see - # references/hindsight-backend-fallback-to-local-qwen-workflow.md if it - # regresses and llama-swap needs to come back temporarily. - HINDSIGHT_API_LLM_BASE_URL: "http://astro-orbiter:8000/v1" + # REVERTED (t_e6facb19, 2026-08-31) -- see full incident writeup below. + # Attempted cutover to vLLM (:8000, Qwen2.5-32B-Instruct-AWQ), validated + # working in isolation (health, /v1/chat/completions, and a live + # hindsight_retain + recall round-trip all succeeded), but reverted + # after discovering vLLM cannot safely stay resident on this 24GB card + # AT THE SAME TIME as llama-swap needs to serve its own generative + # models: with vLLM's two processes loaded, every llama-swap + # /v1/chat/completions request (Qwen3.8-27B, Qwen3-8B aux models) failed + # with "upstream command exited prematurely" (an OOM at llama-server + # spawn time). That would break the 21 Hermes agent profiles' aux-model + # tasks and OpenViking's VLM -- a much larger blast radius than + # Hindsight alone. Since Hindsight's LLM endpoint must be reachable + # continuously (not just in a brief validation window), and vLLM cannot + # be continuously resident without starving llama-swap, pointing + # Hindsight at vLLM is unsafe until a real architectural fix lands (see + # roles/deploy-vllm README "Known Gaps"). Restored to llama-swap + # (astro-orbiter:8001, Qwen3.8-27B-Q4_K_M) -- the pre-task working state. + HINDSIGHT_API_LLM_BASE_URL: "http://astro-orbiter:8001/v1" HINDSIGHT_API_LLM_PROVIDER: "openai" - HINDSIGHT_API_LLM_MODEL: "Qwen2.5-32B-Instruct-AWQ" - # vLLM's Qwen2.5-32B-Instruct-AWQ is deployed with --max-model-len 8192 - # (roles/deploy-vllm/defaults/main.yml vllm_max_model_len; llama-swap's - # Qwen3.8-27B ran at ctx=65536, matching Hindsight's default - # RETAIN_MAX_COMPLETION_TOKENS=64000). Discovered via a live retain 500: - # "max_tokens=64000 cannot be greater than max_model_len=8192" — every - # retain call was requesting a 64K-token completion budget the new - # backend physically cannot serve. Lowered to 4096, comfortably under - # 8192 with room for the ~3000-char (RETAIN_CHUNK_SIZE) input chunk plus - # prompt overhead. Verified with a live hindsight_retain call post-fix. - HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS: "4096" + HINDSIGHT_API_LLM_MODEL: "Qwen3.8-27B-Q4_K_M" # DO NOT set HINDSIGHT_API_EMBEDDINGS_* here (t_e6facb19, 2026-08-31 # attempted this, reverted after a production incident — see below). #