REVERT: vLLM cannot be continuously resident alongside llama-swap (t_e6facb19)

Critical finding: flipping vllm_service_enabled/state=true/started and
restarting llama-swap alongside it broke llama-swap's ability to load
ANY of its own generative models -- every /v1/chat/completions request
against Qwen3.8-27B-Q4_K_M or Qwen3-8B aux models failed with
'upstream command exited prematurely' (llama-server OOM at spawn,
~1.8GB free on this 24GB card once vLLM's ~22.8GB was claimed).
Confirmed by direct A/B: same request 500s with vLLM running, 200s
seconds after stopping it.

This breaks 21 Hermes agent profiles' aux-model tasks (skills_hub,
approval, mcp, title_generation, profile_describer, compression) plus
OpenViking's VLM -- a far larger blast radius than Hindsight's single
LLM endpoint. Reverted:
- vllm_service_enabled/state back to role defaults (false/stopped) --
  vLLM stays staged, startable for a brief validated shadow window,
  NOT safe to leave resident in production.
- Hindsight's HINDSIGHT_API_LLM_BASE_URL back to llama-swap
  (astro-orbiter:8001, Qwen3.8-27B-Q4_K_M) and the API key secret
  source back to the Nous fallback item (pre-task state) --
  the vLLM cutover, while functionally validated in isolation
  (health, /v1/chat/completions, and a live hindsight_retain+recall
  round-trip all succeeded), requires continuous vLLM availability
  which is now known to be unsafe on this card.

Comment posted on t_6dff1ecc: teardown remains correctly blocked --
full cutover is not achievable within this card's VRAM budget as
currently scoped. Needs a human decision on aux-model migration
strategy (see roles/deploy-vllm README's 'Critical architectural
finding' section) before any further progress.

ansible/host_vars/astro-orbiter/vars.yml

@@ -163,3 +163,26 @@ vllm_models:
     trust_remote_code: true
     enabled: true
 
+# --- deploy-vllm role: boot persistence NOT enabled (t_e6facb19, 2026-08-31) ---
+# ATTEMPTED enabling vllm_service_enabled/state=started here, then reverted
+# after a production-breaking discovery: with vLLM's two processes
+# (Qwen2.5-32B + nomic-embed, ~22.8GB combined) resident, llama-swap could no
+# longer load ANY of its own generative models -- every /v1/chat/completions
+# request against Qwen3.8-27B-Q4_K_M or the Qwen3-8B aux models failed with
+# "upstream command exited prematurely" (llama-server's own OOM at spawn
+# time, silently swallowed by llama-swap's generic error). Confirmed by
+# direct A/B: same request 500s with vLLM running, 200s within seconds of
+# `systemctl stop vllm.service vllm-nomic-embed-text-v1.5.service`. This
+# breaks all 21 Hermes agent profiles' aux-model tasks (skills_hub, approval,
+# mcp, title_generation, profile_describer, compression) plus OpenViking's
+# VLM -- a severe regression, worse than the status quo. Left
+# vllm_service_enabled/state at role defaults (false/stopped) -- vLLM stays
+# staged and manually startable for a brief shadow window (same pattern as
+# t_ca1af9fb's original validation), but is NOT safe to leave resident
+# alongside llama-swap on this 24GB card. See README's "Known Gaps" section:
+# full teardown of llama-swap (t_6dff1ecc) is the ONLY path to giving vLLM
+# permanent residency without starving the other models -- this is not
+# solvable by tuning gpu_memory_utilization further, the two stacks
+# together need more VRAM than this card has once both hold real models
+# resident.
+

ansible/roles/deploy-vllm/README.md

@@ -89,25 +89,12 @@ supports (`--quantization awq`) and which fit the VRAM budget:
 - `vllm_service_enabled` defaults to `false` deliberately — see "Deliberate
   staging-first default" above. Flip together with the cutover step, not
   before.
-- **vLLM cannot replace llama-swap's full model roster on this card (t_e6facb19, 2026-08-31).**
+- **vLLM cannot replace llama-swap's full model roster on this card — see
-  This role only wires two of llama-swap's five served models: the primary
+  "Critical architectural finding" section below for the full incident.**
-  completions model (Qwen2.5-32B-Instruct-AWQ, replacing Qwen3.8-27B) and
+  Short version: vLLM's one-model-per-process design plus llama-swap's own
-  the embedding model (nomic-embed-text-v1.5). llama-swap ALSO serves
+  VRAM needs exceed this 24GB card's capacity when both must serve real
-  Qwen3-8B-Q4_K_M(-no_think), Phi-3.5-mini-instruct-Q8_0,
+  models simultaneously. Full llama-swap teardown (t_6dff1ecc) cannot
-  Meta-Llama-3.1-8B-Instruct-Q4_K_M, and Qwen2.5-Coder-14B-Instruct-Q4_K_M —
+  proceed until a human decides the aux-model + VRAM strategy.
-  21 Hermes agent profiles' `custom_providers` reference these model IDs for
-  aux tasks (skills_hub, approval, mcp, title_generation, profile_describer,
-  compression). vLLM 0.28 serves **one model per process**; running 5-6
-  separate vLLM processes concurrently does not fit a 24GB card (each
-  process reserves its own CUDA context + weights + KV cache, unlike
-  llama-swap's matrix DSL which time-shares one GPU across LRU-evicted
-  processes). **Full llama-swap teardown (t_6dff1ecc) cannot proceed until
-  either:** (a) the aux-model consumers are migrated to a different backend
-  (Anthropic, or a smaller local llama.cpp router kept alongside vLLM), or
-  (b) vLLM gains a comparable multi-model time-sharing mode. This is a
-  genuine architecture gap, not a missing role feature — flagging for a
-  human decision on the aux-model strategy before teardown can be
-  unconditionally safe.
 
 ## Embedding-mode support (t_e6facb19, 2026-08-31)
 
@@ -162,32 +149,87 @@ crash-looped repeatedly:
 
 ## Consumer cutover status (t_e6facb19, 2026-08-31)
 
-**Cut over (validated end-to-end):**
+**Attempted, then REVERTED — Hindsight LLM cutover.** Hindsight's
-- Hindsight (`cluster/applications/hindsight/values.yaml` +
+`HINDSIGHT_API_LLM_BASE_URL` was pointed at vLLM `:8000`
-  `externalsecret.yaml`): `HINDSIGHT_API_LLM_BASE_URL` → vLLM `:8000`
+(Qwen2.5-32B-Instruct-AWQ) and validated working in isolation: health,
-  (Qwen2.5-32B-Instruct-AWQ), plus newly-wired `HINDSIGHT_API_EMBEDDINGS_*`
+`/v1/chat/completions`, and a live `hindsight_retain` + recall round-trip
-  env vars pointing at vLLM `:8020` (nomic-embed-text-v1.5). Both endpoints
+all succeeded (after also fixing `HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS`,
-  require vLLM's real API key (unlike llama-swap, which accepted
+which defaulted to 64000 — exceeding vLLM's `max_model_len=8192` — down to
-  any/no key) — ExternalSecret now reads `op://mk-labs/vllm/api-key`
+4096). **Reverted anyway**, because of a severe discovery documented in the
-  (item "vllm") instead of the prior Nous fallback item, and reuses the same
+next section: vLLM cannot stay resident on this card without starving
-  key value for `HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY` (both vLLM
+llama-swap, and Hindsight's LLM endpoint needs continuous availability, not
-  endpoints share one key file per `tasks/api-key.yml`).
+just a validation window. Restored to `http://astro-orbiter:8001/v1`
+(llama-swap, Qwen3.8-27B-Q4_K_M) — the pre-task working state.
 
-**NOT cut over — genuine scope gap requiring a human decision, see
+**NOT cut over — embeddings.** Hindsight was discovered to have NEVER used
-"Known Gaps" above:** the 21 Hermes agent profiles' aux-model
+astro-orbiter for embeddings — it defaults to a bundled local
-`custom_providers` entries (Qwen3-8B-no_think, Phi-3.5-mini, Meta-Llama-3.1-8B,
+`BAAI/bge-small-en-v1.5` (384-dim) embedder whenever
-Qwen2.5-Coder-14B) still point at llama-swap `:8001` — vLLM has no
+`HINDSIGHT_API_EMBEDDINGS_PROVIDER` is unset, which was always the case here.
-equivalent multi-model serving mode on this 24GB card. llama-swap MUST stay
+Pointing it at vLLM's `nomic-embed-text-v1.5` (768-dim) crash-looped the pod:
-up to serve these until that gap is resolved. This is why the teardown task
+`RuntimeError: Cannot change embedding dimension from 384 to 768:
-(t_6dff1ecc) remains blocked even after this task's completion — see the
+memory_units table contains 1289 rows with embeddings.` Re-embedding all
-comment posted there.
+existing memory data across ~20 agent banks is destructive and irreversible
+— reverted immediately, left as a separate, explicitly-approved future task.
 
-- OpenViking (`cluster/platform/openviking/values.yaml`): still points at
+**NOT cut over — 21 Hermes agent profiles' aux models + OpenViking VLM.**
-  llama-swap `:8001` (`Meta-Llama-3.1-8B-Instruct-Q4_K_M` VLM + nomic-embed
+See "Critical architectural finding" below — this was never attempted once
-  for dense embeddings). Left unchanged — its VLM model has no vLLM
+the VRAM collision was discovered, would have made things categorically
-  equivalent staged, and migrating only its embedding path while leaving its
+worse.
-  VLM on llama-swap would still require llama-swap up, providing zero
+
-  teardown benefit. Flagged, not touched, per the same aux-model gap above.
+## Critical architectural finding: vLLM CANNOT be continuously resident alongside llama-swap on this 24GB card (t_e6facb19, 2026-08-31)
+
+After validating vLLM's two processes (Qwen2.5-32B-Instruct-AWQ + nomic-embed-
+text-v1.5, ~22.8GB combined) work correctly in isolation, this role's
+`vllm_service_enabled`/`vllm_service_state` were flipped to `true`/`started`
+as host_vars overrides to make the deployment permanent (per the task's
+"enable for boot" requirement) — llama-swap was then restarted alongside
+vLLM to preserve its own consumers. **Result: llama-swap could no longer
+load ANY of its own generative models.** Every `/v1/chat/completions`
+request against `Qwen3.8-27B-Q4_K_M` or the `Qwen3-8B` aux models failed
+with `{"error":"unspecific error: upstream command exited prematurely",
+"src":"llama-swap"}` — llama-server's own OOM at spawn time, only ~1.8GB
+free on a 24GB card once vLLM's ~22.8GB was already claimed.
+
+**Confirmed by direct A/B test, not inference:** identical
+`Qwen3.8-27B-Q4_K_M` chat completion request returned HTTP 500 with vLLM's
+two processes running, then HTTP 200 with a real completion within seconds
+of `systemctl stop vllm.service vllm-nomic-embed-text-v1.5.service` — same
+llama-swap process, same request, only the GPU memory pressure changed.
+
+**This is a hard architectural collision, not a tunable-parameter problem.**
+llama-swap needs ~18-20GB for its own primary model (Qwen3.8-27B-Q4_K_M);
+vLLM's two processes need ~22.8GB even with `enforce_eager` and a lowered
+`gpu_memory_utilization`. The two together need more VRAM than a 24GB card
+has once both hold real models resident — there is no `gpu_memory_utilization`
+value that resolves this while both stacks serve real production models
+simultaneously.
+
+**Consequence — reverted the boot-persistence flip.** `vllm_service_enabled`
+and `vllm_service_state` are back to role defaults (`false`/`stopped`) in
+`host_vars/astro-orbiter/vars.yml`. vLLM stays staged (venv, model weights,
+systemd units all in place) and can be started for a brief shadow-validation
+window (same pattern as t_ca1af9fb's original Phase 5), but is NOT safe to
+leave resident in production alongside llama-swap.
+
+**Path forward — requires a human decision, not more role tuning:**
+1. Full llama-swap teardown (t_6dff1ecc) BEFORE vLLM gets permanent
+   residency — but that breaks the 21 agent profiles' aux-model tasks and
+   OpenViking's VLM unless those consumers are migrated to a different
+   backend first (Anthropic API, a second smaller local box, or a
+   redesigned single-process serving strategy that covers all the models
+   vLLM and llama-swap currently split between them).
+2. Accept vLLM as a shadow-only / on-demand stack (manually started for
+   specific validated windows, stopped otherwise) and do NOT attempt
+   permanent Hindsight cutover — keeps llama-swap as the sole continuous
+   production serving layer, matching the pre-task state.
+3. A hardware change (larger GPU, or a second GPU) — out of scope for this
+   task, flagging for Ryan's awareness if the aux-model consumer set is
+   expected to grow.
+
+Comment posted on t_6dff1ecc with this finding — the teardown task remains
+correctly blocked; this task's completion does NOT unblock it, because full
+cutover to vLLM is not achievable within this card's VRAM budget as
+currently scoped.
 
 
 ## Validation Log (2026-08-31, t_ca1af9fb)

cluster/applications/hindsight/externalsecret.yaml

@@ -75,15 +75,15 @@ spec:
       remoteRef:
         key: hindsight
         property: HINDSIGHT_API_MCP_AUTH_TOKEN
-    # Cutover to vLLM (t_e6facb19, 2026-08-31): astro-orbiter's llama-swap
+    # 1Password item "nous" (vault mk-labs), field "api-key" (Ryan-provisioned,
-    # (which needed no real auth) is being retired for Hindsight's
+    # decision 4). REVERTED to this source (t_e6facb19, 2026-08-31) after the
-    # completions+embeddings roles. vLLM enforces a real API key — read the
+    # vLLM cutover attempt was reverted -- llama-swap doesn't validate the
-    # SAME key roles/deploy-vllm's api-key.yml phase writes to
+    # API key at all, so this value is effectively unused by the live LLM
-    # /etc/vllm/api-key.env on astro-orbiter, sourced from 1Password
+    # path, but restoring the original source keeps this file's intent
-    # op://mk-labs/vllm/api-key (item "vllm", field "api-key", vault mk-labs).
+    # honest (Nous fallback item, not vLLM's real auth key) until a real
-    # Replaces the prior "nous" item's api-key (Nous free-tier fallback,
+    # vLLM cutover is safe to attempt again. See values.yaml's api.env
-    # no longer the active LLM backend once this cutover lands).
+    # comment for the full incident writeup.
     - secretKey: HINDSIGHT_API_LLM_API_KEY
       remoteRef:
-        key: vllm
+        key: nous
         property: api-key

cluster/applications/hindsight/values.yaml

@@ -60,29 +60,26 @@ existingSecret: hindsight-credentials
 # ----------------------------------------------------------------------------
 api:
   env:
-    # Cutover to vLLM (t_e6facb19, 2026-08-31): astro-orbiter's llama-swap is
+    # REVERTED (t_e6facb19, 2026-08-31) -- see full incident writeup below.
-    # being retired for the completions role Hindsight needs. vLLM serves
+    # Attempted cutover to vLLM (:8000, Qwen2.5-32B-Instruct-AWQ), validated
-    # Qwen2.5-32B-Instruct-AWQ (:8000, completions) — validated end-to-end
+    # working in isolation (health, /v1/chat/completions, and a live
-    # (health, /v1/chat/completions) in a shadow window with llama-swap
+    # hindsight_retain + recall round-trip all succeeded), but reverted
-    # stopped. Model swap from Qwen3.8-27B to Qwen2.5-32B-Instruct-AWQ: both
+    # after discovering vLLM cannot safely stay resident on this 24GB card
-    # are capable instruction-tuned models; confirm structured-extraction/
+    # AT THE SAME TIME as llama-swap needs to serve its own generative
-    # tagging behavior (the prior stepfun/solar 400 "missing tags" failure
+    # models: with vLLM's two processes loaded, every llama-swap
-    # mode) holds up after this swap — see
+    # /v1/chat/completions request (Qwen3.8-27B, Qwen3-8B aux models) failed
-    # references/hindsight-backend-fallback-to-local-qwen-workflow.md if it
+    # with "upstream command exited prematurely" (an OOM at llama-server
-    # regresses and llama-swap needs to come back temporarily.
+    # spawn time). That would break the 21 Hermes agent profiles' aux-model
-    HINDSIGHT_API_LLM_BASE_URL: "http://astro-orbiter:8000/v1"
+    # tasks and OpenViking's VLM -- a much larger blast radius than
+    # Hindsight alone. Since Hindsight's LLM endpoint must be reachable
+    # continuously (not just in a brief validation window), and vLLM cannot
+    # be continuously resident without starving llama-swap, pointing
+    # Hindsight at vLLM is unsafe until a real architectural fix lands (see
+    # roles/deploy-vllm README "Known Gaps"). Restored to llama-swap
+    # (astro-orbiter:8001, Qwen3.8-27B-Q4_K_M) -- the pre-task working state.
+    HINDSIGHT_API_LLM_BASE_URL: "http://astro-orbiter:8001/v1"
     HINDSIGHT_API_LLM_PROVIDER: "openai"
-    HINDSIGHT_API_LLM_MODEL: "Qwen2.5-32B-Instruct-AWQ"
+    HINDSIGHT_API_LLM_MODEL: "Qwen3.8-27B-Q4_K_M"
-    # vLLM's Qwen2.5-32B-Instruct-AWQ is deployed with --max-model-len 8192
-    # (roles/deploy-vllm/defaults/main.yml vllm_max_model_len; llama-swap's
-    # Qwen3.8-27B ran at ctx=65536, matching Hindsight's default
-    # RETAIN_MAX_COMPLETION_TOKENS=64000). Discovered via a live retain 500:
-    # "max_tokens=64000 cannot be greater than max_model_len=8192" — every
-    # retain call was requesting a 64K-token completion budget the new
-    # backend physically cannot serve. Lowered to 4096, comfortably under
-    # 8192 with room for the ~3000-char (RETAIN_CHUNK_SIZE) input chunk plus
-    # prompt overhead. Verified with a live hindsight_retain call post-fix.
-    HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS: "4096"
     # DO NOT set HINDSIGHT_API_EMBEDDINGS_* here (t_e6facb19, 2026-08-31
     # attempted this, reverted after a production incident — see below).
     #