REVERT: vLLM cannot be continuously resident alongside llama-swap (t_e6facb19)

Critical finding: flipping vllm_service_enabled/state=true/started and
restarting llama-swap alongside it broke llama-swap's ability to load
ANY of its own generative models -- every /v1/chat/completions request
against Qwen3.8-27B-Q4_K_M or Qwen3-8B aux models failed with
'upstream command exited prematurely' (llama-server OOM at spawn,
~1.8GB free on this 24GB card once vLLM's ~22.8GB was claimed).
Confirmed by direct A/B: same request 500s with vLLM running, 200s
seconds after stopping it.

This breaks 21 Hermes agent profiles' aux-model tasks (skills_hub,
approval, mcp, title_generation, profile_describer, compression) plus
OpenViking's VLM -- a far larger blast radius than Hindsight's single
LLM endpoint. Reverted:
- vllm_service_enabled/state back to role defaults (false/stopped) --
  vLLM stays staged, startable for a brief validated shadow window,
  NOT safe to leave resident in production.
- Hindsight's HINDSIGHT_API_LLM_BASE_URL back to llama-swap
  (astro-orbiter:8001, Qwen3.8-27B-Q4_K_M) and the API key secret
  source back to the Nous fallback item (pre-task state) --
  the vLLM cutover, while functionally validated in isolation
  (health, /v1/chat/completions, and a live hindsight_retain+recall
  round-trip all succeeded), requires continuous vLLM availability
  which is now known to be unsafe on this card.

Comment posted on t_6dff1ecc: teardown remains correctly blocked --
full cutover is not achievable within this card's VRAM budget as
currently scoped. Needs a human decision on aux-model migration
strategy (see roles/deploy-vllm README's 'Critical architectural
finding' section) before any further progress.

ansible/roles/deploy-vllm/README.md

@@ -89,25 +89,12 @@ supports (`--quantization awq`) and which fit the VRAM budget:
 - `vllm_service_enabled` defaults to `false` deliberately — see "Deliberate
   staging-first default" above. Flip together with the cutover step, not
   before.
-- **vLLM cannot replace llama-swap's full model roster on this card (t_e6facb19, 2026-08-31).**
-  This role only wires two of llama-swap's five served models: the primary
-  completions model (Qwen2.5-32B-Instruct-AWQ, replacing Qwen3.8-27B) and
-  the embedding model (nomic-embed-text-v1.5). llama-swap ALSO serves
-  Qwen3-8B-Q4_K_M(-no_think), Phi-3.5-mini-instruct-Q8_0,
-  Meta-Llama-3.1-8B-Instruct-Q4_K_M, and Qwen2.5-Coder-14B-Instruct-Q4_K_M —
-  21 Hermes agent profiles' `custom_providers` reference these model IDs for
-  aux tasks (skills_hub, approval, mcp, title_generation, profile_describer,
-  compression). vLLM 0.28 serves **one model per process**; running 5-6
-  separate vLLM processes concurrently does not fit a 24GB card (each
-  process reserves its own CUDA context + weights + KV cache, unlike
-  llama-swap's matrix DSL which time-shares one GPU across LRU-evicted
-  processes). **Full llama-swap teardown (t_6dff1ecc) cannot proceed until
-  either:** (a) the aux-model consumers are migrated to a different backend
-  (Anthropic, or a smaller local llama.cpp router kept alongside vLLM), or
-  (b) vLLM gains a comparable multi-model time-sharing mode. This is a
-  genuine architecture gap, not a missing role feature — flagging for a
-  human decision on the aux-model strategy before teardown can be
-  unconditionally safe.
+- **vLLM cannot replace llama-swap's full model roster on this card — see
+  "Critical architectural finding" section below for the full incident.**
+  Short version: vLLM's one-model-per-process design plus llama-swap's own
+  VRAM needs exceed this 24GB card's capacity when both must serve real
+  models simultaneously. Full llama-swap teardown (t_6dff1ecc) cannot
+  proceed until a human decides the aux-model + VRAM strategy.
 
 ## Embedding-mode support (t_e6facb19, 2026-08-31)
 
@@ -162,32 +149,87 @@ crash-looped repeatedly:
 
 ## Consumer cutover status (t_e6facb19, 2026-08-31)
 
-**Cut over (validated end-to-end):**
-- Hindsight (`cluster/applications/hindsight/values.yaml` +
-  `externalsecret.yaml`): `HINDSIGHT_API_LLM_BASE_URL` → vLLM `:8000`
-  (Qwen2.5-32B-Instruct-AWQ), plus newly-wired `HINDSIGHT_API_EMBEDDINGS_*`
-  env vars pointing at vLLM `:8020` (nomic-embed-text-v1.5). Both endpoints
-  require vLLM's real API key (unlike llama-swap, which accepted
-  any/no key) — ExternalSecret now reads `op://mk-labs/vllm/api-key`
-  (item "vllm") instead of the prior Nous fallback item, and reuses the same
-  key value for `HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY` (both vLLM
-  endpoints share one key file per `tasks/api-key.yml`).
+**Attempted, then REVERTED — Hindsight LLM cutover.** Hindsight's
+`HINDSIGHT_API_LLM_BASE_URL` was pointed at vLLM `:8000`
+(Qwen2.5-32B-Instruct-AWQ) and validated working in isolation: health,
+`/v1/chat/completions`, and a live `hindsight_retain` + recall round-trip
+all succeeded (after also fixing `HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS`,
+which defaulted to 64000 — exceeding vLLM's `max_model_len=8192` — down to
+4096). **Reverted anyway**, because of a severe discovery documented in the
+next section: vLLM cannot stay resident on this card without starving
+llama-swap, and Hindsight's LLM endpoint needs continuous availability, not
+just a validation window. Restored to `http://astro-orbiter:8001/v1`
+(llama-swap, Qwen3.8-27B-Q4_K_M) — the pre-task working state.
 
-**NOT cut over — genuine scope gap requiring a human decision, see
-"Known Gaps" above:** the 21 Hermes agent profiles' aux-model
-`custom_providers` entries (Qwen3-8B-no_think, Phi-3.5-mini, Meta-Llama-3.1-8B,
-Qwen2.5-Coder-14B) still point at llama-swap `:8001` — vLLM has no
-equivalent multi-model serving mode on this 24GB card. llama-swap MUST stay
-up to serve these until that gap is resolved. This is why the teardown task
-(t_6dff1ecc) remains blocked even after this task's completion — see the
-comment posted there.
+**NOT cut over — embeddings.** Hindsight was discovered to have NEVER used
+astro-orbiter for embeddings — it defaults to a bundled local
+`BAAI/bge-small-en-v1.5` (384-dim) embedder whenever
+`HINDSIGHT_API_EMBEDDINGS_PROVIDER` is unset, which was always the case here.
+Pointing it at vLLM's `nomic-embed-text-v1.5` (768-dim) crash-looped the pod:
+`RuntimeError: Cannot change embedding dimension from 384 to 768:
+memory_units table contains 1289 rows with embeddings.` Re-embedding all
+existing memory data across ~20 agent banks is destructive and irreversible
+— reverted immediately, left as a separate, explicitly-approved future task.
 
-- OpenViking (`cluster/platform/openviking/values.yaml`): still points at
-  llama-swap `:8001` (`Meta-Llama-3.1-8B-Instruct-Q4_K_M` VLM + nomic-embed
-  for dense embeddings). Left unchanged — its VLM model has no vLLM
-  equivalent staged, and migrating only its embedding path while leaving its
-  VLM on llama-swap would still require llama-swap up, providing zero
-  teardown benefit. Flagged, not touched, per the same aux-model gap above.
+**NOT cut over — 21 Hermes agent profiles' aux models + OpenViking VLM.**
+See "Critical architectural finding" below — this was never attempted once
+the VRAM collision was discovered, would have made things categorically
+worse.
+
+## Critical architectural finding: vLLM CANNOT be continuously resident alongside llama-swap on this 24GB card (t_e6facb19, 2026-08-31)
+
+After validating vLLM's two processes (Qwen2.5-32B-Instruct-AWQ + nomic-embed-
+text-v1.5, ~22.8GB combined) work correctly in isolation, this role's
+`vllm_service_enabled`/`vllm_service_state` were flipped to `true`/`started`
+as host_vars overrides to make the deployment permanent (per the task's
+"enable for boot" requirement) — llama-swap was then restarted alongside
+vLLM to preserve its own consumers. **Result: llama-swap could no longer
+load ANY of its own generative models.** Every `/v1/chat/completions`
+request against `Qwen3.8-27B-Q4_K_M` or the `Qwen3-8B` aux models failed
+with `{"error":"unspecific error: upstream command exited prematurely",
+"src":"llama-swap"}` — llama-server's own OOM at spawn time, only ~1.8GB
+free on a 24GB card once vLLM's ~22.8GB was already claimed.
+
+**Confirmed by direct A/B test, not inference:** identical
+`Qwen3.8-27B-Q4_K_M` chat completion request returned HTTP 500 with vLLM's
+two processes running, then HTTP 200 with a real completion within seconds
+of `systemctl stop vllm.service vllm-nomic-embed-text-v1.5.service` — same
+llama-swap process, same request, only the GPU memory pressure changed.
+
+**This is a hard architectural collision, not a tunable-parameter problem.**
+llama-swap needs ~18-20GB for its own primary model (Qwen3.8-27B-Q4_K_M);
+vLLM's two processes need ~22.8GB even with `enforce_eager` and a lowered
+`gpu_memory_utilization`. The two together need more VRAM than a 24GB card
+has once both hold real models resident — there is no `gpu_memory_utilization`
+value that resolves this while both stacks serve real production models
+simultaneously.
+
+**Consequence — reverted the boot-persistence flip.** `vllm_service_enabled`
+and `vllm_service_state` are back to role defaults (`false`/`stopped`) in
+`host_vars/astro-orbiter/vars.yml`. vLLM stays staged (venv, model weights,
+systemd units all in place) and can be started for a brief shadow-validation
+window (same pattern as t_ca1af9fb's original Phase 5), but is NOT safe to
+leave resident in production alongside llama-swap.
+
+**Path forward — requires a human decision, not more role tuning:**
+1. Full llama-swap teardown (t_6dff1ecc) BEFORE vLLM gets permanent
+   residency — but that breaks the 21 agent profiles' aux-model tasks and
+   OpenViking's VLM unless those consumers are migrated to a different
+   backend first (Anthropic API, a second smaller local box, or a
+   redesigned single-process serving strategy that covers all the models
+   vLLM and llama-swap currently split between them).
+2. Accept vLLM as a shadow-only / on-demand stack (manually started for
+   specific validated windows, stopped otherwise) and do NOT attempt
+   permanent Hindsight cutover — keeps llama-swap as the sole continuous
+   production serving layer, matching the pre-task state.
+3. A hardware change (larger GPU, or a second GPU) — out of scope for this
+   task, flagging for Ryan's awareness if the aux-model consumer set is
+   expected to grow.
+
+Comment posted on t_6dff1ecc with this finding — the teardown task remains
+correctly blocked; this task's completion does NOT unblock it, because full
+cutover to vLLM is not achievable within this card's VRAM budget as
+currently scoped.
 
 
 ## Validation Log (2026-08-31, t_ca1af9fb)