REVERT: vLLM cannot be continuously resident alongside llama-swap (t_e6facb19)

Critical finding: flipping vllm_service_enabled/state=true/started and
restarting llama-swap alongside it broke llama-swap's ability to load
ANY of its own generative models -- every /v1/chat/completions request
against Qwen3.8-27B-Q4_K_M or Qwen3-8B aux models failed with
'upstream command exited prematurely' (llama-server OOM at spawn,
~1.8GB free on this 24GB card once vLLM's ~22.8GB was claimed).
Confirmed by direct A/B: same request 500s with vLLM running, 200s
seconds after stopping it.

This breaks 21 Hermes agent profiles' aux-model tasks (skills_hub,
approval, mcp, title_generation, profile_describer, compression) plus
OpenViking's VLM -- a far larger blast radius than Hindsight's single
LLM endpoint. Reverted:
- vllm_service_enabled/state back to role defaults (false/stopped) --
  vLLM stays staged, startable for a brief validated shadow window,
  NOT safe to leave resident in production.
- Hindsight's HINDSIGHT_API_LLM_BASE_URL back to llama-swap
  (astro-orbiter:8001, Qwen3.8-27B-Q4_K_M) and the API key secret
  source back to the Nous fallback item (pre-task state) --
  the vLLM cutover, while functionally validated in isolation
  (health, /v1/chat/completions, and a live hindsight_retain+recall
  round-trip all succeeded), requires continuous vLLM availability
  which is now known to be unsafe on this card.

Comment posted on t_6dff1ecc: teardown remains correctly blocked --
full cutover is not achievable within this card's VRAM budget as
currently scoped. Needs a human decision on aux-model migration
strategy (see roles/deploy-vllm README's 'Critical architectural
finding' section) before any further progress.

cluster/applications/hindsight/externalsecret.yaml

@@ -75,15 +75,15 @@ spec:
       remoteRef:
         key: hindsight
         property: HINDSIGHT_API_MCP_AUTH_TOKEN
-    # Cutover to vLLM (t_e6facb19, 2026-08-31): astro-orbiter's llama-swap
-    # (which needed no real auth) is being retired for Hindsight's
-    # completions+embeddings roles. vLLM enforces a real API key — read the
-    # SAME key roles/deploy-vllm's api-key.yml phase writes to
-    # /etc/vllm/api-key.env on astro-orbiter, sourced from 1Password
-    # op://mk-labs/vllm/api-key (item "vllm", field "api-key", vault mk-labs).
-    # Replaces the prior "nous" item's api-key (Nous free-tier fallback,
-    # no longer the active LLM backend once this cutover lands).
+    # 1Password item "nous" (vault mk-labs), field "api-key" (Ryan-provisioned,
+    # decision 4). REVERTED to this source (t_e6facb19, 2026-08-31) after the
+    # vLLM cutover attempt was reverted -- llama-swap doesn't validate the
+    # API key at all, so this value is effectively unused by the live LLM
+    # path, but restoring the original source keeps this file's intent
+    # honest (Nous fallback item, not vLLM's real auth key) until a real
+    # vLLM cutover is safe to attempt again. See values.yaml's api.env
+    # comment for the full incident writeup.
     - secretKey: HINDSIGHT_API_LLM_API_KEY
       remoteRef:
-        key: vllm
+        key: nous
         property: api-key

cluster/applications/hindsight/values.yaml

@@ -60,29 +60,26 @@ existingSecret: hindsight-credentials
 # ----------------------------------------------------------------------------
 api:
   env:
-    # Cutover to vLLM (t_e6facb19, 2026-08-31): astro-orbiter's llama-swap is
-    # being retired for the completions role Hindsight needs. vLLM serves
-    # Qwen2.5-32B-Instruct-AWQ (:8000, completions) — validated end-to-end
-    # (health, /v1/chat/completions) in a shadow window with llama-swap
-    # stopped. Model swap from Qwen3.8-27B to Qwen2.5-32B-Instruct-AWQ: both
-    # are capable instruction-tuned models; confirm structured-extraction/
-    # tagging behavior (the prior stepfun/solar 400 "missing tags" failure
-    # mode) holds up after this swap — see
-    # references/hindsight-backend-fallback-to-local-qwen-workflow.md if it
-    # regresses and llama-swap needs to come back temporarily.
-    HINDSIGHT_API_LLM_BASE_URL: "http://astro-orbiter:8000/v1"
+    # REVERTED (t_e6facb19, 2026-08-31) -- see full incident writeup below.
+    # Attempted cutover to vLLM (:8000, Qwen2.5-32B-Instruct-AWQ), validated
+    # working in isolation (health, /v1/chat/completions, and a live
+    # hindsight_retain + recall round-trip all succeeded), but reverted
+    # after discovering vLLM cannot safely stay resident on this 24GB card
+    # AT THE SAME TIME as llama-swap needs to serve its own generative
+    # models: with vLLM's two processes loaded, every llama-swap
+    # /v1/chat/completions request (Qwen3.8-27B, Qwen3-8B aux models) failed
+    # with "upstream command exited prematurely" (an OOM at llama-server
+    # spawn time). That would break the 21 Hermes agent profiles' aux-model
+    # tasks and OpenViking's VLM -- a much larger blast radius than
+    # Hindsight alone. Since Hindsight's LLM endpoint must be reachable
+    # continuously (not just in a brief validation window), and vLLM cannot
+    # be continuously resident without starving llama-swap, pointing
+    # Hindsight at vLLM is unsafe until a real architectural fix lands (see
+    # roles/deploy-vllm README "Known Gaps"). Restored to llama-swap
+    # (astro-orbiter:8001, Qwen3.8-27B-Q4_K_M) -- the pre-task working state.
+    HINDSIGHT_API_LLM_BASE_URL: "http://astro-orbiter:8001/v1"
     HINDSIGHT_API_LLM_PROVIDER: "openai"
-    HINDSIGHT_API_LLM_MODEL: "Qwen2.5-32B-Instruct-AWQ"
-    # vLLM's Qwen2.5-32B-Instruct-AWQ is deployed with --max-model-len 8192
-    # (roles/deploy-vllm/defaults/main.yml vllm_max_model_len; llama-swap's
-    # Qwen3.8-27B ran at ctx=65536, matching Hindsight's default
-    # RETAIN_MAX_COMPLETION_TOKENS=64000). Discovered via a live retain 500:
-    # "max_tokens=64000 cannot be greater than max_model_len=8192" — every
-    # retain call was requesting a 64K-token completion budget the new
-    # backend physically cannot serve. Lowered to 4096, comfortably under
-    # 8192 with room for the ~3000-char (RETAIN_CHUNK_SIZE) input chunk plus
-    # prompt overhead. Verified with a live hindsight_retain call post-fix.
-    HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS: "4096"
+    HINDSIGHT_API_LLM_MODEL: "Qwen3.8-27B-Q4_K_M"
     # DO NOT set HINDSIGHT_API_EMBEDDINGS_* here (t_e6facb19, 2026-08-31
     # attempted this, reverted after a production incident — see below).
     #