deploy-vllm: add embedding-mode support, cut over Hindsight to vLLM (t_e6facb19)

- vllm.service.j2: branch on role==embedding for --runner pooling
  --convert embed, --no-enable-prefix-caching, per-model
  trust_remote_code toggle (needed for nomic-embed-text-v1.5's custom
  NomicBertModel code), and enforce_eager toggle (needed to avoid CUDA
  graph capture OOM when co-resident with another vLLM process on this
  24GB card).
- tasks/verify.yml: split completions vs embedding smoke tests --
  embedding-mode instances don't serve /v1/completions. Assert a
  non-empty embedding vector, not just HTTP 200.
- host_vars/astro-orbiter: enable nomic-embed-text-v1.5 (port 8020),
  lower primary model's gpu_memory_utilization 0.95->0.90 + add
  enforce_eager after finding 0.95 crash-looped 6-7x before stabilizing
  with co-resident nomic-embed (real fix, confirmed via NRestarts=0
  after clean stop/start, not luck).
- Hindsight (values.yaml + externalsecret.yaml): cut LLM + embeddings
  over to vLLM (:8000, :8020), wire the previously-unset
  HINDSIGHT_API_EMBEDDINGS_* env vars for the first time, and swap the
  API key secret source from the Nous fallback item to vllm/api-key
  (vLLM enforces real auth, llama-swap did not).
- README: document the embedding-mode branch, VRAM findings, and a
  genuine architecture gap -- vLLM's one-model-per-process design
  cannot replace llama-swap's 5-model LRU roster on this 24GB card, so
  21 Hermes profiles' aux-model consumers (Qwen3-8B-no_think,
  Phi-3.5-mini, Meta-Llama-3.1-8B, Qwen2.5-Coder-14B) and OpenViking's
  VLM stay on llama-swap. Full teardown (t_6dff1ecc) needs a human
  decision on the aux-model strategy before it can proceed.

ansible/roles/deploy-vllm/README.md

@@ -82,11 +82,6 @@ supports (`--quantization awq`) and which fit the VRAM budget:
 
 ## Known Gaps / Follow-ups
 
-- The `nomic-embed-text-v1.5` entry in `vllm_models` is present but the
-  `vllm.service.j2` template does not yet branch for embedding-mode flags
-  (`--task embed`). Do not flip `enabled: true` on it without first adding
-  that branch and testing `/v1/embeddings` — this is what Hindsight retain
-  actually depends on, so get it right before cutover.
 - Quarterly API key rotation is documented (`/etc/vllm/API_KEY_ROTATION.md`
   on the target, rendered by `tasks/api-key.yml`) but not automated — no cron
   job exists to force rotation on a schedule. Consider a follow-up cron task
@@ -94,6 +89,106 @@ supports (`--quantization awq`) and which fit the VRAM budget:
 - `vllm_service_enabled` defaults to `false` deliberately — see "Deliberate
   staging-first default" above. Flip together with the cutover step, not
   before.
+- **vLLM cannot replace llama-swap's full model roster on this card (t_e6facb19, 2026-08-31).**
+  This role only wires two of llama-swap's five served models: the primary
+  completions model (Qwen2.5-32B-Instruct-AWQ, replacing Qwen3.8-27B) and
+  the embedding model (nomic-embed-text-v1.5). llama-swap ALSO serves
+  Qwen3-8B-Q4_K_M(-no_think), Phi-3.5-mini-instruct-Q8_0,
+  Meta-Llama-3.1-8B-Instruct-Q4_K_M, and Qwen2.5-Coder-14B-Instruct-Q4_K_M —
+  21 Hermes agent profiles' `custom_providers` reference these model IDs for
+  aux tasks (skills_hub, approval, mcp, title_generation, profile_describer,
+  compression). vLLM 0.28 serves **one model per process**; running 5-6
+  separate vLLM processes concurrently does not fit a 24GB card (each
+  process reserves its own CUDA context + weights + KV cache, unlike
+  llama-swap's matrix DSL which time-shares one GPU across LRU-evicted
+  processes). **Full llama-swap teardown (t_6dff1ecc) cannot proceed until
+  either:** (a) the aux-model consumers are migrated to a different backend
+  (Anthropic, or a smaller local llama.cpp router kept alongside vLLM), or
+  (b) vLLM gains a comparable multi-model time-sharing mode. This is a
+  genuine architecture gap, not a missing role feature — flagging for a
+  human decision on the aux-model strategy before teardown can be
+  unconditionally safe.
+
+## Embedding-mode support (t_e6facb19, 2026-08-31)
+
+`vllm.service.j2` now branches on `role: embedding` entries in `vllm_models`:
+adds `--runner pooling --convert embed` (vLLM's embedding-serving flags —
+see https://docs.vllm.ai/en/latest/models/pooling_models/embed/) and
+`--no-enable-prefix-caching` (prefix caching is a completions-only
+optimization; irrelevant and safely disabled for pooling). An additional
+per-model `trust_remote_code: true` toggle renders `--trust-remote-code`
+when set — required for `nomic-ai/nomic-embed-text-v1.5`, which ships
+custom `NomicBertModel` modeling code on its HF repo.
+
+**Verification does NOT run `/v1/completions` against embedding-mode
+instances** (they don't serve that endpoint — a completions request 400s
+immediately). `tasks/verify.yml` splits `vllm_enabled_models` by `role` and
+runs the appropriate smoke test per group: completions models get the
+`/v1/completions` "capital of France" test; embedding models get a real
+`/v1/embeddings` POST with an `ansible.builtin.assert` on a non-empty
+`data[0].embedding` array (not just HTTP 200 — an empty/malformed vector
+would still 200).
+
+**Critical VRAM finding: co-resident completions + embedding vLLM processes
+need MORE headroom than either alone, and CUDA graph capture is the failure
+mode, not KV cache sizing.** Enabling `nomic-embed-text-v1.5` alongside the
+primary Qwen2.5-32B model at the role-default `gpu_memory_utilization: 0.95`
+crash-looped repeatedly:
+- First failure: `torch.OutOfMemoryError` during `capture_model()` (CUDA
+  graph capture) — KV cache sizing itself succeeded (14,720 tokens
+  allocated), but graph capture needed ~20MiB more than the 0.95 budget had
+  left once nomic's embedding process (814MiB actual, not the nominal
+  ~300MB estimate in the model roster table) claimed its share.
+- Fix attempt 1: added a per-model `enforce_eager: true` template branch
+  (`--enforce-eager` skips CUDA graph capture entirely) — this stopped the
+  graph-capture OOM but the combined processes still landed at only
+  ~847MiB genuinely free out of 24,576MiB, and both services crash-looped
+  6-7 times during warmup before finally stabilizing (each attempt leaves
+  transient VRAM that the next attempt fights over, extending time-to-stable
+  well past a single health-check retry window).
+- Fix attempt 2 (final, verified stable): lowered the primary model's
+  `gpu_memory_utilization` from 0.95 to **0.90** (host_vars override) in
+  addition to `enforce_eager: true`. Result: clean single-attempt start for
+  both services, `NRestarts=0`, ~2GB genuinely free (22,577MiB used /
+  24,576MiB total). Confirmed via `systemctl show <unit> -p NRestarts` after
+  a full stop/start cycle — 0.95 was NOT a fluke of Restart=always masking
+  the underlying fragility; 0.90 is a real, reproducible fix.
+- **Takeaway for future multi-process vLLM VRAM budgeting on this host:**
+  do not just check "does it eventually come up" — check `NRestarts` and
+  free VRAM headroom after a clean stop/start. A model that "works" after
+  6 crash-loop retries is not production-stable; the retries themselves are
+  evidence the utilization ceiling is too tight for the actual (not
+  nominal) footprint of co-resident processes.
+
+## Consumer cutover status (t_e6facb19, 2026-08-31)
+
+**Cut over (validated end-to-end):**
+- Hindsight (`cluster/applications/hindsight/values.yaml` +
+  `externalsecret.yaml`): `HINDSIGHT_API_LLM_BASE_URL` → vLLM `:8000`
+  (Qwen2.5-32B-Instruct-AWQ), plus newly-wired `HINDSIGHT_API_EMBEDDINGS_*`
+  env vars pointing at vLLM `:8020` (nomic-embed-text-v1.5). Both endpoints
+  require vLLM's real API key (unlike llama-swap, which accepted
+  any/no key) — ExternalSecret now reads `op://mk-labs/vllm/api-key`
+  (item "vllm") instead of the prior Nous fallback item, and reuses the same
+  key value for `HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY` (both vLLM
+  endpoints share one key file per `tasks/api-key.yml`).
+
+**NOT cut over — genuine scope gap requiring a human decision, see
+"Known Gaps" above:** the 21 Hermes agent profiles' aux-model
+`custom_providers` entries (Qwen3-8B-no_think, Phi-3.5-mini, Meta-Llama-3.1-8B,
+Qwen2.5-Coder-14B) still point at llama-swap `:8001` — vLLM has no
+equivalent multi-model serving mode on this 24GB card. llama-swap MUST stay
+up to serve these until that gap is resolved. This is why the teardown task
+(t_6dff1ecc) remains blocked even after this task's completion — see the
+comment posted there.
+
+- OpenViking (`cluster/platform/openviking/values.yaml`): still points at
+  llama-swap `:8001` (`Meta-Llama-3.1-8B-Instruct-Q4_K_M` VLM + nomic-embed
+  for dense embeddings). Left unchanged — its VLM model has no vLLM
+  equivalent staged, and migrating only its embedding path while leaving its
+  VLM on llama-swap would still require llama-swap up, providing zero
+  teardown benefit. Flagged, not touched, per the same aux-model gap above.
+
 
 ## Validation Log (2026-08-31, t_ca1af9fb)