deploy-vllm: add embedding-mode support, cut over Hindsight to vLLM (t_e6facb19)

- vllm.service.j2: branch on role==embedding for --runner pooling
  --convert embed, --no-enable-prefix-caching, per-model
  trust_remote_code toggle (needed for nomic-embed-text-v1.5's custom
  NomicBertModel code), and enforce_eager toggle (needed to avoid CUDA
  graph capture OOM when co-resident with another vLLM process on this
  24GB card).
- tasks/verify.yml: split completions vs embedding smoke tests --
  embedding-mode instances don't serve /v1/completions. Assert a
  non-empty embedding vector, not just HTTP 200.
- host_vars/astro-orbiter: enable nomic-embed-text-v1.5 (port 8020),
  lower primary model's gpu_memory_utilization 0.95->0.90 + add
  enforce_eager after finding 0.95 crash-looped 6-7x before stabilizing
  with co-resident nomic-embed (real fix, confirmed via NRestarts=0
  after clean stop/start, not luck).
- Hindsight (values.yaml + externalsecret.yaml): cut LLM + embeddings
  over to vLLM (:8000, :8020), wire the previously-unset
  HINDSIGHT_API_EMBEDDINGS_* env vars for the first time, and swap the
  API key secret source from the Nous fallback item to vllm/api-key
  (vLLM enforces real auth, llama-swap did not).
- README: document the embedding-mode branch, VRAM findings, and a
  genuine architecture gap -- vLLM's one-model-per-process design
  cannot replace llama-swap's 5-model LRU roster on this 24GB card, so
  21 Hermes profiles' aux-model consumers (Qwen3-8B-no_think,
  Phi-3.5-mini, Meta-Llama-3.1-8B, Qwen2.5-Coder-14B) and OpenViking's
  VLM stay on llama-swap. Full teardown (t_6dff1ecc) needs a human
  decision on the aux-model strategy before it can proceed.

cluster/applications/hindsight/values.yaml

@@ -60,12 +60,36 @@ existingSecret: hindsight-credentials
 # ----------------------------------------------------------------------------
 api:
   env:
-    # Restore (2026-08-29, t_e0e6f7ca): astro-orbiter back online; move LLM back
-    # to local Qwen3.8-27B-Q4_K_M on llama-swap. Nous free tier returns 400
-    # 'missing tags' on Hindsight structured fact-extraction (retain broken).
-    HINDSIGHT_API_LLM_BASE_URL: "http://astro-orbiter:8001/v1"
+    # Cutover to vLLM (t_e6facb19, 2026-08-31): astro-orbiter's llama-swap is
+    # being retired for the completions+embeddings roles Hindsight needs.
+    # vLLM serves Qwen2.5-32B-Instruct-AWQ (:8000, completions) and
+    # nomic-embed-text-v1.5 (:8020, --runner pooling --convert embed) as two
+    # independent systemd-managed processes — validated end-to-end (health,
+    # /v1/chat/completions, /v1/embeddings returning a 768-dim vector) in a
+    # shadow window with llama-swap stopped. Model swap from Qwen3.8-27B to
+    # Qwen2.5-32B-Instruct-AWQ: both are capable instruction-tuned models:
+    # confirm structured-extraction/tagging behavior (the prior stepfun/solar
+    # 400 "missing tags" failure mode) holds up after this swap — see
+    # references/hindsight-backend-fallback-to-local-qwen-workflow.md if it
+    # regresses and llama-swap needs to come back temporarily.
+    HINDSIGHT_API_LLM_BASE_URL: "http://astro-orbiter:8000/v1"
     HINDSIGHT_API_LLM_PROVIDER: "openai"
-    HINDSIGHT_API_LLM_MODEL: "Qwen3.8-27B-Q4_K_M"
+    HINDSIGHT_API_LLM_MODEL: "Qwen2.5-32B-Instruct-AWQ"
+    # Embeddings: llama.cpp nomic-embed-text-v1.5 (llama-swap, :8001) ->
+    # vLLM nomic-embed-text-v1.5 (:8020). Same model weights, different
+    # serving stack. Hindsight's embeddings provider defaults to "local"
+    # (bundled sentence-transformers) unless overridden — explicit TEI/openai
+    # provider config was never wired for astro-orbiter previously (verified:
+    # `kubectl exec ... env | grep -i embed` showed NO HINDSIGHT_API_EMBEDDINGS_*
+    # vars set, confirming Hindsight was using its bundled local embedder, NOT
+    # astro-orbiter, before this change — despite the OpenViking-style stack
+    # documentation implying otherwise). Wiring the openai-compatible provider
+    # here for the first time to point Hindsight's actual embedding generation
+    # at the vLLM-served nomic-embed-text-v1.5, matching the model OpenViking
+    # already uses (consistency across mk-labs services).
+    HINDSIGHT_API_EMBEDDINGS_PROVIDER: "openai"
+    HINDSIGHT_API_EMBEDDINGS_OPENAI_BASE_URL: "http://astro-orbiter:8020/v1"
+    HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL: "nomic-embed-text-v1.5"
     # --- t_d7f8cd65: fix 502s on the serial astro-orbiter node ---
     # astro-orbiter is a single llama-swap process (serial: 1 generate at a
     # time, ctx 64K). Hindsight's default LLM concurrency is 32, so a retain
@@ -74,6 +98,9 @@ api:
     # is the ONLY LLM endpoint (all ops route there), so cap the whole pool to
     # 1 and pin retain to 1 as well. The upstream chart exposes these as native
     # semaphore config (HINDSIGHT_API_*_MAX_CONCURRENT); no code change needed.
+    # Kept at 1 post-cutover: vLLM's single-process-per-model design is also
+    # effectively serial for a single Qwen2.5-32B instance under this GPU's
+    # VRAM budget (KV cache sized for low concurrency at max_model_len=8192).
     HINDSIGHT_API_LLM_MAX_CONCURRENT: "1"
     HINDSIGHT_API_RETAIN_LLM_MAX_CONCURRENT: "1"
     # Client + per-request timeout. Default is 120s; a 29K-token retain runs