hindsight: revert embeddings cutover after dimension-mismatch crash

HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai pointed at vLLM's
nomic-embed-text-v1.5 (768-dim) crash-looped hindsight-api:
'Cannot change embedding dimension from 384 to 768: memory_units
table contains 1289 rows with embeddings.' Hindsight was never
actually using astro-orbiter for embeddings (defaults to a bundled
local 384-dim sentence-transformers model when the env var is unset)
-- this was a genuine architecture assumption error, not a config
typo. Re-embedding all existing memory data across ~20 agent banks is
a destructive, irreversible operation requiring its own explicit,
approved task -- not something to trigger as a side effect of an
infra migration. Keeps the LLM cutover (astro-orbiter:8000, vLLM
Qwen2.5-32B-Instruct-AWQ) which is safe and already validated.

cluster/applications/hindsight/externalsecret.yaml

@@ -55,15 +55,12 @@ spec:
         # env-var-named keys injected via envFrom
         HINDSIGHT_API_MCP_AUTH_TOKEN: "{{ .HINDSIGHT_API_MCP_AUTH_TOKEN }}"
         HINDSIGHT_API_LLM_API_KEY: "{{ .HINDSIGHT_API_LLM_API_KEY }}"
-        # Cutover to vLLM (t_e6facb19, 2026-08-31): vLLM enforces its API key
+        # NOTE: no HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY here — Hindsight's
-        # on every request (unlike llama-swap, which accepted any/no key) —
+        # embeddings stayed on its bundled local provider after the vLLM
-        # confirmed empirically, a bad/missing key gets a real 401
+        # embeddings cutover attempt crash-looped the API on an embedding
-        # {"error":"Unauthorized"}. Both the completions endpoint (:8000) and
+        # dimension mismatch (384 existing vs 768 nomic-embed). See
-        # the embeddings endpoint (:8020) are separate vLLM processes but
+        # values.yaml's api.env comment for the full incident writeup
-        # share the SAME key (api-key.env is written once, read by both
+        # (t_e6facb19, 2026-08-31).
-        # systemd units per roles/deploy-vllm/tasks/api-key.yml). Reuse
-        # HINDSIGHT_API_LLM_API_KEY's value for the embeddings key too.
-        HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY: "{{ .HINDSIGHT_API_LLM_API_KEY }}"
   data:
     # 1Password item "hindsight", field "postgres-password".
     # letters+digits only / URL-safe: required because the chart interpolates the

cluster/applications/hindsight/values.yaml

@@ -61,35 +61,44 @@ existingSecret: hindsight-credentials
 api:
   env:
     # Cutover to vLLM (t_e6facb19, 2026-08-31): astro-orbiter's llama-swap is
-    # being retired for the completions+embeddings roles Hindsight needs.
+    # being retired for the completions role Hindsight needs. vLLM serves
-    # vLLM serves Qwen2.5-32B-Instruct-AWQ (:8000, completions) and
+    # Qwen2.5-32B-Instruct-AWQ (:8000, completions) — validated end-to-end
-    # nomic-embed-text-v1.5 (:8020, --runner pooling --convert embed) as two
+    # (health, /v1/chat/completions) in a shadow window with llama-swap
-    # independent systemd-managed processes — validated end-to-end (health,
+    # stopped. Model swap from Qwen3.8-27B to Qwen2.5-32B-Instruct-AWQ: both
-    # /v1/chat/completions, /v1/embeddings returning a 768-dim vector) in a
+    # are capable instruction-tuned models; confirm structured-extraction/
-    # shadow window with llama-swap stopped. Model swap from Qwen3.8-27B to
+    # tagging behavior (the prior stepfun/solar 400 "missing tags" failure
-    # Qwen2.5-32B-Instruct-AWQ: both are capable instruction-tuned models:
+    # mode) holds up after this swap — see
-    # confirm structured-extraction/tagging behavior (the prior stepfun/solar
-    # 400 "missing tags" failure mode) holds up after this swap — see
     # references/hindsight-backend-fallback-to-local-qwen-workflow.md if it
     # regresses and llama-swap needs to come back temporarily.
     HINDSIGHT_API_LLM_BASE_URL: "http://astro-orbiter:8000/v1"
     HINDSIGHT_API_LLM_PROVIDER: "openai"
     HINDSIGHT_API_LLM_MODEL: "Qwen2.5-32B-Instruct-AWQ"
-    # Embeddings: llama.cpp nomic-embed-text-v1.5 (llama-swap, :8001) ->
+    # DO NOT set HINDSIGHT_API_EMBEDDINGS_* here (t_e6facb19, 2026-08-31
-    # vLLM nomic-embed-text-v1.5 (:8020). Same model weights, different
+    # attempted this, reverted after a production incident — see below).
-    # serving stack. Hindsight's embeddings provider defaults to "local"
+    #
-    # (bundled sentence-transformers) unless overridden — explicit TEI/openai
+    # DISCOVERY: Hindsight's embeddings provider was NEVER pointed at
-    # provider config was never wired for astro-orbiter previously (verified:
+    # astro-orbiter. It defaults to "local" (bundled sentence-transformers,
-    # `kubectl exec ... env | grep -i embed` showed NO HINDSIGHT_API_EMBEDDINGS_*
+    # BAAI/bge-small-en-v1.5, 384 dimensions) whenever
-    # vars set, confirming Hindsight was using its bundled local embedder, NOT
+    # HINDSIGHT_API_EMBEDDINGS_PROVIDER is unset — verified via
-    # astro-orbiter, before this change — despite the OpenViking-style stack
+    # `kubectl exec ... env | grep -i embed` showing NO
-    # documentation implying otherwise). Wiring the openai-compatible provider
+    # HINDSIGHT_API_EMBEDDINGS_* vars in the live pod, despite this file's
-    # here for the first time to point Hindsight's actual embedding generation
+    # LLM section referencing astro-orbiter for years. The nomic-embed-
-    # at the vLLM-served nomic-embed-text-v1.5, matching the model OpenViking
+    # text-v1.5 model documented across mk-labs skills as "Hindsight's
-    # already uses (consistency across mk-labs services).
+    # embedding model" was OpenViking's embedding model, not Hindsight's.
-    HINDSIGHT_API_EMBEDDINGS_PROVIDER: "openai"
+    #
-    HINDSIGHT_API_EMBEDDINGS_OPENAI_BASE_URL: "http://astro-orbiter:8020/v1"
+    # INCIDENT: pointing HINDSIGHT_API_EMBEDDINGS_PROVIDER at vLLM's
-    HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL: "nomic-embed-text-v1.5"
+    # nomic-embed-text-v1.5 (768 dimensions) crash-looped hindsight-api on
+    # rollout: `RuntimeError: Cannot change embedding dimension from 384 to
+    # 768: memory_units table contains 1289 rows with embeddings.` The
+    # migration path (`ensure_embedding_dimension` in migrations.py) refuses
+    # a live dimension change without either re-embedding everything or
+    # deleting all existing memory_units rows across every bank (jarvis,
+    # hermes, war-machine, and ~18 other agent banks) — a destructive,
+    # irreversible operation requiring explicit human approval, not
+    # something to do as a side effect of an infra migration task. Reverted
+    # immediately; Hindsight keeps its bundled local embedder (384-dim,
+    # unchanged, zero data risk) until a deliberate, approved re-embedding
+    # migration is planned as its own task.
     # --- t_d7f8cd65: fix 502s on the serial astro-orbiter node ---
     # astro-orbiter is a single llama-swap process (serial: 1 generate at a
     # time, ctx 64K). Hindsight's default LLM concurrency is 32, so a retain