hindsight: revert embeddings cutover after dimension-mismatch crash

HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai pointed at vLLM's
nomic-embed-text-v1.5 (768-dim) crash-looped hindsight-api:
'Cannot change embedding dimension from 384 to 768: memory_units
table contains 1289 rows with embeddings.' Hindsight was never
actually using astro-orbiter for embeddings (defaults to a bundled
local 384-dim sentence-transformers model when the env var is unset)
-- this was a genuine architecture assumption error, not a config
typo. Re-embedding all existing memory data across ~20 agent banks is
a destructive, irreversible operation requiring its own explicit,
approved task -- not something to trigger as a side effect of an
infra migration. Keeps the LLM cutover (astro-orbiter:8000, vLLM
Qwen2.5-32B-Instruct-AWQ) which is safe and already validated.
This commit is contained in:
Hermes Agent service account
2026-08-31 18:17:25 -05:00
parent 2cc9370f3d
commit 9d6869ad9d
2 changed files with 39 additions and 33 deletions

View File

@@ -55,15 +55,12 @@ spec:
# env-var-named keys injected via envFrom
HINDSIGHT_API_MCP_AUTH_TOKEN: "{{ .HINDSIGHT_API_MCP_AUTH_TOKEN }}"
HINDSIGHT_API_LLM_API_KEY: "{{ .HINDSIGHT_API_LLM_API_KEY }}"
# Cutover to vLLM (t_e6facb19, 2026-08-31): vLLM enforces its API key
# on every request (unlike llama-swap, which accepted any/no key) —
# confirmed empirically, a bad/missing key gets a real 401
# {"error":"Unauthorized"}. Both the completions endpoint (:8000) and
# the embeddings endpoint (:8020) are separate vLLM processes but
# share the SAME key (api-key.env is written once, read by both
# systemd units per roles/deploy-vllm/tasks/api-key.yml). Reuse
# HINDSIGHT_API_LLM_API_KEY's value for the embeddings key too.
HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY: "{{ .HINDSIGHT_API_LLM_API_KEY }}"
# NOTE: no HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY here — Hindsight's
# embeddings stayed on its bundled local provider after the vLLM
# embeddings cutover attempt crash-looped the API on an embedding
# dimension mismatch (384 existing vs 768 nomic-embed). See
# values.yaml's api.env comment for the full incident writeup
# (t_e6facb19, 2026-08-31).
data:
# 1Password item "hindsight", field "postgres-password".
# letters+digits only / URL-safe: required because the chart interpolates the