9d6869ad9d
HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai pointed at vLLM's nomic-embed-text-v1.5 (768-dim) crash-looped hindsight-api: 'Cannot change embedding dimension from 384 to 768: memory_units table contains 1289 rows with embeddings.' Hindsight was never actually using astro-orbiter for embeddings (defaults to a bundled local 384-dim sentence-transformers model when the env var is unset) -- this was a genuine architecture assumption error, not a config typo. Re-embedding all existing memory data across ~20 agent banks is a destructive, irreversible operation requiring its own explicit, approved task -- not something to trigger as a side effect of an infra migration. Keeps the LLM cutover (astro-orbiter:8000, vLLM Qwen2.5-32B-Instruct-AWQ) which is safe and already validated.
90 lines
4.4 KiB
YAML
90 lines
4.4 KiB
YAML
# ============================================================================
|
|
# ExternalSecret - Hindsight Credentials
|
|
# Peter Parker, Phase C, t_6d47a360
|
|
#
|
|
# Syncs Hindsight credentials from the 1Password `mk-labs` vault (item
|
|
# "hindsight", UUID q6pwoohexogdyvitt7sm2xcusu).
|
|
# Pattern: openviking proven pattern (cluster/platform/openviking/externalsecret.yaml).
|
|
# Store: onepassword-connect ClusterSecretStore.
|
|
# Namespace: hindsight.
|
|
#
|
|
# Wave: -1 — must sync BEFORE the chart's Deployments (which default to wave 0)
|
|
# so the materialized Secret exists before the api / control-plane pods attempt
|
|
# envFrom / secretKeyRef injection.
|
|
#
|
|
# The materialized Secret (hindsight-credentials) must carry keys in TWO shapes
|
|
# (Phase B record: inbox/ryan/2026-08-24-hindsight-phase-b-secrets.md):
|
|
# 1. `postgres-password` (lowercase, hyphenated) — read by the chart's EXPLICIT
|
|
# secretKeyRef for POSTGRES_PASSWORD, NOT via envFrom. A naming error here
|
|
# silently drops the DB password (pod starts, then fails to connect).
|
|
# 2. `HINDSIGHT_API_LLM_API_KEY` / `HINDSIGHT_API_MCP_AUTH_TOKEN` — env-var-named,
|
|
# injected via envFrom (always runs when existingSecret is set).
|
|
#
|
|
# 1Password item "hindsight" fields (all CONCEALED):
|
|
# - postgres-password (32 chars, letters+digits only, URL-safe)
|
|
# - HINDSIGHT_API_MCP_AUTH_TOKEN (48 chars urlsafe bearer token)
|
|
# - HINDSIGHT_API_LLM_API_KEY (literal "local-placeholder" — astro-orbiter
|
|
# does not validate; must rotate if OpenRouter
|
|
# hosted fallback is ever enabled)
|
|
# ============================================================================
|
|
apiVersion: external-secrets.io/v1beta1
|
|
kind: ExternalSecret
|
|
metadata:
|
|
name: hindsight-credentials
|
|
namespace: hindsight
|
|
annotations:
|
|
# Wave -1: ensure the ExternalSecret syncs BEFORE the chart's Deployments
|
|
# (wave 0). Guarantees the Secret exists before the api / control-plane
|
|
# pods attempt envFrom / secretKeyRef injection.
|
|
argocd.argoproj.io/sync-wave: "-1"
|
|
description: "Phase C secrets for Hindsight deployment (1Password mk-labs item: hindsight)"
|
|
spec:
|
|
refreshInterval: "1h"
|
|
secretStoreRef:
|
|
kind: ClusterSecretStore
|
|
name: onepassword-connect
|
|
target:
|
|
name: hindsight-credentials
|
|
creationPolicy: Owner
|
|
template:
|
|
engineVersion: v2
|
|
data:
|
|
# LOWERCASE, HYPHENATED — read by the chart's explicit secretKeyRef for
|
|
# POSTGRES_PASSWORD. Must be exactly "postgres-password".
|
|
postgres-password: "{{ .postgresPassword }}"
|
|
# env-var-named keys injected via envFrom
|
|
HINDSIGHT_API_MCP_AUTH_TOKEN: "{{ .HINDSIGHT_API_MCP_AUTH_TOKEN }}"
|
|
HINDSIGHT_API_LLM_API_KEY: "{{ .HINDSIGHT_API_LLM_API_KEY }}"
|
|
# NOTE: no HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY here — Hindsight's
|
|
# embeddings stayed on its bundled local provider after the vLLM
|
|
# embeddings cutover attempt crash-looped the API on an embedding
|
|
# dimension mismatch (384 existing vs 768 nomic-embed). See
|
|
# values.yaml's api.env comment for the full incident writeup
|
|
# (t_e6facb19, 2026-08-31).
|
|
data:
|
|
# 1Password item "hindsight", field "postgres-password".
|
|
# letters+digits only / URL-safe: required because the chart interpolates the
|
|
# value RAW into HINDSIGHT_API_DATABASE_URL (no URL-encoding).
|
|
- secretKey: postgresPassword
|
|
remoteRef:
|
|
key: hindsight
|
|
property: postgres-password
|
|
# 1Password item "hindsight", field "HINDSIGHT_API_MCP_AUTH_TOKEN"
|
|
# (MCP bearer token, D6 — enables bearer-auth on the /mcp endpoint)
|
|
- secretKey: HINDSIGHT_API_MCP_AUTH_TOKEN
|
|
remoteRef:
|
|
key: hindsight
|
|
property: HINDSIGHT_API_MCP_AUTH_TOKEN
|
|
# Cutover to vLLM (t_e6facb19, 2026-08-31): astro-orbiter's llama-swap
|
|
# (which needed no real auth) is being retired for Hindsight's
|
|
# completions+embeddings roles. vLLM enforces a real API key — read the
|
|
# SAME key roles/deploy-vllm's api-key.yml phase writes to
|
|
# /etc/vllm/api-key.env on astro-orbiter, sourced from 1Password
|
|
# op://mk-labs/vllm/api-key (item "vllm", field "api-key", vault mk-labs).
|
|
# Replaces the prior "nous" item's api-key (Nous free-tier fallback,
|
|
# no longer the active LLM backend once this cutover lands).
|
|
- secretKey: HINDSIGHT_API_LLM_API_KEY
|
|
remoteRef:
|
|
key: vllm
|
|
property: api-key
|