1af645d272
Critical finding: flipping vllm_service_enabled/state=true/started and restarting llama-swap alongside it broke llama-swap's ability to load ANY of its own generative models -- every /v1/chat/completions request against Qwen3.8-27B-Q4_K_M or Qwen3-8B aux models failed with 'upstream command exited prematurely' (llama-server OOM at spawn, ~1.8GB free on this 24GB card once vLLM's ~22.8GB was claimed). Confirmed by direct A/B: same request 500s with vLLM running, 200s seconds after stopping it. This breaks 21 Hermes agent profiles' aux-model tasks (skills_hub, approval, mcp, title_generation, profile_describer, compression) plus OpenViking's VLM -- a far larger blast radius than Hindsight's single LLM endpoint. Reverted: - vllm_service_enabled/state back to role defaults (false/stopped) -- vLLM stays staged, startable for a brief validated shadow window, NOT safe to leave resident in production. - Hindsight's HINDSIGHT_API_LLM_BASE_URL back to llama-swap (astro-orbiter:8001, Qwen3.8-27B-Q4_K_M) and the API key secret source back to the Nous fallback item (pre-task state) -- the vLLM cutover, while functionally validated in isolation (health, /v1/chat/completions, and a live hindsight_retain+recall round-trip all succeeded), requires continuous vLLM availability which is now known to be unsafe on this card. Comment posted on t_6dff1ecc: teardown remains correctly blocked -- full cutover is not achievable within this card's VRAM budget as currently scoped. Needs a human decision on aux-model migration strategy (see roles/deploy-vllm README's 'Critical architectural finding' section) before any further progress.
90 lines
4.4 KiB
YAML
90 lines
4.4 KiB
YAML
# ============================================================================
|
|
# ExternalSecret - Hindsight Credentials
|
|
# Peter Parker, Phase C, t_6d47a360
|
|
#
|
|
# Syncs Hindsight credentials from the 1Password `mk-labs` vault (item
|
|
# "hindsight", UUID q6pwoohexogdyvitt7sm2xcusu).
|
|
# Pattern: openviking proven pattern (cluster/platform/openviking/externalsecret.yaml).
|
|
# Store: onepassword-connect ClusterSecretStore.
|
|
# Namespace: hindsight.
|
|
#
|
|
# Wave: -1 — must sync BEFORE the chart's Deployments (which default to wave 0)
|
|
# so the materialized Secret exists before the api / control-plane pods attempt
|
|
# envFrom / secretKeyRef injection.
|
|
#
|
|
# The materialized Secret (hindsight-credentials) must carry keys in TWO shapes
|
|
# (Phase B record: inbox/ryan/2026-08-24-hindsight-phase-b-secrets.md):
|
|
# 1. `postgres-password` (lowercase, hyphenated) — read by the chart's EXPLICIT
|
|
# secretKeyRef for POSTGRES_PASSWORD, NOT via envFrom. A naming error here
|
|
# silently drops the DB password (pod starts, then fails to connect).
|
|
# 2. `HINDSIGHT_API_LLM_API_KEY` / `HINDSIGHT_API_MCP_AUTH_TOKEN` — env-var-named,
|
|
# injected via envFrom (always runs when existingSecret is set).
|
|
#
|
|
# 1Password item "hindsight" fields (all CONCEALED):
|
|
# - postgres-password (32 chars, letters+digits only, URL-safe)
|
|
# - HINDSIGHT_API_MCP_AUTH_TOKEN (48 chars urlsafe bearer token)
|
|
# - HINDSIGHT_API_LLM_API_KEY (literal "local-placeholder" — astro-orbiter
|
|
# does not validate; must rotate if OpenRouter
|
|
# hosted fallback is ever enabled)
|
|
# ============================================================================
|
|
apiVersion: external-secrets.io/v1beta1
|
|
kind: ExternalSecret
|
|
metadata:
|
|
name: hindsight-credentials
|
|
namespace: hindsight
|
|
annotations:
|
|
# Wave -1: ensure the ExternalSecret syncs BEFORE the chart's Deployments
|
|
# (wave 0). Guarantees the Secret exists before the api / control-plane
|
|
# pods attempt envFrom / secretKeyRef injection.
|
|
argocd.argoproj.io/sync-wave: "-1"
|
|
description: "Phase C secrets for Hindsight deployment (1Password mk-labs item: hindsight)"
|
|
spec:
|
|
refreshInterval: "1h"
|
|
secretStoreRef:
|
|
kind: ClusterSecretStore
|
|
name: onepassword-connect
|
|
target:
|
|
name: hindsight-credentials
|
|
creationPolicy: Owner
|
|
template:
|
|
engineVersion: v2
|
|
data:
|
|
# LOWERCASE, HYPHENATED — read by the chart's explicit secretKeyRef for
|
|
# POSTGRES_PASSWORD. Must be exactly "postgres-password".
|
|
postgres-password: "{{ .postgresPassword }}"
|
|
# env-var-named keys injected via envFrom
|
|
HINDSIGHT_API_MCP_AUTH_TOKEN: "{{ .HINDSIGHT_API_MCP_AUTH_TOKEN }}"
|
|
HINDSIGHT_API_LLM_API_KEY: "{{ .HINDSIGHT_API_LLM_API_KEY }}"
|
|
# NOTE: no HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY here — Hindsight's
|
|
# embeddings stayed on its bundled local provider after the vLLM
|
|
# embeddings cutover attempt crash-looped the API on an embedding
|
|
# dimension mismatch (384 existing vs 768 nomic-embed). See
|
|
# values.yaml's api.env comment for the full incident writeup
|
|
# (t_e6facb19, 2026-08-31).
|
|
data:
|
|
# 1Password item "hindsight", field "postgres-password".
|
|
# letters+digits only / URL-safe: required because the chart interpolates the
|
|
# value RAW into HINDSIGHT_API_DATABASE_URL (no URL-encoding).
|
|
- secretKey: postgresPassword
|
|
remoteRef:
|
|
key: hindsight
|
|
property: postgres-password
|
|
# 1Password item "hindsight", field "HINDSIGHT_API_MCP_AUTH_TOKEN"
|
|
# (MCP bearer token, D6 — enables bearer-auth on the /mcp endpoint)
|
|
- secretKey: HINDSIGHT_API_MCP_AUTH_TOKEN
|
|
remoteRef:
|
|
key: hindsight
|
|
property: HINDSIGHT_API_MCP_AUTH_TOKEN
|
|
# 1Password item "nous" (vault mk-labs), field "api-key" (Ryan-provisioned,
|
|
# decision 4). REVERTED to this source (t_e6facb19, 2026-08-31) after the
|
|
# vLLM cutover attempt was reverted -- llama-swap doesn't validate the
|
|
# API key at all, so this value is effectively unused by the live LLM
|
|
# path, but restoring the original source keeps this file's intent
|
|
# honest (Nous fallback item, not vLLM's real auth key) until a real
|
|
# vLLM cutover is safe to attempt again. See values.yaml's api.env
|
|
# comment for the full incident writeup.
|
|
- secretKey: HINDSIGHT_API_LLM_API_KEY
|
|
remoteRef:
|
|
key: nous
|
|
property: api-key
|