astro-orbiter llama-server-router retired :8002 on 2026-08-18
(llama-swap now serves the OpenAI-compatible API on :8001).
OpenViking still pointed at the dead :8002, so every search/find
call failed: dense embedding -> Connection error -> 500 INTERNAL
server error on /api/v1/search/search and /api/v1/search/find.
Verified from the pod: :8002 = connection refused,
:8001/v1/embeddings (nomic-embed-text-v1.5) = 200 with vectors.
Repoints embedding.dense.api_base and vlm.api_base to :8001.
t_abc05db5
- vlm.model was 'llama3.1-8b' which doesn't exist on astro-orbiter's
/v1/models, causing every summarization call to 400 and endless
circuit-breaker retries. Correct id: Meta-Llama-3.1-8B-Instruct-Q4_K_M.
- embedding.max_input_tokens=1536 still let chunks through that actually
tokenized to 2000-2860 real tokens (estimator undercounts vs llama.cpp's
tokenizer by 1.35x-1.86x on this corpus). Lowered to 1024 for real margin
under the 2048 n_ctx ceiling.
astro-orbiter's llama.cpp router hard-caps nomic-embed-text-v1.5 effective
context at 2048 tokens regardless of ctx-size (known nomic-bert/RoPE limitation
in llama.cpp, not fixable server-side). OpenViking chunks observed at
2000-3400 tokens were tripping 400 exceed_context_size_error and endless
circuit-breaker re-enqueue for viking://temp/default/08140552_5f1c9e/homelab.tar/*.
Set embedding.max_input_tokens: 1536 (well under 2048) since OpenViking's
chunk-time token estimator uses a different tokenizer than llama.cpp's context
counter, so token counts won't match 1:1 - 1536 leaves ~25% headroom.
Approved by Ryan as lowest-risk mitigation (does not touch astro-orbiter/
llama.cpp serving config, which is War Machine's domain and already fixed
separately for the ubatch-size issue).
- Build/push image: the-seas.local.mk-labs.cloud/library/maelstrom-ui:v0.3.17-1
(upstream volcengine/openviking web-studio/, pinned to commit 3cd1d4e9)
- Deployment + Service serving the static SPA via nginx (reverse-proxies
/api, /health, /ready to openviking backend; /bot deliberately NOT proxied)
- Ingress at maelstrom.local.mk-labs.cloud (TLS via letsencrypt-internal)
- ExternalSecret wiring scoped maelstrom-ui-key from
op://mk-labs/openviking/maelstrom-ui-key into the pod env (MAELSTROM_UI_KEY)
Per approved plan: inbox/ryan/2026-08-14-maelstrom-ui-deployment-plan.md
Key mint + approval: system/inbox/agents/nick-fury/2026-08-14-maelstrom-ui-key-mint-complete.md
Ryan approval: inbox/ryan/2026-08-14-maelstrom-key-approval.md
The letsencrypt-internal ClusterIssuer does not exist on the cluster.
Only letsencrypt-prod and letsencrypt-staging are available.
Both use DNS-01 via Cloudflare for the mk-labs.cloud zone,
so they work for internal-only hosts with no public HTTP reachability.
Fixes: https://github.com/volcengine/openviking/issues/...
Closes: kanban task t_1c2cc2db
Bug 1: ExternalSecret referenced three separate 1Password items
(openviking-root-api-key, openviking-embedding-api-key, openviking-vlm-api-key)
but Ryan created ONE item 'openviking' with three fields inside.
Changed all remoteRef.key values to 'openviking' and corrected field property names.
Bug 2: values.yaml had two invalid embedding config fields:
- encoding_format: 'float' (not in upstream schema, removed)
- max_concurrent under embedding.dense (wrong nesting, moved to embedding level)
Verified against upstream chart schema at github.com/volcengine/openviking
Remove resource-level sync-wave annotations that caused ArgoCD deadlock.
The wave 8 annotation was meant for platform-level ordering (apps-of-apps)
but was incorrectly applied to individual resources within the Application.
This caused ArgoCD to apply Deployment (wave 0) before ExternalSecret (wave 8),
resulting in CreateContainerConfigError since the pod needed the secret first.
Changes:
- namespace.yaml: Remove sync-wave annotation, add clarifying comment
- externalsecret.yaml: Change sync-wave from 8 to -1 (must sync before Deployment at wave 0)
- application.yaml: Remove sync-wave annotation, document as platform-level only in comments
This ensures:
1. ExternalSecret syncs first (wave -1)
2. Deployment uses it immediately (implicit wave 0)
3. No deadlock
Task: t_3906c41a
- Changed source 1 from 'chart: deploy/helm/openviking' to 'path: deploy/helm/openviking'
- ArgoCD multi-source now correctly resolves the Helm chart from the git repo
- targetRevision: main now correctly refers to a git branch, not a chart version
- Fixes: invalid revision 'main': improper constraint error
Platform Knowledge Infrastructure pilot - context database for large
file trees, shared skills, and long-term logs to reduce agent token
consumption. Pilot scope: two corpora (hermes/ skills library,
personal/homelab/), two consumer profiles (Wong, Shuri) for before/after
token comparison.
- namespace.yaml: openviking namespace, sync-wave 8 (after Harbor wave 7)
- externalsecret.yaml: credentials from 1Password via onepassword-connect
ClusterSecretStore (Wong, t_32766900)
- values.yaml: Helm overrides - px-fa-direct-access storage (30Gi),
embedding (nomic-embed-text-v1.5) + VLM (Llama-3.1-8B) via astro-orbiter
router (:8002), internal-only ingress
- application.yaml: multi-source ArgoCD Application, Harbor pattern
(Peter Parker, t_eefdcc17 + reconciled in t_3e54efa8)
Prerequisites verified complete before this commit:
- nomic-embed-text-v1.5-Q4_K_M live on astro-orbiter router (War Machine,
t_34b96e83, commit ad70b34)
- All 3 1Password items provisioned (root/embedding/vlm api keys)
- Storage class corrected to px-fa-direct-access after live PV audit
showed pure-block/pure-file have zero provisioned volumes (t_77b3ff79)
- Dry-run validated against live cluster prior to commit
Constraint: vault (~/friday) remains canonical source of truth; OpenViking
index is a derived cache, rebuilt from vault source files.
Honcho/lincoln explicitly out of scope for this work.
