astro-orbiter llama-server-router retired :8002 on 2026-08-18
(llama-swap now serves the OpenAI-compatible API on :8001).
OpenViking still pointed at the dead :8002, so every search/find
call failed: dense embedding -> Connection error -> 500 INTERNAL
server error on /api/v1/search/search and /api/v1/search/find.
Verified from the pod: :8002 = connection refused,
:8001/v1/embeddings (nomic-embed-text-v1.5) = 200 with vectors.
Repoints embedding.dense.api_base and vlm.api_base to :8001.
t_abc05db5
- vlm.model was 'llama3.1-8b' which doesn't exist on astro-orbiter's
/v1/models, causing every summarization call to 400 and endless
circuit-breaker retries. Correct id: Meta-Llama-3.1-8B-Instruct-Q4_K_M.
- embedding.max_input_tokens=1536 still let chunks through that actually
tokenized to 2000-2860 real tokens (estimator undercounts vs llama.cpp's
tokenizer by 1.35x-1.86x on this corpus). Lowered to 1024 for real margin
under the 2048 n_ctx ceiling.
astro-orbiter's llama.cpp router hard-caps nomic-embed-text-v1.5 effective
context at 2048 tokens regardless of ctx-size (known nomic-bert/RoPE limitation
in llama.cpp, not fixable server-side). OpenViking chunks observed at
2000-3400 tokens were tripping 400 exceed_context_size_error and endless
circuit-breaker re-enqueue for viking://temp/default/08140552_5f1c9e/homelab.tar/*.
Set embedding.max_input_tokens: 1536 (well under 2048) since OpenViking's
chunk-time token estimator uses a different tokenizer than llama.cpp's context
counter, so token counts won't match 1:1 - 1536 leaves ~25% headroom.
Approved by Ryan as lowest-risk mitigation (does not touch astro-orbiter/
llama.cpp serving config, which is War Machine's domain and already fixed
separately for the ubatch-size issue).
- Build/push image: the-seas.local.mk-labs.cloud/library/maelstrom-ui:v0.3.17-1
(upstream volcengine/openviking web-studio/, pinned to commit 3cd1d4e9)
- Deployment + Service serving the static SPA via nginx (reverse-proxies
/api, /health, /ready to openviking backend; /bot deliberately NOT proxied)
- Ingress at maelstrom.local.mk-labs.cloud (TLS via letsencrypt-internal)
- ExternalSecret wiring scoped maelstrom-ui-key from
op://mk-labs/openviking/maelstrom-ui-key into the pod env (MAELSTROM_UI_KEY)
Per approved plan: inbox/ryan/2026-08-14-maelstrom-ui-deployment-plan.md
Key mint + approval: system/inbox/agents/nick-fury/2026-08-14-maelstrom-ui-key-mint-complete.md
Ryan approval: inbox/ryan/2026-08-14-maelstrom-key-approval.md
The letsencrypt-internal ClusterIssuer does not exist on the cluster.
Only letsencrypt-prod and letsencrypt-staging are available.
Both use DNS-01 via Cloudflare for the mk-labs.cloud zone,
so they work for internal-only hosts with no public HTTP reachability.
Fixes: https://github.com/volcengine/openviking/issues/...
Closes: kanban task t_1c2cc2db
Bug 1: ExternalSecret referenced three separate 1Password items
(openviking-root-api-key, openviking-embedding-api-key, openviking-vlm-api-key)
but Ryan created ONE item 'openviking' with three fields inside.
Changed all remoteRef.key values to 'openviking' and corrected field property names.
Bug 2: values.yaml had two invalid embedding config fields:
- encoding_format: 'float' (not in upstream schema, removed)
- max_concurrent under embedding.dense (wrong nesting, moved to embedding level)
Verified against upstream chart schema at github.com/volcengine/openviking
Remove resource-level sync-wave annotations that caused ArgoCD deadlock.
The wave 8 annotation was meant for platform-level ordering (apps-of-apps)
but was incorrectly applied to individual resources within the Application.
This caused ArgoCD to apply Deployment (wave 0) before ExternalSecret (wave 8),
resulting in CreateContainerConfigError since the pod needed the secret first.
Changes:
- namespace.yaml: Remove sync-wave annotation, add clarifying comment
- externalsecret.yaml: Change sync-wave from 8 to -1 (must sync before Deployment at wave 0)
- application.yaml: Remove sync-wave annotation, document as platform-level only in comments
This ensures:
1. ExternalSecret syncs first (wave -1)
2. Deployment uses it immediately (implicit wave 0)
3. No deadlock
Task: t_3906c41a
- Changed source 1 from 'chart: deploy/helm/openviking' to 'path: deploy/helm/openviking'
- ArgoCD multi-source now correctly resolves the Helm chart from the git repo
- targetRevision: main now correctly refers to a git branch, not a chart version
- Fixes: invalid revision 'main': improper constraint error
Platform Knowledge Infrastructure pilot - context database for large
file trees, shared skills, and long-term logs to reduce agent token
consumption. Pilot scope: two corpora (hermes/ skills library,
personal/homelab/), two consumer profiles (Wong, Shuri) for before/after
token comparison.
- namespace.yaml: openviking namespace, sync-wave 8 (after Harbor wave 7)
- externalsecret.yaml: credentials from 1Password via onepassword-connect
ClusterSecretStore (Wong, t_32766900)
- values.yaml: Helm overrides - px-fa-direct-access storage (30Gi),
embedding (nomic-embed-text-v1.5) + VLM (Llama-3.1-8B) via astro-orbiter
router (:8002), internal-only ingress
- application.yaml: multi-source ArgoCD Application, Harbor pattern
(Peter Parker, t_eefdcc17 + reconciled in t_3e54efa8)
Prerequisites verified complete before this commit:
- nomic-embed-text-v1.5-Q4_K_M live on astro-orbiter router (War Machine,
t_34b96e83, commit ad70b34)
- All 3 1Password items provisioned (root/embedding/vlm api keys)
- Storage class corrected to px-fa-direct-access after live PV audit
showed pure-block/pure-file have zero provisioned volumes (t_77b3ff79)
- Dry-run validated against live cluster prior to commit
Constraint: vault (~/friday) remains canonical source of truth; OpenViking
index is a derived cache, rebuilt from vault source files.
Honcho/lincoln explicitly out of scope for this work.
- Stable udev device symlinks (/dev/jmri/nce, /dev/jmri/loconet, /dev/jmri/lcc)
- jmri-monitor: polls Leviton Decora Smart switch to start/stop JMRI automatically
- Quiet hours 1-10 AM (no polling)
- 30s off-delay before shutdown
- LCRR config cloned from Gitea (ssh://gitea.mk-labs.cloud:2221/rblundon/LCRR.git)
- ~/.jmri symlinked to LCRR repo for GitOps config management
- jmri-gui: X11 remote GUI access (PanelPro/DecoderPro) via ssh -X as jmri user
- Stops daemon, launches GUI, restarts daemon on exit if layout still on
- jmri user gets login shell + SSH key for GUI sessions
- Full JRE installed (openjdk-21-jre) for AWT/X11 support
- PaperMC 26.1.2 via itzg/minecraft-server:2026.7.0
- Namespace: minecraft, Service: journey-into-imagination
- TCP port 10182 (non-standard) via ingress-nginx tcp forwarding
- Pure Storage CSI PVC (pure-block, 50Gi) for world data
- World seed hardcoded: -5177989977648707969
- RCON password via ExternalSecret + 1Password Connect
- SleepMost v5.6.2 plugin for single-player sleep
- Whitelist off at launch, toggle-ready
- ExternalDNS annotations for internal Technitium record
- Manual steps: UniFi port forward WAN:10182→10.1.71.80:10182,
Cloudflare A record + SRV for journey-into-imagination.mk-labs.cloud
- Deploy Portworx Operator + CSI driver via ArgoCD
- Support both iSCSI block and NFS file storage from FlashArray
- Integrate with 1Password External Secrets for FlashArray credentials
- Include comprehensive deployment documentation and validation script
- Storage classes: pure-block (iSCSI) and pure-file (NFS)
- Talos Linux compatible with iSCSI/multipath configuration
Deployment name: connections → signal-cli-rest-api (what it IS)
Service name: connections (unchanged - Epcot-themed role)
Updated labels throughout deployment and service selector to match.
- Set namespace pod-security.kubernetes.io/enforce to baseline
- Remove restrictive container securityContext
- Allows signal-cli-rest-api container to run its entrypoint script
which requires user/group modification capabilities
- Add container securityContext to satisfy PodSecurity policy
- Set SIGNAL_CLI_UID/GID to 0 to disable user modification attempts
- Fixes CrashLoopBackOff due to groupmod permission denied
The signal-cli-rest-api container's entrypoint script requires
privileges to modify user/group settings. Removing securityContext
allows the container to run with its default settings.
- Deploy signal-cli-rest-api 0.85 for Hermes Signal notifications
- Replace broken astro-orbiter VM (10.1.71.130:8080) with K8s service
- ArgoCD-managed GitOps deployment in connections namespace
- NFS-backed persistent storage for signal-cli state
- Fixes UNREGISTERED_FAILURE affecting midday market cron job
- Epcot-themed service (communication pavilion concept)
Service endpoint: http://connections.connections.svc.cluster.local:8080
ExternalDNS v0.15.1 does not support --target as a CLI flag.
Remove the extraArgs stanza from external-dns-cloudflare values.yaml
and instead add the target annotation directly on the CouchDB ingress:
external-dns.alpha.kubernetes.io/target: ingress.mk-labs.cloud
This achieves the same result (Cloudflare CNAME -> ingress.mk-labs.cloud)
without crashing the controller.
- external-dns-cloudflare: add excludeDomains: [local.mk-labs.cloud] to stop
internal subdomain records from leaking to Cloudflare
- external-dns-cloudflare: replace hostname annotationFilter with opt-in model
(external-dns.alpha.kubernetes.io/public=true) so only explicitly tagged
services get public Cloudflare records
- external-dns-cloudflare: add extraArgs --target=lb.mk-labs.cloud
- external-dns (Technitium/rfc2136): add extraArgs
--target=lightning-lane.local.mk-labs.cloud for internal records
- couchdb: add external-dns.alpha.kubernetes.io/public: 'true' annotation —
first service to opt in to public DNS; will create communicore.mk-labs.cloud
pointing to lb.mk-labs.cloud via Cloudflare ExternalDNS
URGENT: Cloudflare was creating records for local.mk-labs.cloud hosts.
Ryan: manually delete any *.local.mk-labs.cloud records currently in Cloudflare
(look for communicore.local.mk-labs.cloud and any other local.* entries).
Deploy a second ExternalDNS instance targeting Cloudflare to manage
public DNS records in the mk-labs.cloud zone. The existing Technitium
(rfc2136) instance handling local.mk-labs.cloud is unchanged.
Components:
- application.yaml: ArgoCD Application, wave 6, namespace external-dns-cloudflare
- values.yaml: Cloudflare provider, domainFilters: mk-labs.cloud, txtOwnerId: fastpass
- externalsecret.yaml: ExternalSecret pulling CF_API_TOKEN from 1Password
PREREQUISITE (manual): Ryan must create the following in 1Password before
the ExternalSecret will sync:
Item name: cloudflare-external-dns
Field name: api-token
Value: Cloudflare API token with DNS Edit on mk-labs.cloud
Until then, the ExternalSecret will show SecretSyncedError — expected.
- Added config.json key to harbor-credentials ExternalSecret
This ensures kaniko can find the Docker auth config at /kaniko/.docker/config.json
(previously only .dockerconfigjson was present)
- Created test-app-build PipelineRun manifest for validation testing
- Successfully validated end-to-end pipeline:
✅ git-clone Task deployed and working
✅ kaniko-build Task deployed and working
✅ container-build Pipeline deployed and working
✅ Harbor authentication working with robot account
✅ Test image built and pushed: the-seas.local.mk-labs.cloud/library/test-app:v1.0.0
✅ Image digest: sha256:aa143f4a01795a1d307b711108ca0c89f36e00ea38fddb9d7b2febd5fffc46d7
Pipeline test results:
- PipelineRun: test-app-build-005 - SUCCEEDED
- fetch-repository TaskRun - SUCCEEDED
- build-and-push TaskRun - SUCCEEDED
Tekton CI/CD platform is now operational and ready for production workloads.
Updated README to reflect the full workflow:
1. Robot account creation via Job
2. Secret regeneration and capture
3. Storage in 1Password (harbor-robot-accounts item)
4. Automatic sync via ExternalSecrets to K8s
Creates two docker-registry secrets from 1Password:
1. harbor-tekton-robot - for Tekton CI/CD pipeline push/pull
2. harbor-pull-secret - for fastpass cluster image pulls
Both sync from 1Password item 'harbor-robot-accounts' with fields:
- tekton-builder-username / tekton-builder-password
- fastpass-cluster-username / fastpass-cluster-password
Credentials document placed in PKA inbox for manual 1Password entry.
Once stored, ESO will automatically sync and create the secrets.
Documents robot account management via GitOps:
- tekton-builder and fastpass-cluster robot accounts
- Manual Job execution (PostSync hooks don't work with multi-source)
- Credential retrieval and storage in 1Password
- ImagePullSecret creation for K8s clusters
DNS was pointing to Gateway (10.1.71.90) instead of NGINX Ingress (10.1.71.80)
after we removed the HTTPRoute manifests. This caused traffic to hit the
Gateway's wildcard cert and get 'no healthy upstream' errors.
Added external-dns annotations to Ingress to direct DNS to correct IP:
- external-dns.alpha.kubernetes.io/hostname: the-seas.local.mk-labs.cloud
- external-dns.alpha.kubernetes.io/target: 10.1.71.80
This will update Technitium DNS to point to NGINX Ingress Controller.
