- Add comprehensive UPGRADES-AND-EXTENSIONS.md guide covering:
- System extensions via schematics and Image Factory
- Talos version upgrade procedures (control plane + workers)
- Kubernetes version upgrades
- Rolling upgrade best practices
- Troubleshooting common upgrade issues
- Add rolling-upgrade-workers.sh script for automated worker upgrades
- Includes safe wait times and confirmation prompts
- Production gateway on astro-orbiter VM working correctly
- K8s deployment ready but cannot complete Signal registration
- Signal servers reject WebSocket connections from K8s cluster network path
- Document migration procedure for when astro-orbiter is decommissioned
- Remove --account flag to let signal-cli auto-detect registered accounts
- Multi-account mode will find the registered +126****8840 account automatically
- Use bbernhard/signal-cli-rest-api:latest (known working image)
- Override command to run /usr/bin/signal-cli directly in daemon HTTP mode
- Bypasses REST API wrapper to expose native JSON-RPC endpoint at /api/v1/rpc
- Switch from non-existent GitLab registry image to official asamk/signal-cli:v0.14.5
- Maintain data path at /home/.local/share/signal-cli (where data was copied from astro-orbiter)
- Remove unnecessary command override, let entrypoint handle signal-cli execution
- Replace bbernhard/signal-cli-rest-api wrapper with native signal-cli 0.14.5
- Run signal-cli daemon in HTTP mode matching astro-orbiter working config
- Expose JSON-RPC API at /api/v1/rpc for Hermes gateway compatibility
- Switch health probes from HTTP /v1/health to TCP port check
- Maintain existing PVC mount path /home/.local/share/signal-cli
This fixes the missing /api/v1/rpc endpoint that Hermes requires for
Signal message delivery.
- Move manifests from cluster/platform/connections to cluster/applications/signal-cli
- Add Ingress for connections.local.mk-labs.cloud with cert-manager TLS
- Update ArgoCD application path to cluster/applications/signal-cli
- Configure letsencrypt-prod cluster issuer for automatic TLS certificates
This enables external HTTPS access to the Signal CLI REST API for Hermes
notifications with automatic certificate management.
Deployment name: connections → signal-cli-rest-api (what it IS)
Service name: connections (unchanged - Epcot-themed role)
Updated labels throughout deployment and service selector to match.
- Set namespace pod-security.kubernetes.io/enforce to baseline
- Remove restrictive container securityContext
- Allows signal-cli-rest-api container to run its entrypoint script
which requires user/group modification capabilities
- Add container securityContext to satisfy PodSecurity policy
- Set SIGNAL_CLI_UID/GID to 0 to disable user modification attempts
- Fixes CrashLoopBackOff due to groupmod permission denied
The signal-cli-rest-api container's entrypoint script requires
privileges to modify user/group settings. Removing securityContext
allows the container to run with its default settings.
- Deploy signal-cli-rest-api 0.85 for Hermes Signal notifications
- Replace broken astro-orbiter VM (10.1.71.130:8080) with K8s service
- ArgoCD-managed GitOps deployment in connections namespace
- NFS-backed persistent storage for signal-cli state
- Fixes UNREGISTERED_FAILURE affecting midday market cron job
- Epcot-themed service (communication pavilion concept)
Service endpoint: http://connections.connections.svc.cluster.local:8080
Per Tony's recommendation — eliminates the explicit include filter.
New manifests go in templates/ and are picked up automatically by ArgoCD.
No filter to update when adding future resources.
Moved: namespace.yaml, externalsecret.yaml, ingress-public.yaml -> templates/
ExternalDNS v0.15.1 does not support --target as a CLI flag.
Remove the extraArgs stanza from external-dns-cloudflare values.yaml
and instead add the target annotation directly on the CouchDB ingress:
external-dns.alpha.kubernetes.io/target: ingress.mk-labs.cloud
This achieves the same result (Cloudflare CNAME -> ingress.mk-labs.cloud)
without crashing the controller.
- external-dns-cloudflare: add excludeDomains: [local.mk-labs.cloud] to stop
internal subdomain records from leaking to Cloudflare
- external-dns-cloudflare: replace hostname annotationFilter with opt-in model
(external-dns.alpha.kubernetes.io/public=true) so only explicitly tagged
services get public Cloudflare records
- external-dns-cloudflare: add extraArgs --target=lb.mk-labs.cloud
- external-dns (Technitium/rfc2136): add extraArgs
--target=lightning-lane.local.mk-labs.cloud for internal records
- couchdb: add external-dns.alpha.kubernetes.io/public: 'true' annotation —
first service to opt in to public DNS; will create communicore.mk-labs.cloud
pointing to lb.mk-labs.cloud via Cloudflare ExternalDNS
URGENT: Cloudflare was creating records for local.mk-labs.cloud hosts.
Ryan: manually delete any *.local.mk-labs.cloud records currently in Cloudflare
(look for communicore.local.mk-labs.cloud and any other local.* entries).
Deploy a second ExternalDNS instance targeting Cloudflare to manage
public DNS records in the mk-labs.cloud zone. The existing Technitium
(rfc2136) instance handling local.mk-labs.cloud is unchanged.
Components:
- application.yaml: ArgoCD Application, wave 6, namespace external-dns-cloudflare
- values.yaml: Cloudflare provider, domainFilters: mk-labs.cloud, txtOwnerId: fastpass
- externalsecret.yaml: ExternalSecret pulling CF_API_TOKEN from 1Password
PREREQUISITE (manual): Ryan must create the following in 1Password before
the ExternalSecret will sync:
Item name: cloudflare-external-dns
Field name: api-token
Value: Cloudflare API token with DNS Edit on mk-labs.cloud
Until then, the ExternalSecret will show SecretSyncedError — expected.
Semaphore runs ansible-playbook from the repo root, so ansible.cfg in
ansible/ is never loaded. The env var is the reliable path.
Also persists the setting in group_vars so semaphore_configure re-runs
don't regress it.
Semaphore clones the repo and runs ansible-playbook from the working
directory, so ansible.cfg is loaded automatically. This is more
reliable than env vars inherited through podman exec subprocesses.
/opt/ansible-collections is bind-mounted into the container and
populated by the semaphore role's collections task.
Host dir was owned by root (0755), blocking writes from uid=1001 inside
the container. Set owner to semaphore_container_uid=1001 so podman exec
can write collections into the bind-mount.
- Removed :ro from volume mount — ansible-galaxy writes via podman exec
into the container, so the mount must be writable during role runs
- Fixed deprecated ANSIBLE_COLLECTIONS_PATHS -> ANSIBLE_COLLECTIONS_PATH
/home/semaphore/.ansible/ is owned by root after Podman creates the
bind-mount dir, so ansible-galaxy can't create sibling tmp dirs.
Mount to a neutral /opt/ansible-collections path and point Ansible
at it via ANSIBLE_COLLECTIONS_PATHS env var instead.
Binary lives inside the container at /opt/semaphore/apps/ansible/<ver>/venv/bin/.
Use podman exec to run the install, writing to /home/semaphore/.ansible/collections
which is bind-mounted from the host-side directory.
ansible-galaxy is not on the system PATH on figment — Semaphore manages
its own venv under /opt/semaphore/apps/ansible/<ver>/venv/bin/.
Discover the binary dynamically rather than relying on PATH.
- New tasks/collections.yml installs collections from defaults list
into /opt/semaphore/ansible-collections on the host
- semaphore.container.j2: bind-mounts that dir into the container at
/home/semaphore/.ansible/collections (read-only)
- defaults/main.yml: semaphore_collections_dir + semaphore_ansible_collections
list (containers.podman, effectivelywild.technitium_dns)
- main.yml: collections task wired in after semaphore.yml, before verify
- Collections survive container restarts/rebuilds without image changes
- Re-run with --tags collections to add new collections without full redeploy
Collection not installed in Semaphore's Ansible environment.
Direct HTTP API call to Technitium on :5380 is simpler, has no
collection dependency, and is naturally idempotent (add returns ok
on duplicate records).
- New entrypoints: gitea-ssh/:2221, jarvis-ssh/:10171
- Expose both ports from the Traefik container
- gitea.yml: TCP passthrough router -> 10.1.71.129:2221
- jarvis.yml: TCP passthrough router -> 10.1.71.131:22
- Both use HostSNI(*) — dedicated entrypoints, no TLS wrapping needed
- UniFi to forward both ports to lightning-lane
Playwright service doesn't expose a health endpoint at /, causing HTTP
probes to fail with 404. Switch to tcpSocket probes which simply verify
the port is listening. Service is already confirmed running on port 3000.
- Changed worker deployment probes from HTTP to TCP (port 3005)
* Worker liveness endpoint doesn't serve HTTP at '/' path
* TCP socket check more appropriate for background worker
* Resolves pod restart loop and readiness failures
- Corrected HTTPRoute gateway reference
* Changed from 'gateway' in 'default' namespace
* To 'fastpass-gateway' in 'gateway' namespace
* HTTPRoute now properly accepted by gateway
All 7 deployments Running and Ready. System operational.
- Added config.json key to harbor-credentials ExternalSecret
This ensures kaniko can find the Docker auth config at /kaniko/.docker/config.json
(previously only .dockerconfigjson was present)
- Created test-app-build PipelineRun manifest for validation testing
- Successfully validated end-to-end pipeline:
✅ git-clone Task deployed and working
✅ kaniko-build Task deployed and working
✅ container-build Pipeline deployed and working
✅ Harbor authentication working with robot account
✅ Test image built and pushed: the-seas.local.mk-labs.cloud/library/test-app:v1.0.0
✅ Image digest: sha256:aa143f4a01795a1d307b711108ca0c89f36e00ea38fddb9d7b2febd5fffc46d7
Pipeline test results:
- PipelineRun: test-app-build-005 - SUCCEEDED
- fetch-repository TaskRun - SUCCEEDED
- build-and-push TaskRun - SUCCEEDED
Tekton CI/CD platform is now operational and ready for production workloads.
