- minecraft-prometheus-exporter v3.1.2 plugin (port 9225)
- ServiceMonitor for Prometheus scraping
- Grafana dashboard ConfigMap (ID 20659, Minecraft server stats)
- metrics port added to Service and Deployment containerPorts
- PaperMC 26.1.2 via itzg/minecraft-server:2026.7.0
- Namespace: minecraft, Service: journey-into-imagination
- TCP port 10182 (non-standard) via ingress-nginx tcp forwarding
- Pure Storage CSI PVC (pure-block, 50Gi) for world data
- World seed hardcoded: -5177989977648707969
- RCON password via ExternalSecret + 1Password Connect
- SleepMost v5.6.2 plugin for single-player sleep
- Whitelist off at launch, toggle-ready
- ExternalDNS annotations for internal Technitium record
- Manual steps: UniFi port forward WAN:10182→10.1.71.80:10182,
Cloudflare A record + SRV for journey-into-imagination.mk-labs.cloud
- New ansible/roles/jmri role: installs OpenJDK 21 headless, creates
jmri service user, downloads JMRI 5.10, deploys JmriFaceless systemd unit
- Handles dialout group membership for serial device access
- Config restore task for post-reinstall recovery from GitHub backup
- host_vars/main-street-station: profile_id and serial device (TODO: fill in)
- Inventory: jmri_server group with main-street-station at 192.168.10.45
- Playbook: day1_deploy_jmri.yml (linux-baseline + jmri)
Fixes 413 Entity Too Large error in Obsidian LiveSync sync operations.
Applies to both internal (communicore.local) and public (communicore.mk-labs.cloud) ingress routes.
Successfully writes /system/etc/multipath.conf but cannot write to /etc
due to Talos read-only filesystem restrictions.
Attempts made:
- nsenter with sh/cat/ln - commands don't exist in Talos minimal env
- Mount /proc/1/root/etc - still read-only
- Bind mount - invalid argument
Blocker: Talos /etc is truly read-only post-boot. PX-CSI also fails
with same nsenter/command issues when trying to validate multipath.conf.
Next: Investigate PX-CSI configuration options or Talos machine config alternatives.
Use nsenter to write multipath.conf in host's mount namespace instead
of trying to write to /host/etc which is read-only in containers.
Talos mounts /etc as read-only in container namespaces but allows writes
in the host mount namespace. This fix uses nsenter to access PID 1's
mount namespace where /etc is writable.
Also removed unnecessary volumeMounts and volumes since we're using
nsenter instead of hostPath mounts.
Fixes: Init:Error - 'can't create /host/etc/multipath.conf: Read-only file system'
Comprehensive summary of diagnosis, fix, and recovery status.
Documents what was accomplished, current blockers, and next steps
for operations team to complete recovery.
Documents root cause analysis and recovery procedure for jungle-cruise
node failure after applying multipath.conf via machine.files.
Includes three recovery options depending on available credentials:
- Apply fixed config (requires talosctl + existing configs)
- Force reboot (quickest)
- Full regeneration (requires SOPS keys)
Removes /etc/multipath.conf from machine.files section which causes
jungle-cruise boot failure. This reverts the problematic change from
commit adc415e.
Root cause: Writing /etc/multipath.conf during early boot via machine.files
causes writeUserFiles to fail on read-only filesystem.
Solution: Use DaemonSet (iscsi-multipath-init.yaml) to write multipath.conf
after boot when filesystem is fully writable.
Fixes: jungle-cruise NotReady status (kubelet stopped posting)
PX-CSI requires /etc/multipath.conf to exist on nodes.
Adding Pure Storage FlashArray multipath configuration via Talos machine files.
This fixes node-plugin crash: '/etc/multipath.conf not found'
- Add dm_round_robin kernel module for Pure Storage multipath
- Uncomment and enable /etc/multipath.conf with Pure-specific settings
- Add apply-iscsi-config.sh script for rolling worker node updates
- Deploy Portworx Operator + CSI driver via ArgoCD
- Support both iSCSI block and NFS file storage from FlashArray
- Integrate with 1Password External Secrets for FlashArray credentials
- Include comprehensive deployment documentation and validation script
- Storage classes: pure-block (iSCSI) and pure-file (NFS)
- Talos Linux compatible with iSCSI/multipath configuration
- Add comprehensive UPGRADES-AND-EXTENSIONS.md guide covering:
- System extensions via schematics and Image Factory
- Talos version upgrade procedures (control plane + workers)
- Kubernetes version upgrades
- Rolling upgrade best practices
- Troubleshooting common upgrade issues
- Add rolling-upgrade-workers.sh script for automated worker upgrades
- Includes safe wait times and confirmation prompts
- Production gateway on astro-orbiter VM working correctly
- K8s deployment ready but cannot complete Signal registration
- Signal servers reject WebSocket connections from K8s cluster network path
- Document migration procedure for when astro-orbiter is decommissioned
- Remove --account flag to let signal-cli auto-detect registered accounts
- Multi-account mode will find the registered +126****8840 account automatically
- Use bbernhard/signal-cli-rest-api:latest (known working image)
- Override command to run /usr/bin/signal-cli directly in daemon HTTP mode
- Bypasses REST API wrapper to expose native JSON-RPC endpoint at /api/v1/rpc
- Switch from non-existent GitLab registry image to official asamk/signal-cli:v0.14.5
- Maintain data path at /home/.local/share/signal-cli (where data was copied from astro-orbiter)
- Remove unnecessary command override, let entrypoint handle signal-cli execution
- Replace bbernhard/signal-cli-rest-api wrapper with native signal-cli 0.14.5
- Run signal-cli daemon in HTTP mode matching astro-orbiter working config
- Expose JSON-RPC API at /api/v1/rpc for Hermes gateway compatibility
- Switch health probes from HTTP /v1/health to TCP port check
- Maintain existing PVC mount path /home/.local/share/signal-cli
This fixes the missing /api/v1/rpc endpoint that Hermes requires for
Signal message delivery.
- Move manifests from cluster/platform/connections to cluster/applications/signal-cli
- Add Ingress for connections.local.mk-labs.cloud with cert-manager TLS
- Update ArgoCD application path to cluster/applications/signal-cli
- Configure letsencrypt-prod cluster issuer for automatic TLS certificates
This enables external HTTPS access to the Signal CLI REST API for Hermes
notifications with automatic certificate management.
Deployment name: connections → signal-cli-rest-api (what it IS)
Service name: connections (unchanged - Epcot-themed role)
Updated labels throughout deployment and service selector to match.
- Set namespace pod-security.kubernetes.io/enforce to baseline
- Remove restrictive container securityContext
- Allows signal-cli-rest-api container to run its entrypoint script
which requires user/group modification capabilities
- Add container securityContext to satisfy PodSecurity policy
- Set SIGNAL_CLI_UID/GID to 0 to disable user modification attempts
- Fixes CrashLoopBackOff due to groupmod permission denied
The signal-cli-rest-api container's entrypoint script requires
privileges to modify user/group settings. Removing securityContext
allows the container to run with its default settings.
- Deploy signal-cli-rest-api 0.85 for Hermes Signal notifications
- Replace broken astro-orbiter VM (10.1.71.130:8080) with K8s service
- ArgoCD-managed GitOps deployment in connections namespace
- NFS-backed persistent storage for signal-cli state
- Fixes UNREGISTERED_FAILURE affecting midday market cron job
- Epcot-themed service (communication pavilion concept)
Service endpoint: http://connections.connections.svc.cluster.local:8080
Per Tony's recommendation — eliminates the explicit include filter.
New manifests go in templates/ and are picked up automatically by ArgoCD.
No filter to update when adding future resources.
Moved: namespace.yaml, externalsecret.yaml, ingress-public.yaml -> templates/
ExternalDNS v0.15.1 does not support --target as a CLI flag.
Remove the extraArgs stanza from external-dns-cloudflare values.yaml
and instead add the target annotation directly on the CouchDB ingress:
external-dns.alpha.kubernetes.io/target: ingress.mk-labs.cloud
This achieves the same result (Cloudflare CNAME -> ingress.mk-labs.cloud)
without crashing the controller.
- external-dns-cloudflare: add excludeDomains: [local.mk-labs.cloud] to stop
internal subdomain records from leaking to Cloudflare
- external-dns-cloudflare: replace hostname annotationFilter with opt-in model
(external-dns.alpha.kubernetes.io/public=true) so only explicitly tagged
services get public Cloudflare records
- external-dns-cloudflare: add extraArgs --target=lb.mk-labs.cloud
- external-dns (Technitium/rfc2136): add extraArgs
--target=lightning-lane.local.mk-labs.cloud for internal records
- couchdb: add external-dns.alpha.kubernetes.io/public: 'true' annotation —
first service to opt in to public DNS; will create communicore.mk-labs.cloud
pointing to lb.mk-labs.cloud via Cloudflare ExternalDNS
URGENT: Cloudflare was creating records for local.mk-labs.cloud hosts.
Ryan: manually delete any *.local.mk-labs.cloud records currently in Cloudflare
(look for communicore.local.mk-labs.cloud and any other local.* entries).
Deploy a second ExternalDNS instance targeting Cloudflare to manage
public DNS records in the mk-labs.cloud zone. The existing Technitium
(rfc2136) instance handling local.mk-labs.cloud is unchanged.
Components:
- application.yaml: ArgoCD Application, wave 6, namespace external-dns-cloudflare
- values.yaml: Cloudflare provider, domainFilters: mk-labs.cloud, txtOwnerId: fastpass
- externalsecret.yaml: ExternalSecret pulling CF_API_TOKEN from 1Password
PREREQUISITE (manual): Ryan must create the following in 1Password before
the ExternalSecret will sync:
Item name: cloudflare-external-dns
Field name: api-token
Value: Cloudflare API token with DNS Edit on mk-labs.cloud
Until then, the ExternalSecret will show SecretSyncedError — expected.
Semaphore runs ansible-playbook from the repo root, so ansible.cfg in
ansible/ is never loaded. The env var is the reliable path.
Also persists the setting in group_vars so semaphore_configure re-runs
don't regress it.
Semaphore clones the repo and runs ansible-playbook from the working
directory, so ansible.cfg is loaded automatically. This is more
reliable than env vars inherited through podman exec subprocesses.
/opt/ansible-collections is bind-mounted into the container and
populated by the semaphore role's collections task.
Host dir was owned by root (0755), blocking writes from uid=1001 inside
the container. Set owner to semaphore_container_uid=1001 so podman exec
can write collections into the bind-mount.
- Removed :ro from volume mount — ansible-galaxy writes via podman exec
into the container, so the mount must be writable during role runs
- Fixed deprecated ANSIBLE_COLLECTIONS_PATHS -> ANSIBLE_COLLECTIONS_PATH
/home/semaphore/.ansible/ is owned by root after Podman creates the
bind-mount dir, so ansible-galaxy can't create sibling tmp dirs.
Mount to a neutral /opt/ansible-collections path and point Ansible
at it via ANSIBLE_COLLECTIONS_PATHS env var instead.
Binary lives inside the container at /opt/semaphore/apps/ansible/<ver>/venv/bin/.
Use podman exec to run the install, writing to /home/semaphore/.ansible/collections
which is bind-mounted from the host-side directory.
ansible-galaxy is not on the system PATH on figment — Semaphore manages
its own venv under /opt/semaphore/apps/ansible/<ver>/venv/bin/.
Discover the binary dynamically rather than relying on PATH.
- New tasks/collections.yml installs collections from defaults list
into /opt/semaphore/ansible-collections on the host
- semaphore.container.j2: bind-mounts that dir into the container at
/home/semaphore/.ansible/collections (read-only)
- defaults/main.yml: semaphore_collections_dir + semaphore_ansible_collections
list (containers.podman, effectivelywild.technitium_dns)
- main.yml: collections task wired in after semaphore.yml, before verify
- Collections survive container restarts/rebuilds without image changes
- Re-run with --tags collections to add new collections without full redeploy
Collection not installed in Semaphore's Ansible environment.
Direct HTTP API call to Technitium on :5380 is simpler, has no
collection dependency, and is naturally idempotent (add returns ok
on duplicate records).
- New entrypoints: gitea-ssh/:2221, jarvis-ssh/:10171
- Expose both ports from the Traefik container
- gitea.yml: TCP passthrough router -> 10.1.71.129:2221
- jarvis.yml: TCP passthrough router -> 10.1.71.131:22
- Both use HostSNI(*) — dedicated entrypoints, no TLS wrapping needed
- UniFi to forward both ports to lightning-lane
Playwright service doesn't expose a health endpoint at /, causing HTTP
probes to fail with 404. Switch to tcpSocket probes which simply verify
the port is listening. Service is already confirmed running on port 3000.
- Changed worker deployment probes from HTTP to TCP (port 3005)
* Worker liveness endpoint doesn't serve HTTP at '/' path
* TCP socket check more appropriate for background worker
* Resolves pod restart loop and readiness failures
- Corrected HTTPRoute gateway reference
* Changed from 'gateway' in 'default' namespace
* To 'fastpass-gateway' in 'gateway' namespace
* HTTPRoute now properly accepted by gateway
All 7 deployments Running and Ready. System operational.
- Added config.json key to harbor-credentials ExternalSecret
This ensures kaniko can find the Docker auth config at /kaniko/.docker/config.json
(previously only .dockerconfigjson was present)
- Created test-app-build PipelineRun manifest for validation testing
- Successfully validated end-to-end pipeline:
✅ git-clone Task deployed and working
✅ kaniko-build Task deployed and working
✅ container-build Pipeline deployed and working
✅ Harbor authentication working with robot account
✅ Test image built and pushed: the-seas.local.mk-labs.cloud/library/test-app:v1.0.0
✅ Image digest: sha256:aa143f4a01795a1d307b711108ca0c89f36e00ea38fddb9d7b2febd5fffc46d7
Pipeline test results:
- PipelineRun: test-app-build-005 - SUCCEEDED
- fetch-repository TaskRun - SUCCEEDED
- build-and-push TaskRun - SUCCEEDED
Tekton CI/CD platform is now operational and ready for production workloads.
Replace letsencrypt-staging with letsencrypt-prod for trusted certificate.
Staging cert was causing connection resets due to certificate validation failures.
Change CouchDB ingress hostname from communicore.mk-labs.cloud to
communicore.local.mk-labs.cloud to align with External-DNS configuration.
CouchDB is an internal service and should use the .local.mk-labs.cloud
zone managed by Technitium DNS, not the public .mk-labs.cloud zone
managed by Cloudflare.
This ensures External-DNS will properly create the A record in the
internal DNS server.
Updated README to reflect the full workflow:
1. Robot account creation via Job
2. Secret regeneration and capture
3. Storage in 1Password (harbor-robot-accounts item)
4. Automatic sync via ExternalSecrets to K8s
Creates two docker-registry secrets from 1Password:
1. harbor-tekton-robot - for Tekton CI/CD pipeline push/pull
2. harbor-pull-secret - for fastpass cluster image pulls
Both sync from 1Password item 'harbor-robot-accounts' with fields:
- tekton-builder-username / tekton-builder-password
- fastpass-cluster-username / fastpass-cluster-password
Credentials document placed in PKA inbox for manual 1Password entry.
Once stored, ESO will automatically sync and create the secrets.
Documents robot account management via GitOps:
- tekton-builder and fastpass-cluster robot accounts
- Manual Job execution (PostSync hooks don't work with multi-source)
- Credential retrieval and storage in 1Password
- ImagePullSecret creation for K8s clusters
DNS was pointing to Gateway (10.1.71.90) instead of NGINX Ingress (10.1.71.80)
after we removed the HTTPRoute manifests. This caused traffic to hit the
Gateway's wildcard cert and get 'no healthy upstream' errors.
Added external-dns annotations to Ingress to direct DNS to correct IP:
- external-dns.alpha.kubernetes.io/hostname: the-seas.local.mk-labs.cloud
- external-dns.alpha.kubernetes.io/target: 10.1.71.80
This will update Technitium DNS to point to NGINX Ingress Controller.
