- Production gateway on astro-orbiter VM working correctly
- K8s deployment ready but cannot complete Signal registration
- Signal servers reject WebSocket connections from K8s cluster network path
- Document migration procedure for when astro-orbiter is decommissioned
- Remove --account flag to let signal-cli auto-detect registered accounts
- Multi-account mode will find the registered +126****8840 account automatically
- Use bbernhard/signal-cli-rest-api:latest (known working image)
- Override command to run /usr/bin/signal-cli directly in daemon HTTP mode
- Bypasses REST API wrapper to expose native JSON-RPC endpoint at /api/v1/rpc
- Switch from non-existent GitLab registry image to official asamk/signal-cli:v0.14.5
- Maintain data path at /home/.local/share/signal-cli (where data was copied from astro-orbiter)
- Remove unnecessary command override, let entrypoint handle signal-cli execution
- Replace bbernhard/signal-cli-rest-api wrapper with native signal-cli 0.14.5
- Run signal-cli daemon in HTTP mode matching astro-orbiter working config
- Expose JSON-RPC API at /api/v1/rpc for Hermes gateway compatibility
- Switch health probes from HTTP /v1/health to TCP port check
- Maintain existing PVC mount path /home/.local/share/signal-cli
This fixes the missing /api/v1/rpc endpoint that Hermes requires for
Signal message delivery.
- Move manifests from cluster/platform/connections to cluster/applications/signal-cli
- Add Ingress for connections.local.mk-labs.cloud with cert-manager TLS
- Update ArgoCD application path to cluster/applications/signal-cli
- Configure letsencrypt-prod cluster issuer for automatic TLS certificates
This enables external HTTPS access to the Signal CLI REST API for Hermes
notifications with automatic certificate management.
Per Tony's recommendation — eliminates the explicit include filter.
New manifests go in templates/ and are picked up automatically by ArgoCD.
No filter to update when adding future resources.
Moved: namespace.yaml, externalsecret.yaml, ingress-public.yaml -> templates/
ExternalDNS v0.15.1 does not support --target as a CLI flag.
Remove the extraArgs stanza from external-dns-cloudflare values.yaml
and instead add the target annotation directly on the CouchDB ingress:
external-dns.alpha.kubernetes.io/target: ingress.mk-labs.cloud
This achieves the same result (Cloudflare CNAME -> ingress.mk-labs.cloud)
without crashing the controller.
- external-dns-cloudflare: add excludeDomains: [local.mk-labs.cloud] to stop
internal subdomain records from leaking to Cloudflare
- external-dns-cloudflare: replace hostname annotationFilter with opt-in model
(external-dns.alpha.kubernetes.io/public=true) so only explicitly tagged
services get public Cloudflare records
- external-dns-cloudflare: add extraArgs --target=lb.mk-labs.cloud
- external-dns (Technitium/rfc2136): add extraArgs
--target=lightning-lane.local.mk-labs.cloud for internal records
- couchdb: add external-dns.alpha.kubernetes.io/public: 'true' annotation —
first service to opt in to public DNS; will create communicore.mk-labs.cloud
pointing to lb.mk-labs.cloud via Cloudflare ExternalDNS
URGENT: Cloudflare was creating records for local.mk-labs.cloud hosts.
Ryan: manually delete any *.local.mk-labs.cloud records currently in Cloudflare
(look for communicore.local.mk-labs.cloud and any other local.* entries).
Playwright service doesn't expose a health endpoint at /, causing HTTP
probes to fail with 404. Switch to tcpSocket probes which simply verify
the port is listening. Service is already confirmed running on port 3000.
- Changed worker deployment probes from HTTP to TCP (port 3005)
* Worker liveness endpoint doesn't serve HTTP at '/' path
* TCP socket check more appropriate for background worker
* Resolves pod restart loop and readiness failures
- Corrected HTTPRoute gateway reference
* Changed from 'gateway' in 'default' namespace
* To 'fastpass-gateway' in 'gateway' namespace
* HTTPRoute now properly accepted by gateway
All 7 deployments Running and Ready. System operational.
Replace letsencrypt-staging with letsencrypt-prod for trusted certificate.
Staging cert was causing connection resets due to certificate validation failures.
Change CouchDB ingress hostname from communicore.mk-labs.cloud to
communicore.local.mk-labs.cloud to align with External-DNS configuration.
CouchDB is an internal service and should use the .local.mk-labs.cloud
zone managed by Technitium DNS, not the public .mk-labs.cloud zone
managed by Cloudflare.
This ensures External-DNS will properly create the A record in the
internal DNS server.
Removing Firecrawl ArgoCD application and all manifests. The deployment
was failing due to missing container images that need to be built from
source. This requires platform infrastructure we don't have yet.
Will return to Firecrawl deployment after Harbor registry and Tekton
pipelines are deployed and configured.
Note: ArgoCD also needs a thematic EPCOT name at some point.
- Changed from ghcr.io/mendableai/* to ghcr.io/firecrawl/*
- Updated all three services: main API, playwright-service, and nuq-postgres
- Changed tag from v1.0.0 to latest (official images use latest tag)
- Fixes ImagePullBackOff errors caused by incorrect registry namespace
Per official Firecrawl docker-compose.yaml, images are published under
ghcr.io/firecrawl/, not ghcr.io/mendableai/
CRITICAL FIXES:
✅ Add PersistentVolumeClaims for all stateful services
- PostgreSQL: 20Gi PVC on nfs-emporium (data persistence)
- Redis: 10Gi PVC on nfs-emporium (cache and queue)
- RabbitMQ: 5Gi PVC on nfs-emporium (message queue)
✅ Pin all image versions (no more 'latest' tags)
- Firecrawl API/Worker: v1.0.0
- Playwright Service: v1.0.0
- PostgreSQL (nuq-postgres): v1.0.0
- Redis: 7.4.1-alpine
- RabbitMQ: 3.13.7-management-alpine
✅ Convert raw manifests to proper Helm chart
- Template-based configuration
- Centralized values.yaml
- Proper Helm helpers and labels
- Easy configuration management
WHAT CHANGED:
- Created chart/ directory with full Helm chart structure
- Moved old manifests to old-manifests/ for reference
- Updated ArgoCD Application to use Helm chart source
- Added comprehensive README and MIGRATION docs
- All services now use nfs-emporium storage class
- Redis configured with AOF persistence
- Proper resource limits and health checks
- Gateway/HTTPRoute configs integrated into chart
DEPLOYMENT:
ArgoCD will automatically sync and apply changes.
Old ephemeral data will be lost (fresh start with persistence).
Resolves data loss issues and brings deployment to production standards.
