/home/semaphore/.ansible/ is owned by root after Podman creates the
bind-mount dir, so ansible-galaxy can't create sibling tmp dirs.
Mount to a neutral /opt/ansible-collections path and point Ansible
at it via ANSIBLE_COLLECTIONS_PATHS env var instead.
Binary lives inside the container at /opt/semaphore/apps/ansible/<ver>/venv/bin/.
Use podman exec to run the install, writing to /home/semaphore/.ansible/collections
which is bind-mounted from the host-side directory.
ansible-galaxy is not on the system PATH on figment — Semaphore manages
its own venv under /opt/semaphore/apps/ansible/<ver>/venv/bin/.
Discover the binary dynamically rather than relying on PATH.
- New tasks/collections.yml installs collections from defaults list
into /opt/semaphore/ansible-collections on the host
- semaphore.container.j2: bind-mounts that dir into the container at
/home/semaphore/.ansible/collections (read-only)
- defaults/main.yml: semaphore_collections_dir + semaphore_ansible_collections
list (containers.podman, effectivelywild.technitium_dns)
- main.yml: collections task wired in after semaphore.yml, before verify
- Collections survive container restarts/rebuilds without image changes
- Re-run with --tags collections to add new collections without full redeploy
Collection not installed in Semaphore's Ansible environment.
Direct HTTP API call to Technitium on :5380 is simpler, has no
collection dependency, and is naturally idempotent (add returns ok
on duplicate records).
- New entrypoints: gitea-ssh/:2221, jarvis-ssh/:10171
- Expose both ports from the Traefik container
- gitea.yml: TCP passthrough router -> 10.1.71.129:2221
- jarvis.yml: TCP passthrough router -> 10.1.71.131:22
- Both use HostSNI(*) — dedicated entrypoints, no TLS wrapping needed
- UniFi to forward both ports to lightning-lane
Playwright service doesn't expose a health endpoint at /, causing HTTP
probes to fail with 404. Switch to tcpSocket probes which simply verify
the port is listening. Service is already confirmed running on port 3000.
- Changed worker deployment probes from HTTP to TCP (port 3005)
* Worker liveness endpoint doesn't serve HTTP at '/' path
* TCP socket check more appropriate for background worker
* Resolves pod restart loop and readiness failures
- Corrected HTTPRoute gateway reference
* Changed from 'gateway' in 'default' namespace
* To 'fastpass-gateway' in 'gateway' namespace
* HTTPRoute now properly accepted by gateway
All 7 deployments Running and Ready. System operational.
- Added config.json key to harbor-credentials ExternalSecret
This ensures kaniko can find the Docker auth config at /kaniko/.docker/config.json
(previously only .dockerconfigjson was present)
- Created test-app-build PipelineRun manifest for validation testing
- Successfully validated end-to-end pipeline:
✅ git-clone Task deployed and working
✅ kaniko-build Task deployed and working
✅ container-build Pipeline deployed and working
✅ Harbor authentication working with robot account
✅ Test image built and pushed: the-seas.local.mk-labs.cloud/library/test-app:v1.0.0
✅ Image digest: sha256:aa143f4a01795a1d307b711108ca0c89f36e00ea38fddb9d7b2febd5fffc46d7
Pipeline test results:
- PipelineRun: test-app-build-005 - SUCCEEDED
- fetch-repository TaskRun - SUCCEEDED
- build-and-push TaskRun - SUCCEEDED
Tekton CI/CD platform is now operational and ready for production workloads.
Replace letsencrypt-staging with letsencrypt-prod for trusted certificate.
Staging cert was causing connection resets due to certificate validation failures.
Change CouchDB ingress hostname from communicore.mk-labs.cloud to
communicore.local.mk-labs.cloud to align with External-DNS configuration.
CouchDB is an internal service and should use the .local.mk-labs.cloud
zone managed by Technitium DNS, not the public .mk-labs.cloud zone
managed by Cloudflare.
This ensures External-DNS will properly create the A record in the
internal DNS server.
Updated README to reflect the full workflow:
1. Robot account creation via Job
2. Secret regeneration and capture
3. Storage in 1Password (harbor-robot-accounts item)
4. Automatic sync via ExternalSecrets to K8s
Creates two docker-registry secrets from 1Password:
1. harbor-tekton-robot - for Tekton CI/CD pipeline push/pull
2. harbor-pull-secret - for fastpass cluster image pulls
Both sync from 1Password item 'harbor-robot-accounts' with fields:
- tekton-builder-username / tekton-builder-password
- fastpass-cluster-username / fastpass-cluster-password
Credentials document placed in PKA inbox for manual 1Password entry.
Once stored, ESO will automatically sync and create the secrets.
Documents robot account management via GitOps:
- tekton-builder and fastpass-cluster robot accounts
- Manual Job execution (PostSync hooks don't work with multi-source)
- Credential retrieval and storage in 1Password
- ImagePullSecret creation for K8s clusters
DNS was pointing to Gateway (10.1.71.90) instead of NGINX Ingress (10.1.71.80)
after we removed the HTTPRoute manifests. This caused traffic to hit the
Gateway's wildcard cert and get 'no healthy upstream' errors.
Added external-dns annotations to Ingress to direct DNS to correct IP:
- external-dns.alpha.kubernetes.io/hostname: the-seas.local.mk-labs.cloud
- external-dns.alpha.kubernetes.io/target: 10.1.71.80
This will update Technitium DNS to point to NGINX Ingress Controller.
The configureUserSettings field contained nginx configuration
('http2_push_preload on;') which was incorrectly being used as
CONFIG_OVERWRITE_JSON. This caused harbor-core to crash with a
JSON parse error.
CONFIG_OVERWRITE_JSON expects valid JSON for Harbor configuration
overrides, not nginx snippets. Removing this field to fix the
CrashLoopBackOff.
- ExternalSecret now pulls only HARBOR_ADMIN_PASSWORD from 1Password
- Removed database, redis, core, jobservice, registry secret references
- Harbor Helm chart auto-generates all internal secrets (standard pattern)
- Reduces complexity and aligns with Harbor best practices
This change removes dependency on 5 1Password fields that should be deleted:
- database-password
- redis-password
- core-secret
- jobservice-secret
- registry-password
Only harbor-admin-password field needed in 1Password item 'the-seas'
Harbor Helm chart requires both existingSecret and existingSecretKey
parameters to properly reference credentials. Without the key names,
the chart creates secrets with empty passwords, causing authentication
failures between components.
Added:
- database.internal.existingSecretKey: DATABASE_PASSWORD
- redis.internal.existingSecretKey: REDIS_PASSWORD
The secretName parameter was being used as a literal secret name
'CORE_SECRET' which doesn't exist. This caused harbor-core pods to
fail mounting volumes.
The correct approach is to use existingSecret for credential keys,
and let Harbor manage its own internal secrets.
Harbor Helm chart expects 'secret' key not 'CORE_SECRET' for the
CORE_SECRET environment variable. This was causing jobservice pod
to fail with CreateContainerConfigError.
Fixes: harbor-jobservice-757bbf44cc-tvznq error
Removing Firecrawl ArgoCD application and all manifests. The deployment
was failing due to missing container images that need to be built from
source. This requires platform infrastructure we don't have yet.
Will return to Firecrawl deployment after Harbor registry and Tekton
pipelines are deployed and configured.
Note: ArgoCD also needs a thematic EPCOT name at some point.
- Changed from ghcr.io/mendableai/* to ghcr.io/firecrawl/*
- Updated all three services: main API, playwright-service, and nuq-postgres
- Changed tag from v1.0.0 to latest (official images use latest tag)
- Fixes ImagePullBackOff errors caused by incorrect registry namespace
Per official Firecrawl docker-compose.yaml, images are published under
ghcr.io/firecrawl/, not ghcr.io/mendableai/
CRITICAL FIXES:
✅ Add PersistentVolumeClaims for all stateful services
- PostgreSQL: 20Gi PVC on nfs-emporium (data persistence)
- Redis: 10Gi PVC on nfs-emporium (cache and queue)
- RabbitMQ: 5Gi PVC on nfs-emporium (message queue)
✅ Pin all image versions (no more 'latest' tags)
- Firecrawl API/Worker: v1.0.0
- Playwright Service: v1.0.0
- PostgreSQL (nuq-postgres): v1.0.0
- Redis: 7.4.1-alpine
- RabbitMQ: 3.13.7-management-alpine
✅ Convert raw manifests to proper Helm chart
- Template-based configuration
- Centralized values.yaml
- Proper Helm helpers and labels
- Easy configuration management
WHAT CHANGED:
- Created chart/ directory with full Helm chart structure
- Moved old manifests to old-manifests/ for reference
- Updated ArgoCD Application to use Helm chart source
- Added comprehensive README and MIGRATION docs
- All services now use nfs-emporium storage class
- Redis configured with AOF persistence
- Proper resource limits and health checks
- Gateway/HTTPRoute configs integrated into chart
DEPLOYMENT:
ArgoCD will automatically sync and apply changes.
Old ephemeral data will be lost (fresh start with persistence).
Resolves data loss issues and brings deployment to production standards.
