The configureUserSettings field contained nginx configuration
('http2_push_preload on;') which was incorrectly being used as
CONFIG_OVERWRITE_JSON. This caused harbor-core to crash with a
JSON parse error.
CONFIG_OVERWRITE_JSON expects valid JSON for Harbor configuration
overrides, not nginx snippets. Removing this field to fix the
CrashLoopBackOff.
- ExternalSecret now pulls only HARBOR_ADMIN_PASSWORD from 1Password
- Removed database, redis, core, jobservice, registry secret references
- Harbor Helm chart auto-generates all internal secrets (standard pattern)
- Reduces complexity and aligns with Harbor best practices
This change removes dependency on 5 1Password fields that should be deleted:
- database-password
- redis-password
- core-secret
- jobservice-secret
- registry-password
Only harbor-admin-password field needed in 1Password item 'the-seas'
Harbor Helm chart requires both existingSecret and existingSecretKey
parameters to properly reference credentials. Without the key names,
the chart creates secrets with empty passwords, causing authentication
failures between components.
Added:
- database.internal.existingSecretKey: DATABASE_PASSWORD
- redis.internal.existingSecretKey: REDIS_PASSWORD
The secretName parameter was being used as a literal secret name
'CORE_SECRET' which doesn't exist. This caused harbor-core pods to
fail mounting volumes.
The correct approach is to use existingSecret for credential keys,
and let Harbor manage its own internal secrets.
Harbor Helm chart expects 'secret' key not 'CORE_SECRET' for the
CORE_SECRET environment variable. This was causing jobservice pod
to fail with CreateContainerConfigError.
Fixes: harbor-jobservice-757bbf44cc-tvznq error
Removing Firecrawl ArgoCD application and all manifests. The deployment
was failing due to missing container images that need to be built from
source. This requires platform infrastructure we don't have yet.
Will return to Firecrawl deployment after Harbor registry and Tekton
pipelines are deployed and configured.
Note: ArgoCD also needs a thematic EPCOT name at some point.
- Changed from ghcr.io/mendableai/* to ghcr.io/firecrawl/*
- Updated all three services: main API, playwright-service, and nuq-postgres
- Changed tag from v1.0.0 to latest (official images use latest tag)
- Fixes ImagePullBackOff errors caused by incorrect registry namespace
Per official Firecrawl docker-compose.yaml, images are published under
ghcr.io/firecrawl/, not ghcr.io/mendableai/
CRITICAL FIXES:
✅ Add PersistentVolumeClaims for all stateful services
- PostgreSQL: 20Gi PVC on nfs-emporium (data persistence)
- Redis: 10Gi PVC on nfs-emporium (cache and queue)
- RabbitMQ: 5Gi PVC on nfs-emporium (message queue)
✅ Pin all image versions (no more 'latest' tags)
- Firecrawl API/Worker: v1.0.0
- Playwright Service: v1.0.0
- PostgreSQL (nuq-postgres): v1.0.0
- Redis: 7.4.1-alpine
- RabbitMQ: 3.13.7-management-alpine
✅ Convert raw manifests to proper Helm chart
- Template-based configuration
- Centralized values.yaml
- Proper Helm helpers and labels
- Easy configuration management
WHAT CHANGED:
- Created chart/ directory with full Helm chart structure
- Moved old manifests to old-manifests/ for reference
- Updated ArgoCD Application to use Helm chart source
- Added comprehensive README and MIGRATION docs
- All services now use nfs-emporium storage class
- Redis configured with AOF persistence
- Proper resource limits and health checks
- Gateway/HTTPRoute configs integrated into chart
DEPLOYMENT:
ArgoCD will automatically sync and apply changes.
Old ephemeral data will be lost (fresh start with persistence).
Resolves data loss issues and brings deployment to production standards.
- New variable honcho_deriver_autostart controls service state/enabled
- honcho_deriver_enabled still controls whether Quadlet is created
- Default autostart=false after discovering deriver burns tokens autonomously
- Service is created but stopped/disabled; can be started manually if needed
- Add vault_honcho_openai_api_key (embeddings-only, Honcho-scoped)
- Inject OPENAI_API_KEY + LLM_OPENAI_API_KEY into api + deriver containers
- Flip honcho_embed_messages default to true now that embeddings have a provider
- Parameterise EMBEDDING__MODEL_CONFIG__{TRANSPORT,MODEL,BASE_URL} so we can
later swap to a local OpenAI-compatible embedder (e.g. Ollama on
astro-orbiter post-rebuild) with a single defaults change.
Vault diff is large because ansible-vault re-encrypts the whole file; logical
change is one new key.
Semaphore v2.18 source (db/TemplateVault.go) shows FillTemplateVault
only loads the key when type==TemplateVaultPassword. With type left
as the default empty string, the vault key is stored but never
materialised at task-run time; ansible-playbook gets no password and
fails with 'Attempting to decrypt but no vault secrets found'.
Confirmed by patching live templates via PUT with type='password';
next task run succeeded.
Semaphore v2.18 replaced the single vault_key_id field with a vaults[]
array supporting multi-vault per template. Sending vault_key_id is
silently ignored — template stores no vault association, runs fail
at the playbook stage with 'Attempting to decrypt but no vault
secrets found'.
Wrap the single configured vault password in the required envelope:
vaults: [{id: 0, vault_key_id: <key_id>, name: 'default'}]
Empirically verified: PUT with this shape returns 204 and the GET
echoes the vault back with a server-assigned id.
Semaphore runs ansible-playbook from the cloned repo root, not from
the playbook directory. The previous value '../roles' resolved
outside the repo. Correct path is 'ansible/roles'.
Semaphore runs ansible-playbook from the playbook's directory, not
from ansible/ where ansible.cfg lives. Roles aren't found at runtime:
'role linux-baseline was not found'.
Set ANSIBLE_ROLES_PATH=../roles in the default environment so the
search picks up ansible/roles/ relative to the playbook directory.
Semaphore rejected the previous become_key=wed-ssh with 'access key
type not supported for ansible become user' — that field is for a
sudo PASSWORD (login_password type), not a reused SSH key. wed has
passwordless sudo on every host (set up by the VM template), so the
correct value is the built-in 'None' key.
Adds wed-ssh (the universal automation account pre-baked in every
VM template) to the declared key set and switches the production
inventory to use it for both ssh_key and become_key. Retains
jarvis-ssh for cases that specifically need admin-level access.
This aligns Semaphore-driven jobs with the established homelab
convention: wed runs the playbooks, jarvis is the higher-privilege
admin account provisioned by linux-baseline.
Operator (Ryan) cleaned out the previous templates + inventory in
Semaphore before this commit so the configure step re-creates them
wired to wed-ssh on its next run.
The 'uri' module with body_format=json was sending integer fields as
strings when they came from quoted Jinja expressions in inline YAML
(e.g. ssh_key_id: "{{ ... | int }}"), because YAML loads the quoted
form as a string and Ansible doesn't coerce back. Semaphore rejects
that with HTTP 400.
Build each body as a Jinja dict literal in a folded scalar so types
survive: integers stay integers, strings stay strings.
Also restores no_log: true on key creation now that we're past the
debug round.
Jinja interpreted semaphore_config.keys as the dict method rather than
the 'keys' field, causing 'Invalid data passed to loop' failure. Bracket
subscript disambiguates.
Adds an idempotent configuration pass that drives a freshly-deployed
Semaphore instance into its desired state via the REST API. Declared
in group_vars/all/semaphore.yml, applied by tasks/configure.yml,
toggled by semaphore_configure feature flag (default off).
Object types managed:
- Project (mk-labs)
- Keys (ansible-vault-pass, gitea-deploy, jarvis-ssh)
- Repositories (homelab on gitea)
- Inventories (production -> ansible/inventory.yml in homelab repo)
- Environments (default with ANSIBLE_HOST_KEY_CHECKING=False)
- Templates (day0_linux_baseline + variants, day1_deploy_semaphore)
with survey vars for runtime parameters
Each object found-or-created by name; existing ones never modified.
no_log on token-bearing calls to keep secrets out of stdout.
Inputs (already in vault):
vault_semaphore_api_token
vault_jarvis_ssh_private_key
vault_gitea_deploy_key
vault_ansible_vault_password
Three new vault entries required for Semaphore config-as-code:
- vault_jarvis_ssh_private_key (matches jarvis_ssh_public_key in
group_vars/all/vars; used by Semaphore to SSH to the fleet)
- vault_gitea_deploy_key (existing deploy key on the homelab
repo; used by Semaphore to clone)
- vault_semaphore_api_token (minted from the Semaphore UI; used
by Ansible to drive Semaphore configuration)
These wire up the inputs the upcoming semaphore configure step
will consume.
Quadlet regenerates the systemd unit on daemon-reload but does not
restart the running container — the process keeps its original
environment until restarted. Add explicit 'state: restarted' tasks
gated on the Quadlet template's changed status so env-var updates
actually take effect on re-runs.
The SPA was rendering <base href="https://semaphore.local.mk-labs.cloud/">
regardless of which hostname served the page, causing the imagineering
alias to load the same UI but rewrite all in-page navigation back to the
semaphore hostname. Confusing for users hitting either Traefik alias.
Leave SEMAPHORE_WEB_ROOT empty so Semaphore emits relative URLs and is
fully host-agnostic. Both DNS names (semaphore + imagineering) now serve
cleanly without preference between them.
The template only emits SEMAPHORE_WEB_ROOT when the variable is set, so
the env-var is absent from the container when not needed.
The v2.18 image's entrypoint runs the setup wizard on first boot. Without
the SEMAPHORE_ADMIN_* variables it prompts on stdin, fails with 'Username
cannot be empty', and the container exits — leading to a crash loop.
Set:
SEMAPHORE_ADMIN=admin
SEMAPHORE_ADMIN_NAME=Administrator
SEMAPHORE_ADMIN_EMAIL=admin@local.mk-labs.cloud
SEMAPHORE_ADMIN_PASSWORD={{ vault_semaphore_admin_password }}
SEMAPHORE_PLAYBOOK_PATH=/var/lib/semaphore/playbooks
The env-var bootstrap path is stable in v2.x; only the legacy
'semaphore user add' CLI invocation was unreliable. Drop the manual
user-add step from the README.
Complete rewrite of the semaphore role. Supersedes three prior
iterations whose admin-user-creation logic was unreliable across
Semaphore CLI versions.
Architecture:
- Rootful Podman Quadlet under /etc/containers/systemd/
- Separate PostgreSQL 16-alpine container on a user-defined
podman network (semaphore-net)
- Named volumes for both data stores (semaphore_data,
semaphore_postgres_data) so container recreation is
non-destructive
- Pinned image tags: semaphoreui/semaphore:v2.18.5-ansible2.16.5
and postgres:16-alpine
- Post-deploy HTTP health check fails the playbook if Semaphore
doesn't respond on /api/ping within ~60s
Admin user creation remains intentionally manual after first deploy;
the role README documents the exact podman exec command.
Removes the duplicate deploy_semaphore.yml and the now-unneeded
cleanup_semaphore.yml; day1_deploy_semaphore.yml is the canonical
entry point.
Previous template used inline arithmetic that miscounted the box
width, leaving the right border misaligned on real hostnames. Switch
to computed labels + a fixed inner_width so any hostname / OS string
pads to the same border position.
Introduces a single, idempotent baseline role to supersede the
overlapping day0-baseline and common roles. Capabilities are
feature-flagged so they can be toggled per-host:
- packages (common + OS-family + per-host extras)
- timezone + locale
- chrony time sync against sundial
- baseline users (jarvis admin account with SSH key + NOPASSWD sudo)
- SSH hardening via /etc/ssh/sshd_config.d/ drop-in
- unattended security upgrades (Debian family)
- sysctl drop-in at /etc/sysctl.d/99-mk-labs.conf
- journald retention caps
- branded MOTD
Ubuntu/Debian is first-class; vars/RedHat.yml provides a placeholder
for future distros via the ansible_os_family pattern.
The legacy day0-baseline and common roles remain in place for now and
will be removed during the playbook cleanup sweep, alongside the
existing playbook naming inconsistencies.
- Explicit playbook for removing old container, systemd services, and quadlets
- Optional semaphore_force_clean variable for data removal
- Safer than tags for destructive operations
- Add quadlet tasks and handlers for modern rootless Podman deployment
- Fix broken Jinja in semaphore.container.j2
- Add proper 0777 permissions handling for rootless
- Support semaphore_use_postgres toggle with fallback to legacy BoltDB path
- Parse Host() rules from router definitions
- Supports multiple hostnames per service file (e.g. semaphore + imagineering)
- More robust and future-proof
- Switch podman_container env to SEMAPHORE_DB_DIALECT
- Remove creates: guard on podman generate systemd task
- Add changed_when so unit is always updated on role run
- Updated container quadlet template to use modern SEMAPHORE_DB_DIALECT variable
- Updated env.j2 template for consistency with current SemaphoreUI expectations
- Resolves BoltDB not being honoured on v2.18.x
- Move all roles from playbooks/roles/ to roles/
- Update roles_path in ansible.cfg
- Add cast user to common role
- Create standalone podman role
- Add semaphore role with Podman + Quadlet support
The previous Hermes deployment configuration has been superseded.
This directory is no longer used and is being removed as part of
the transition to the new command centre on carousel-of-progress.
