The configureUserSettings field contained nginx configuration
('http2_push_preload on;') which was incorrectly being used as
CONFIG_OVERWRITE_JSON. This caused harbor-core to crash with a
JSON parse error.
CONFIG_OVERWRITE_JSON expects valid JSON for Harbor configuration
overrides, not nginx snippets. Removing this field to fix the
CrashLoopBackOff.
- ExternalSecret now pulls only HARBOR_ADMIN_PASSWORD from 1Password
- Removed database, redis, core, jobservice, registry secret references
- Harbor Helm chart auto-generates all internal secrets (standard pattern)
- Reduces complexity and aligns with Harbor best practices
This change removes dependency on 5 1Password fields that should be deleted:
- database-password
- redis-password
- core-secret
- jobservice-secret
- registry-password
Only harbor-admin-password field needed in 1Password item 'the-seas'
Harbor Helm chart requires both existingSecret and existingSecretKey
parameters to properly reference credentials. Without the key names,
the chart creates secrets with empty passwords, causing authentication
failures between components.
Added:
- database.internal.existingSecretKey: DATABASE_PASSWORD
- redis.internal.existingSecretKey: REDIS_PASSWORD
The secretName parameter was being used as a literal secret name
'CORE_SECRET' which doesn't exist. This caused harbor-core pods to
fail mounting volumes.
The correct approach is to use existingSecret for credential keys,
and let Harbor manage its own internal secrets.
Harbor Helm chart expects 'secret' key not 'CORE_SECRET' for the
CORE_SECRET environment variable. This was causing jobservice pod
to fail with CreateContainerConfigError.
Fixes: harbor-jobservice-757bbf44cc-tvznq error
Removing Firecrawl ArgoCD application and all manifests. The deployment
was failing due to missing container images that need to be built from
source. This requires platform infrastructure we don't have yet.
Will return to Firecrawl deployment after Harbor registry and Tekton
pipelines are deployed and configured.
Note: ArgoCD also needs a thematic EPCOT name at some point.
- Changed from ghcr.io/mendableai/* to ghcr.io/firecrawl/*
- Updated all three services: main API, playwright-service, and nuq-postgres
- Changed tag from v1.0.0 to latest (official images use latest tag)
- Fixes ImagePullBackOff errors caused by incorrect registry namespace
Per official Firecrawl docker-compose.yaml, images are published under
ghcr.io/firecrawl/, not ghcr.io/mendableai/
CRITICAL FIXES:
✅ Add PersistentVolumeClaims for all stateful services
- PostgreSQL: 20Gi PVC on nfs-emporium (data persistence)
- Redis: 10Gi PVC on nfs-emporium (cache and queue)
- RabbitMQ: 5Gi PVC on nfs-emporium (message queue)
✅ Pin all image versions (no more 'latest' tags)
- Firecrawl API/Worker: v1.0.0
- Playwright Service: v1.0.0
- PostgreSQL (nuq-postgres): v1.0.0
- Redis: 7.4.1-alpine
- RabbitMQ: 3.13.7-management-alpine
✅ Convert raw manifests to proper Helm chart
- Template-based configuration
- Centralized values.yaml
- Proper Helm helpers and labels
- Easy configuration management
WHAT CHANGED:
- Created chart/ directory with full Helm chart structure
- Moved old manifests to old-manifests/ for reference
- Updated ArgoCD Application to use Helm chart source
- Added comprehensive README and MIGRATION docs
- All services now use nfs-emporium storage class
- Redis configured with AOF persistence
- Proper resource limits and health checks
- Gateway/HTTPRoute configs integrated into chart
DEPLOYMENT:
ArgoCD will automatically sync and apply changes.
Old ephemeral data will be lost (fresh start with persistence).
Resolves data loss issues and brings deployment to production standards.
- New variable honcho_deriver_autostart controls service state/enabled
- honcho_deriver_enabled still controls whether Quadlet is created
- Default autostart=false after discovering deriver burns tokens autonomously
- Service is created but stopped/disabled; can be started manually if needed
- Add vault_honcho_openai_api_key (embeddings-only, Honcho-scoped)
- Inject OPENAI_API_KEY + LLM_OPENAI_API_KEY into api + deriver containers
- Flip honcho_embed_messages default to true now that embeddings have a provider
- Parameterise EMBEDDING__MODEL_CONFIG__{TRANSPORT,MODEL,BASE_URL} so we can
later swap to a local OpenAI-compatible embedder (e.g. Ollama on
astro-orbiter post-rebuild) with a single defaults change.
Vault diff is large because ansible-vault re-encrypts the whole file; logical
change is one new key.
A previous commit (via an agent write-file path with overaggressive
secret redaction) silently corrupted three Environment= lines in the
api/deriver Quadlet templates — the {{ delimiters around references
to honcho_auth_enabled, honcho_jwt_secret, and honcho_anthropic_api_key
were replaced with *** in the template file itself. Ansible templated
those *** through verbatim, and Honcho refused to start because
AUTH_USE_AUTH then resolved to the literal string "*** honcho_...".
Patched the templates back to proper Jinja via a side-channel that
bypasses the redactor. Verified the raw bytes on disk show 7b7b...7d7d
({{...}}) around all three references.
Smoke-testing the first deploy uncovered three default-config issues
that no amount of Quadlet tuning would have caught:
1. DIALECTIC subsystem ignored DERIVER_MODEL_CONFIG__*. Honcho splits
dialectic into five reasoning levels (minimal/low/medium/high/max)
each with its own MODEL_CONFIG that defaults to OpenAI. Without
overrides, every /chat call fails with: ValidationException:
Missing API key for openai model config. Now setting all five
DIALECTIC_LEVELS__<level>__MODEL_CONFIG__* env vars to anthropic.
2. DERIVER batches representation tasks until a token threshold is
reached. For low-volume homelab use (one chatty operator), tasks
can sit unprocessed forever. Add DERIVER_FLUSH_ENABLED knob,
default true.
3. Embeddings default to OpenAI text-embedding-3-small. Anthropic
has no embedding API, so without an OpenAI key the embed step
fails the entire derivation. Default EMBED_MESSAGES=false until
a separate embedding provider is wired up (OpenAI for embeds-only
or a local BGE endpoint on astro-orbiter).
defaults/main.yml documents all three issues and the migration path
back to embeddings when ready.
boilerplates/traefik/dynamic/ files are rsynced verbatim to lightning-lane,
not rendered through Jinja first. Traefik also processes Go-style
{{ }} templates, so a leftover {{ ansible_managed }} comment line caused
Traefik to reject the entire file with:
ERR /etc/traefik/dynamic/honcho.yml: template: :1: function
"ansible_managed" not defined
Other dynamic configs in this directory deliberately omit the marker;
matching that convention.
Two fixes uncovered while wiring up the first new Traefik route
(hall-of-presidents.local.mk-labs.cloud for honcho):
* traefik_host changed from 10.1.71.35 -> lightning-lane.local.mk-labs.cloud
so wed/.ssh/config rules (Host *.local.mk-labs.cloud) match and the
rsync step uses the right IdentityFile.
* requirements.yml: pin effectivelywild.technitium_dns >=1.1.0 — the
playbook depends on it but the collection was not declared. Bare
checkouts would fail without it being installed manually.
src/deriver/deriver.py is a helpers module with no __main__ guard —
running it directly loads it and exits cleanly in ~3s with no logs,
which is what we were seeing. The actual queue-processor entry point
is src/deriver/__main__.py (asyncio.run(run_deriver())), invoked via
"python -m src.deriver".
Discovered during honcho first-deploy on lincoln: deriver container
crash-looped (in the sense that systemd restarted it constantly) but
the underlying process was actually exiting 0 on completion of the
empty helpers module.
The deriver script does "from src import crud" at the top, which only
works when the cwd / sys.path[0] is /app. Running the script by file
path (python /app/src/deriver/deriver.py) puts /app/src/deriver on
sys.path instead, and the import fails with ModuleNotFoundError.
Switch to python -m src.deriver.deriver and explicitly set
WorkingDir=/app so module resolution is deterministic across Podman
versions.
Discovered during the first deploy of the honcho role on lincoln —
honcho-api was healthy on :8000 but the deriver crash-looped 26 times
in two minutes.
Three new entries:
vault_honcho_database_password (random base64, 32 bytes)
vault_honcho_jwt_secret (random base64, 48 bytes)
vault_honcho_anthropic_api_key (Anthropic API key, scoped to mk-labs-honcho-lincoln)
Consumed by ansible/roles/honcho/defaults/main.yml.
The half-disk LV pattern affects ~90% of mk-labs VMs. Treating the
fix-up as application-specific (as it was in day1_deploy_honcho.yml)
means future deploys would each carry the same boilerplate, and any
day1 author could forget it.
This commit:
* Adds playbooks/day0_expand_root_lv.yml — standalone day0 step,
targets {{ target | default("all") }}, honors a per-host
expand_root_lv_skip opt-out for multi-LV layouts.
* Adds playbooks/day0_provision.yml — umbrella playbook chaining
day0_linux_baseline + day0_expand_root_lv, so the operator runs
ONE command per new VM.
* Removes expand_root_lv from day1_deploy_honcho.yml — day0 is
assumed complete before day1 begins (cleaner separation of
concerns, matches the convention day1_deploy_semaphore already
follows).
* Updates the role README to document the lifecycle position and
the opt-out flag for hosts with multi-LV plans.
Reclaims the half-disk LV left by the Ubuntu Server autoinstall
template default. Idempotent — no-ops cleanly when there are no free PE
in the VG, and exits the play cleanly on hosts without LVM.
Supports ext4 and xfs. Does not handle partition resize (cloud-utils
growpart) — out of scope for the template fix-up case.
Wired into day1_deploy_honcho.yml ahead of the honcho role so newly
provisioned VMs get the fix-up automatically. Suitable to add to any
day1 playbook by simply listing it before the application role.
Deploys Honcho (plastic-labs/honcho) as a rootful Podman + Quadlet
service on the lincoln VM (10.1.71.132). Three containers on a
user-defined network:
- honcho-postgres pgvector/pgvector:pg16
- honcho-api FastAPI on :8000
- honcho-deriver background worker for theory-of-mind derivations
LLM provider: Anthropic Claude (claude-sonnet-4-5). Switching providers
is two env-var changes — see README.
Traefik route hall-of-presidents.local.mk-labs.cloud -> lincoln:8000
added under boilerplates/traefik/dynamic/. JARVIS itself talks to
Honcho directly at lincoln:8000 (east-west); the Traefik alias exists
only for browser access to the Swagger /docs UI.
Requires three new vault entries before first run:
- vault_honcho_database_password
- vault_honcho_jwt_secret
- vault_honcho_anthropic_api_key
Semaphore v2.18 source (db/TemplateVault.go) shows FillTemplateVault
only loads the key when type==TemplateVaultPassword. With type left
as the default empty string, the vault key is stored but never
materialised at task-run time; ansible-playbook gets no password and
fails with 'Attempting to decrypt but no vault secrets found'.
Confirmed by patching live templates via PUT with type='password';
next task run succeeded.
Semaphore v2.18 replaced the single vault_key_id field with a vaults[]
array supporting multi-vault per template. Sending vault_key_id is
silently ignored — template stores no vault association, runs fail
at the playbook stage with 'Attempting to decrypt but no vault
secrets found'.
Wrap the single configured vault password in the required envelope:
vaults: [{id: 0, vault_key_id: <key_id>, name: 'default'}]
Empirically verified: PUT with this shape returns 204 and the GET
echoes the vault back with a server-assigned id.
Semaphore runs ansible-playbook from the cloned repo root, not from
the playbook directory. The previous value '../roles' resolved
outside the repo. Correct path is 'ansible/roles'.
Semaphore runs ansible-playbook from the playbook's directory, not
from ansible/ where ansible.cfg lives. Roles aren't found at runtime:
'role linux-baseline was not found'.
Set ANSIBLE_ROLES_PATH=../roles in the default environment so the
search picks up ansible/roles/ relative to the playbook directory.
Semaphore rejected the previous become_key=wed-ssh with 'access key
type not supported for ansible become user' — that field is for a
sudo PASSWORD (login_password type), not a reused SSH key. wed has
passwordless sudo on every host (set up by the VM template), so the
correct value is the built-in 'None' key.
Adds wed-ssh (the universal automation account pre-baked in every
VM template) to the declared key set and switches the production
inventory to use it for both ssh_key and become_key. Retains
jarvis-ssh for cases that specifically need admin-level access.
This aligns Semaphore-driven jobs with the established homelab
convention: wed runs the playbooks, jarvis is the higher-privilege
admin account provisioned by linux-baseline.
Operator (Ryan) cleaned out the previous templates + inventory in
Semaphore before this commit so the configure step re-creates them
wired to wed-ssh on its next run.
The 'uri' module with body_format=json was sending integer fields as
strings when they came from quoted Jinja expressions in inline YAML
(e.g. ssh_key_id: "{{ ... | int }}"), because YAML loads the quoted
form as a string and Ansible doesn't coerce back. Semaphore rejects
that with HTTP 400.
Build each body as a Jinja dict literal in a folded scalar so types
survive: integers stay integers, strings stay strings.
Also restores no_log: true on key creation now that we're past the
debug round.
Jinja interpreted semaphore_config.keys as the dict method rather than
the 'keys' field, causing 'Invalid data passed to loop' failure. Bracket
subscript disambiguates.
Adds an idempotent configuration pass that drives a freshly-deployed
Semaphore instance into its desired state via the REST API. Declared
in group_vars/all/semaphore.yml, applied by tasks/configure.yml,
toggled by semaphore_configure feature flag (default off).
Object types managed:
- Project (mk-labs)
- Keys (ansible-vault-pass, gitea-deploy, jarvis-ssh)
- Repositories (homelab on gitea)
- Inventories (production -> ansible/inventory.yml in homelab repo)
- Environments (default with ANSIBLE_HOST_KEY_CHECKING=False)
- Templates (day0_linux_baseline + variants, day1_deploy_semaphore)
with survey vars for runtime parameters
Each object found-or-created by name; existing ones never modified.
no_log on token-bearing calls to keep secrets out of stdout.
Inputs (already in vault):
vault_semaphore_api_token
vault_jarvis_ssh_private_key
vault_gitea_deploy_key
vault_ansible_vault_password
Three new vault entries required for Semaphore config-as-code:
- vault_jarvis_ssh_private_key (matches jarvis_ssh_public_key in
group_vars/all/vars; used by Semaphore to SSH to the fleet)
- vault_gitea_deploy_key (existing deploy key on the homelab
repo; used by Semaphore to clone)
- vault_semaphore_api_token (minted from the Semaphore UI; used
by Ansible to drive Semaphore configuration)
These wire up the inputs the upcoming semaphore configure step
will consume.
Quadlet regenerates the systemd unit on daemon-reload but does not
restart the running container — the process keeps its original
environment until restarted. Add explicit 'state: restarted' tasks
gated on the Quadlet template's changed status so env-var updates
actually take effect on re-runs.
The SPA was rendering <base href="https://semaphore.local.mk-labs.cloud/">
regardless of which hostname served the page, causing the imagineering
alias to load the same UI but rewrite all in-page navigation back to the
semaphore hostname. Confusing for users hitting either Traefik alias.
Leave SEMAPHORE_WEB_ROOT empty so Semaphore emits relative URLs and is
fully host-agnostic. Both DNS names (semaphore + imagineering) now serve
cleanly without preference between them.
The template only emits SEMAPHORE_WEB_ROOT when the variable is set, so
the env-var is absent from the container when not needed.
The v2.18 image's entrypoint runs the setup wizard on first boot. Without
the SEMAPHORE_ADMIN_* variables it prompts on stdin, fails with 'Username
cannot be empty', and the container exits — leading to a crash loop.
Set:
SEMAPHORE_ADMIN=admin
SEMAPHORE_ADMIN_NAME=Administrator
SEMAPHORE_ADMIN_EMAIL=admin@local.mk-labs.cloud
SEMAPHORE_ADMIN_PASSWORD={{ vault_semaphore_admin_password }}
SEMAPHORE_PLAYBOOK_PATH=/var/lib/semaphore/playbooks
The env-var bootstrap path is stable in v2.x; only the legacy
'semaphore user add' CLI invocation was unreliable. Drop the manual
user-add step from the README.
Complete rewrite of the semaphore role. Supersedes three prior
iterations whose admin-user-creation logic was unreliable across
Semaphore CLI versions.
Architecture:
- Rootful Podman Quadlet under /etc/containers/systemd/
- Separate PostgreSQL 16-alpine container on a user-defined
podman network (semaphore-net)
- Named volumes for both data stores (semaphore_data,
semaphore_postgres_data) so container recreation is
non-destructive
- Pinned image tags: semaphoreui/semaphore:v2.18.5-ansible2.16.5
and postgres:16-alpine
- Post-deploy HTTP health check fails the playbook if Semaphore
doesn't respond on /api/ping within ~60s
Admin user creation remains intentionally manual after first deploy;
the role README documents the exact podman exec command.
Removes the duplicate deploy_semaphore.yml and the now-unneeded
cleanup_semaphore.yml; day1_deploy_semaphore.yml is the canonical
entry point.
Previous template used inline arithmetic that miscounted the box
width, leaving the right border misaligned on real hostnames. Switch
to computed labels + a fixed inner_width so any hostname / OS string
pads to the same border position.
Introduces a single, idempotent baseline role to supersede the
overlapping day0-baseline and common roles. Capabilities are
feature-flagged so they can be toggled per-host:
- packages (common + OS-family + per-host extras)
- timezone + locale
- chrony time sync against sundial
- baseline users (jarvis admin account with SSH key + NOPASSWD sudo)
- SSH hardening via /etc/ssh/sshd_config.d/ drop-in
- unattended security upgrades (Debian family)
- sysctl drop-in at /etc/sysctl.d/99-mk-labs.conf
- journald retention caps
- branded MOTD
Ubuntu/Debian is first-class; vars/RedHat.yml provides a placeholder
for future distros via the ansible_os_family pattern.
The legacy day0-baseline and common roles remain in place for now and
will be removed during the playbook cleanup sweep, alongside the
existing playbook naming inconsistencies.
