Host dir was owned by root (0755), blocking writes from uid=1001 inside
the container. Set owner to semaphore_container_uid=1001 so podman exec
can write collections into the bind-mount.
- Removed :ro from volume mount — ansible-galaxy writes via podman exec
into the container, so the mount must be writable during role runs
- Fixed deprecated ANSIBLE_COLLECTIONS_PATHS -> ANSIBLE_COLLECTIONS_PATH
/home/semaphore/.ansible/ is owned by root after Podman creates the
bind-mount dir, so ansible-galaxy can't create sibling tmp dirs.
Mount to a neutral /opt/ansible-collections path and point Ansible
at it via ANSIBLE_COLLECTIONS_PATHS env var instead.
Binary lives inside the container at /opt/semaphore/apps/ansible/<ver>/venv/bin/.
Use podman exec to run the install, writing to /home/semaphore/.ansible/collections
which is bind-mounted from the host-side directory.
ansible-galaxy is not on the system PATH on figment — Semaphore manages
its own venv under /opt/semaphore/apps/ansible/<ver>/venv/bin/.
Discover the binary dynamically rather than relying on PATH.
- New tasks/collections.yml installs collections from defaults list
into /opt/semaphore/ansible-collections on the host
- semaphore.container.j2: bind-mounts that dir into the container at
/home/semaphore/.ansible/collections (read-only)
- defaults/main.yml: semaphore_collections_dir + semaphore_ansible_collections
list (containers.podman, effectivelywild.technitium_dns)
- main.yml: collections task wired in after semaphore.yml, before verify
- Collections survive container restarts/rebuilds without image changes
- Re-run with --tags collections to add new collections without full redeploy
- New variable honcho_deriver_autostart controls service state/enabled
- honcho_deriver_enabled still controls whether Quadlet is created
- Default autostart=false after discovering deriver burns tokens autonomously
- Service is created but stopped/disabled; can be started manually if needed
- Add vault_honcho_openai_api_key (embeddings-only, Honcho-scoped)
- Inject OPENAI_API_KEY + LLM_OPENAI_API_KEY into api + deriver containers
- Flip honcho_embed_messages default to true now that embeddings have a provider
- Parameterise EMBEDDING__MODEL_CONFIG__{TRANSPORT,MODEL,BASE_URL} so we can
later swap to a local OpenAI-compatible embedder (e.g. Ollama on
astro-orbiter post-rebuild) with a single defaults change.
Vault diff is large because ansible-vault re-encrypts the whole file; logical
change is one new key.
A previous commit (via an agent write-file path with overaggressive
secret redaction) silently corrupted three Environment= lines in the
api/deriver Quadlet templates — the {{ delimiters around references
to honcho_auth_enabled, honcho_jwt_secret, and honcho_anthropic_api_key
were replaced with *** in the template file itself. Ansible templated
those *** through verbatim, and Honcho refused to start because
AUTH_USE_AUTH then resolved to the literal string "*** honcho_...".
Patched the templates back to proper Jinja via a side-channel that
bypasses the redactor. Verified the raw bytes on disk show 7b7b...7d7d
({{...}}) around all three references.
Smoke-testing the first deploy uncovered three default-config issues
that no amount of Quadlet tuning would have caught:
1. DIALECTIC subsystem ignored DERIVER_MODEL_CONFIG__*. Honcho splits
dialectic into five reasoning levels (minimal/low/medium/high/max)
each with its own MODEL_CONFIG that defaults to OpenAI. Without
overrides, every /chat call fails with: ValidationException:
Missing API key for openai model config. Now setting all five
DIALECTIC_LEVELS__<level>__MODEL_CONFIG__* env vars to anthropic.
2. DERIVER batches representation tasks until a token threshold is
reached. For low-volume homelab use (one chatty operator), tasks
can sit unprocessed forever. Add DERIVER_FLUSH_ENABLED knob,
default true.
3. Embeddings default to OpenAI text-embedding-3-small. Anthropic
has no embedding API, so without an OpenAI key the embed step
fails the entire derivation. Default EMBED_MESSAGES=false until
a separate embedding provider is wired up (OpenAI for embeds-only
or a local BGE endpoint on astro-orbiter).
defaults/main.yml documents all three issues and the migration path
back to embeddings when ready.
src/deriver/deriver.py is a helpers module with no __main__ guard —
running it directly loads it and exits cleanly in ~3s with no logs,
which is what we were seeing. The actual queue-processor entry point
is src/deriver/__main__.py (asyncio.run(run_deriver())), invoked via
"python -m src.deriver".
Discovered during honcho first-deploy on lincoln: deriver container
crash-looped (in the sense that systemd restarted it constantly) but
the underlying process was actually exiting 0 on completion of the
empty helpers module.
The deriver script does "from src import crud" at the top, which only
works when the cwd / sys.path[0] is /app. Running the script by file
path (python /app/src/deriver/deriver.py) puts /app/src/deriver on
sys.path instead, and the import fails with ModuleNotFoundError.
Switch to python -m src.deriver.deriver and explicitly set
WorkingDir=/app so module resolution is deterministic across Podman
versions.
Discovered during the first deploy of the honcho role on lincoln —
honcho-api was healthy on :8000 but the deriver crash-looped 26 times
in two minutes.
The half-disk LV pattern affects ~90% of mk-labs VMs. Treating the
fix-up as application-specific (as it was in day1_deploy_honcho.yml)
means future deploys would each carry the same boilerplate, and any
day1 author could forget it.
This commit:
* Adds playbooks/day0_expand_root_lv.yml — standalone day0 step,
targets {{ target | default("all") }}, honors a per-host
expand_root_lv_skip opt-out for multi-LV layouts.
* Adds playbooks/day0_provision.yml — umbrella playbook chaining
day0_linux_baseline + day0_expand_root_lv, so the operator runs
ONE command per new VM.
* Removes expand_root_lv from day1_deploy_honcho.yml — day0 is
assumed complete before day1 begins (cleaner separation of
concerns, matches the convention day1_deploy_semaphore already
follows).
* Updates the role README to document the lifecycle position and
the opt-out flag for hosts with multi-LV plans.
Reclaims the half-disk LV left by the Ubuntu Server autoinstall
template default. Idempotent — no-ops cleanly when there are no free PE
in the VG, and exits the play cleanly on hosts without LVM.
Supports ext4 and xfs. Does not handle partition resize (cloud-utils
growpart) — out of scope for the template fix-up case.
Wired into day1_deploy_honcho.yml ahead of the honcho role so newly
provisioned VMs get the fix-up automatically. Suitable to add to any
day1 playbook by simply listing it before the application role.
Deploys Honcho (plastic-labs/honcho) as a rootful Podman + Quadlet
service on the lincoln VM (10.1.71.132). Three containers on a
user-defined network:
- honcho-postgres pgvector/pgvector:pg16
- honcho-api FastAPI on :8000
- honcho-deriver background worker for theory-of-mind derivations
LLM provider: Anthropic Claude (claude-sonnet-4-5). Switching providers
is two env-var changes — see README.
Traefik route hall-of-presidents.local.mk-labs.cloud -> lincoln:8000
added under boilerplates/traefik/dynamic/. JARVIS itself talks to
Honcho directly at lincoln:8000 (east-west); the Traefik alias exists
only for browser access to the Swagger /docs UI.
Requires three new vault entries before first run:
- vault_honcho_database_password
- vault_honcho_jwt_secret
- vault_honcho_anthropic_api_key
Semaphore v2.18 source (db/TemplateVault.go) shows FillTemplateVault
only loads the key when type==TemplateVaultPassword. With type left
as the default empty string, the vault key is stored but never
materialised at task-run time; ansible-playbook gets no password and
fails with 'Attempting to decrypt but no vault secrets found'.
Confirmed by patching live templates via PUT with type='password';
next task run succeeded.
Semaphore v2.18 replaced the single vault_key_id field with a vaults[]
array supporting multi-vault per template. Sending vault_key_id is
silently ignored — template stores no vault association, runs fail
at the playbook stage with 'Attempting to decrypt but no vault
secrets found'.
Wrap the single configured vault password in the required envelope:
vaults: [{id: 0, vault_key_id: <key_id>, name: 'default'}]
Empirically verified: PUT with this shape returns 204 and the GET
echoes the vault back with a server-assigned id.
The 'uri' module with body_format=json was sending integer fields as
strings when they came from quoted Jinja expressions in inline YAML
(e.g. ssh_key_id: "{{ ... | int }}"), because YAML loads the quoted
form as a string and Ansible doesn't coerce back. Semaphore rejects
that with HTTP 400.
Build each body as a Jinja dict literal in a folded scalar so types
survive: integers stay integers, strings stay strings.
Also restores no_log: true on key creation now that we're past the
debug round.
Jinja interpreted semaphore_config.keys as the dict method rather than
the 'keys' field, causing 'Invalid data passed to loop' failure. Bracket
subscript disambiguates.
Adds an idempotent configuration pass that drives a freshly-deployed
Semaphore instance into its desired state via the REST API. Declared
in group_vars/all/semaphore.yml, applied by tasks/configure.yml,
toggled by semaphore_configure feature flag (default off).
Object types managed:
- Project (mk-labs)
- Keys (ansible-vault-pass, gitea-deploy, jarvis-ssh)
- Repositories (homelab on gitea)
- Inventories (production -> ansible/inventory.yml in homelab repo)
- Environments (default with ANSIBLE_HOST_KEY_CHECKING=False)
- Templates (day0_linux_baseline + variants, day1_deploy_semaphore)
with survey vars for runtime parameters
Each object found-or-created by name; existing ones never modified.
no_log on token-bearing calls to keep secrets out of stdout.
Inputs (already in vault):
vault_semaphore_api_token
vault_jarvis_ssh_private_key
vault_gitea_deploy_key
vault_ansible_vault_password
Quadlet regenerates the systemd unit on daemon-reload but does not
restart the running container — the process keeps its original
environment until restarted. Add explicit 'state: restarted' tasks
gated on the Quadlet template's changed status so env-var updates
actually take effect on re-runs.
The SPA was rendering <base href="https://semaphore.local.mk-labs.cloud/">
regardless of which hostname served the page, causing the imagineering
alias to load the same UI but rewrite all in-page navigation back to the
semaphore hostname. Confusing for users hitting either Traefik alias.
Leave SEMAPHORE_WEB_ROOT empty so Semaphore emits relative URLs and is
fully host-agnostic. Both DNS names (semaphore + imagineering) now serve
cleanly without preference between them.
The template only emits SEMAPHORE_WEB_ROOT when the variable is set, so
the env-var is absent from the container when not needed.
The v2.18 image's entrypoint runs the setup wizard on first boot. Without
the SEMAPHORE_ADMIN_* variables it prompts on stdin, fails with 'Username
cannot be empty', and the container exits — leading to a crash loop.
Set:
SEMAPHORE_ADMIN=admin
SEMAPHORE_ADMIN_NAME=Administrator
SEMAPHORE_ADMIN_EMAIL=admin@local.mk-labs.cloud
SEMAPHORE_ADMIN_PASSWORD={{ vault_semaphore_admin_password }}
SEMAPHORE_PLAYBOOK_PATH=/var/lib/semaphore/playbooks
The env-var bootstrap path is stable in v2.x; only the legacy
'semaphore user add' CLI invocation was unreliable. Drop the manual
user-add step from the README.
Complete rewrite of the semaphore role. Supersedes three prior
iterations whose admin-user-creation logic was unreliable across
Semaphore CLI versions.
Architecture:
- Rootful Podman Quadlet under /etc/containers/systemd/
- Separate PostgreSQL 16-alpine container on a user-defined
podman network (semaphore-net)
- Named volumes for both data stores (semaphore_data,
semaphore_postgres_data) so container recreation is
non-destructive
- Pinned image tags: semaphoreui/semaphore:v2.18.5-ansible2.16.5
and postgres:16-alpine
- Post-deploy HTTP health check fails the playbook if Semaphore
doesn't respond on /api/ping within ~60s
Admin user creation remains intentionally manual after first deploy;
the role README documents the exact podman exec command.
Removes the duplicate deploy_semaphore.yml and the now-unneeded
cleanup_semaphore.yml; day1_deploy_semaphore.yml is the canonical
entry point.
Previous template used inline arithmetic that miscounted the box
width, leaving the right border misaligned on real hostnames. Switch
to computed labels + a fixed inner_width so any hostname / OS string
pads to the same border position.
Introduces a single, idempotent baseline role to supersede the
overlapping day0-baseline and common roles. Capabilities are
feature-flagged so they can be toggled per-host:
- packages (common + OS-family + per-host extras)
- timezone + locale
- chrony time sync against sundial
- baseline users (jarvis admin account with SSH key + NOPASSWD sudo)
- SSH hardening via /etc/ssh/sshd_config.d/ drop-in
- unattended security upgrades (Debian family)
- sysctl drop-in at /etc/sysctl.d/99-mk-labs.conf
- journald retention caps
- branded MOTD
Ubuntu/Debian is first-class; vars/RedHat.yml provides a placeholder
for future distros via the ansible_os_family pattern.
The legacy day0-baseline and common roles remain in place for now and
will be removed during the playbook cleanup sweep, alongside the
existing playbook naming inconsistencies.
- Add quadlet tasks and handlers for modern rootless Podman deployment
- Fix broken Jinja in semaphore.container.j2
- Add proper 0777 permissions handling for rootless
- Support semaphore_use_postgres toggle with fallback to legacy BoltDB path
