boilerplates/traefik/dynamic/ files are rsynced verbatim to lightning-lane,
not rendered through Jinja first. Traefik also processes Go-style
{{ }} templates, so a leftover {{ ansible_managed }} comment line caused
Traefik to reject the entire file with:
ERR /etc/traefik/dynamic/honcho.yml: template: :1: function
"ansible_managed" not defined
Other dynamic configs in this directory deliberately omit the marker;
matching that convention.
Two fixes uncovered while wiring up the first new Traefik route
(hall-of-presidents.local.mk-labs.cloud for honcho):
* traefik_host changed from 10.1.71.35 -> lightning-lane.local.mk-labs.cloud
so wed/.ssh/config rules (Host *.local.mk-labs.cloud) match and the
rsync step uses the right IdentityFile.
* requirements.yml: pin effectivelywild.technitium_dns >=1.1.0 — the
playbook depends on it but the collection was not declared. Bare
checkouts would fail without it being installed manually.
src/deriver/deriver.py is a helpers module with no __main__ guard —
running it directly loads it and exits cleanly in ~3s with no logs,
which is what we were seeing. The actual queue-processor entry point
is src/deriver/__main__.py (asyncio.run(run_deriver())), invoked via
"python -m src.deriver".
Discovered during honcho first-deploy on lincoln: deriver container
crash-looped (in the sense that systemd restarted it constantly) but
the underlying process was actually exiting 0 on completion of the
empty helpers module.
The deriver script does "from src import crud" at the top, which only
works when the cwd / sys.path[0] is /app. Running the script by file
path (python /app/src/deriver/deriver.py) puts /app/src/deriver on
sys.path instead, and the import fails with ModuleNotFoundError.
Switch to python -m src.deriver.deriver and explicitly set
WorkingDir=/app so module resolution is deterministic across Podman
versions.
Discovered during the first deploy of the honcho role on lincoln —
honcho-api was healthy on :8000 but the deriver crash-looped 26 times
in two minutes.
Three new entries:
vault_honcho_database_password (random base64, 32 bytes)
vault_honcho_jwt_secret (random base64, 48 bytes)
vault_honcho_anthropic_api_key (Anthropic API key, scoped to mk-labs-honcho-lincoln)
Consumed by ansible/roles/honcho/defaults/main.yml.
The half-disk LV pattern affects ~90% of mk-labs VMs. Treating the
fix-up as application-specific (as it was in day1_deploy_honcho.yml)
means future deploys would each carry the same boilerplate, and any
day1 author could forget it.
This commit:
* Adds playbooks/day0_expand_root_lv.yml — standalone day0 step,
targets {{ target | default("all") }}, honors a per-host
expand_root_lv_skip opt-out for multi-LV layouts.
* Adds playbooks/day0_provision.yml — umbrella playbook chaining
day0_linux_baseline + day0_expand_root_lv, so the operator runs
ONE command per new VM.
* Removes expand_root_lv from day1_deploy_honcho.yml — day0 is
assumed complete before day1 begins (cleaner separation of
concerns, matches the convention day1_deploy_semaphore already
follows).
* Updates the role README to document the lifecycle position and
the opt-out flag for hosts with multi-LV plans.
Reclaims the half-disk LV left by the Ubuntu Server autoinstall
template default. Idempotent — no-ops cleanly when there are no free PE
in the VG, and exits the play cleanly on hosts without LVM.
Supports ext4 and xfs. Does not handle partition resize (cloud-utils
growpart) — out of scope for the template fix-up case.
Wired into day1_deploy_honcho.yml ahead of the honcho role so newly
provisioned VMs get the fix-up automatically. Suitable to add to any
day1 playbook by simply listing it before the application role.
Deploys Honcho (plastic-labs/honcho) as a rootful Podman + Quadlet
service on the lincoln VM (10.1.71.132). Three containers on a
user-defined network:
- honcho-postgres pgvector/pgvector:pg16
- honcho-api FastAPI on :8000
- honcho-deriver background worker for theory-of-mind derivations
LLM provider: Anthropic Claude (claude-sonnet-4-5). Switching providers
is two env-var changes — see README.
Traefik route hall-of-presidents.local.mk-labs.cloud -> lincoln:8000
added under boilerplates/traefik/dynamic/. JARVIS itself talks to
Honcho directly at lincoln:8000 (east-west); the Traefik alias exists
only for browser access to the Swagger /docs UI.
Requires three new vault entries before first run:
- vault_honcho_database_password
- vault_honcho_jwt_secret
- vault_honcho_anthropic_api_key
JARVIS