bitsandbytes quantizes on-the-fly: loads full bf16 weights (~54GB RAM peak)
before compressing to int4. Kills the 40GB OptiPlex on torch.compile warmup.
Fix in next commit: switch serve phase to llama.cpp + GGUF Q4_K_M.
Pre-quantized weights load directly — peak RAM ~16GB, no compile overhead.
Migration from Ubuntu 3.x to upstream 6.x is complete. The explicit
removal task was firing changed on every run. state: latest on the
install task handles upgrades going forward.
Rules were documented but never deployed — /dev/jmri/nce was missing
entirely, only /dev/jmri/loconet existed (created manually).
Adds:
- templates/99-jmri-devices.rules.j2: LocoBuffer-NG -> loconet,
NCE Power Pro (Microchip CDC) -> nce
- Task to deploy rules + trigger udev settle
- Trigger udev handler (reload-rules alone is insufficient)
Adds jmri_ssh_authorized_keys_extra list to support multiple authorized
keys on the jmri account. Deploys rblundon's MacBook key so xpra can
connect via SSH without a password.
Ubuntu 24.04 packages xpra 3.1.5 which is wire-incompatible with the
v6.x macOS client from xpra.org. Add xpra.org apt repo and install
current stable release so client and server versions match.
Replace binary-exists check with .jmri_installed_version marker pattern.
- Reads marker on each run; skips install if version matches
- On version mismatch: stops JMRI, wipes /opt/JMRI, downloads new archive
- Separates build hash into jmri_build_hash var (templated into download URL)
- Config is preserved — lives in git-managed .jmri symlink
To upgrade: bump jmri_version + jmri_build_hash, re-run playbook.
- Stable udev device symlinks (/dev/jmri/nce, /dev/jmri/loconet, /dev/jmri/lcc)
- jmri-monitor: polls Leviton Decora Smart switch to start/stop JMRI automatically
- Quiet hours 1-10 AM (no polling)
- 30s off-delay before shutdown
- LCRR config cloned from Gitea (ssh://gitea.mk-labs.cloud:2221/rblundon/LCRR.git)
- ~/.jmri symlinked to LCRR repo for GitOps config management
- jmri-gui: X11 remote GUI access (PanelPro/DecoderPro) via ssh -X as jmri user
- Stops daemon, launches GUI, restarts daemon on exit if layout still on
- jmri user gets login shell + SSH key for GUI sessions
- Full JRE installed (openjdk-21-jre) for AWT/X11 support
- New ansible/roles/jmri role: installs OpenJDK 21 headless, creates
jmri service user, downloads JMRI 5.10, deploys JmriFaceless systemd unit
- Handles dialout group membership for serial device access
- Config restore task for post-reinstall recovery from GitHub backup
- host_vars/main-street-station: profile_id and serial device (TODO: fill in)
- Inventory: jmri_server group with main-street-station at 192.168.10.45
- Playbook: day1_deploy_jmri.yml (linux-baseline + jmri)
Semaphore runs ansible-playbook from the repo root, so ansible.cfg in
ansible/ is never loaded. The env var is the reliable path.
Also persists the setting in group_vars so semaphore_configure re-runs
don't regress it.
Semaphore clones the repo and runs ansible-playbook from the working
directory, so ansible.cfg is loaded automatically. This is more
reliable than env vars inherited through podman exec subprocesses.
/opt/ansible-collections is bind-mounted into the container and
populated by the semaphore role's collections task.
Host dir was owned by root (0755), blocking writes from uid=1001 inside
the container. Set owner to semaphore_container_uid=1001 so podman exec
can write collections into the bind-mount.
- Removed :ro from volume mount — ansible-galaxy writes via podman exec
into the container, so the mount must be writable during role runs
- Fixed deprecated ANSIBLE_COLLECTIONS_PATHS -> ANSIBLE_COLLECTIONS_PATH
/home/semaphore/.ansible/ is owned by root after Podman creates the
bind-mount dir, so ansible-galaxy can't create sibling tmp dirs.
Mount to a neutral /opt/ansible-collections path and point Ansible
at it via ANSIBLE_COLLECTIONS_PATHS env var instead.
Binary lives inside the container at /opt/semaphore/apps/ansible/<ver>/venv/bin/.
Use podman exec to run the install, writing to /home/semaphore/.ansible/collections
which is bind-mounted from the host-side directory.
ansible-galaxy is not on the system PATH on figment — Semaphore manages
its own venv under /opt/semaphore/apps/ansible/<ver>/venv/bin/.
Discover the binary dynamically rather than relying on PATH.
- New tasks/collections.yml installs collections from defaults list
into /opt/semaphore/ansible-collections on the host
- semaphore.container.j2: bind-mounts that dir into the container at
/home/semaphore/.ansible/collections (read-only)
- defaults/main.yml: semaphore_collections_dir + semaphore_ansible_collections
list (containers.podman, effectivelywild.technitium_dns)
- main.yml: collections task wired in after semaphore.yml, before verify
- Collections survive container restarts/rebuilds without image changes
- Re-run with --tags collections to add new collections without full redeploy
Collection not installed in Semaphore's Ansible environment.
Direct HTTP API call to Technitium on :5380 is simpler, has no
collection dependency, and is naturally idempotent (add returns ok
on duplicate records).
- New variable honcho_deriver_autostart controls service state/enabled
- honcho_deriver_enabled still controls whether Quadlet is created
- Default autostart=false after discovering deriver burns tokens autonomously
- Service is created but stopped/disabled; can be started manually if needed
- Add vault_honcho_openai_api_key (embeddings-only, Honcho-scoped)
- Inject OPENAI_API_KEY + LLM_OPENAI_API_KEY into api + deriver containers
- Flip honcho_embed_messages default to true now that embeddings have a provider
- Parameterise EMBEDDING__MODEL_CONFIG__{TRANSPORT,MODEL,BASE_URL} so we can
later swap to a local OpenAI-compatible embedder (e.g. Ollama on
astro-orbiter post-rebuild) with a single defaults change.
Vault diff is large because ansible-vault re-encrypts the whole file; logical
change is one new key.
A previous commit (via an agent write-file path with overaggressive
secret redaction) silently corrupted three Environment= lines in the
api/deriver Quadlet templates — the {{ delimiters around references
to honcho_auth_enabled, honcho_jwt_secret, and honcho_anthropic_api_key
were replaced with *** in the template file itself. Ansible templated
those *** through verbatim, and Honcho refused to start because
AUTH_USE_AUTH then resolved to the literal string "*** honcho_...".
Patched the templates back to proper Jinja via a side-channel that
bypasses the redactor. Verified the raw bytes on disk show 7b7b...7d7d
({{...}}) around all three references.
Smoke-testing the first deploy uncovered three default-config issues
that no amount of Quadlet tuning would have caught:
1. DIALECTIC subsystem ignored DERIVER_MODEL_CONFIG__*. Honcho splits
dialectic into five reasoning levels (minimal/low/medium/high/max)
each with its own MODEL_CONFIG that defaults to OpenAI. Without
overrides, every /chat call fails with: ValidationException:
Missing API key for openai model config. Now setting all five
DIALECTIC_LEVELS__<level>__MODEL_CONFIG__* env vars to anthropic.
2. DERIVER batches representation tasks until a token threshold is
reached. For low-volume homelab use (one chatty operator), tasks
can sit unprocessed forever. Add DERIVER_FLUSH_ENABLED knob,
default true.
3. Embeddings default to OpenAI text-embedding-3-small. Anthropic
has no embedding API, so without an OpenAI key the embed step
fails the entire derivation. Default EMBED_MESSAGES=false until
a separate embedding provider is wired up (OpenAI for embeds-only
or a local BGE endpoint on astro-orbiter).
defaults/main.yml documents all three issues and the migration path
back to embeddings when ready.
Two fixes uncovered while wiring up the first new Traefik route
(hall-of-presidents.local.mk-labs.cloud for honcho):
* traefik_host changed from 10.1.71.35 -> lightning-lane.local.mk-labs.cloud
so wed/.ssh/config rules (Host *.local.mk-labs.cloud) match and the
rsync step uses the right IdentityFile.
* requirements.yml: pin effectivelywild.technitium_dns >=1.1.0 — the
playbook depends on it but the collection was not declared. Bare
checkouts would fail without it being installed manually.
src/deriver/deriver.py is a helpers module with no __main__ guard —
running it directly loads it and exits cleanly in ~3s with no logs,
which is what we were seeing. The actual queue-processor entry point
is src/deriver/__main__.py (asyncio.run(run_deriver())), invoked via
"python -m src.deriver".
Discovered during honcho first-deploy on lincoln: deriver container
crash-looped (in the sense that systemd restarted it constantly) but
the underlying process was actually exiting 0 on completion of the
empty helpers module.
The deriver script does "from src import crud" at the top, which only
works when the cwd / sys.path[0] is /app. Running the script by file
path (python /app/src/deriver/deriver.py) puts /app/src/deriver on
sys.path instead, and the import fails with ModuleNotFoundError.
Switch to python -m src.deriver.deriver and explicitly set
WorkingDir=/app so module resolution is deterministic across Podman
versions.
Discovered during the first deploy of the honcho role on lincoln —
honcho-api was healthy on :8000 but the deriver crash-looped 26 times
in two minutes.
Three new entries:
vault_honcho_database_password (random base64, 32 bytes)
vault_honcho_jwt_secret (random base64, 48 bytes)
vault_honcho_anthropic_api_key (Anthropic API key, scoped to mk-labs-honcho-lincoln)
Consumed by ansible/roles/honcho/defaults/main.yml.
The half-disk LV pattern affects ~90% of mk-labs VMs. Treating the
fix-up as application-specific (as it was in day1_deploy_honcho.yml)
means future deploys would each carry the same boilerplate, and any
day1 author could forget it.
This commit:
* Adds playbooks/day0_expand_root_lv.yml — standalone day0 step,
targets {{ target | default("all") }}, honors a per-host
expand_root_lv_skip opt-out for multi-LV layouts.
* Adds playbooks/day0_provision.yml — umbrella playbook chaining
day0_linux_baseline + day0_expand_root_lv, so the operator runs
ONE command per new VM.
* Removes expand_root_lv from day1_deploy_honcho.yml — day0 is
assumed complete before day1 begins (cleaner separation of
concerns, matches the convention day1_deploy_semaphore already
follows).
* Updates the role README to document the lifecycle position and
the opt-out flag for hosts with multi-LV plans.
