Phase 2 (systemd tag) notified per-service restart handlers and then
called meta: flush_handlers itself, so any run where either unit's
template content changed (including first apply) restarted BOTH
live services immediately in Phase 2 -- before Phase 3 firewall
scoping or Phase 4 smoke tests ran. This contradicted the phase's
documented purpose (units land on disk only, nothing starts/restarts
until Phase 4).
Fix: Phase 2 only reloads the systemd daemon and registers each
template task's changed result. Phase 4 (verify.yml) now decides
start vs restart per-service based on that recorded change, so
restarts remain independent per instance and never fire before
Phase 4.
- Prometheus scrape configs for node/gpu/llama-server exporters on
astro-orbiter now declared in cluster/applications/monitoring/values.yaml
(additionalScrapeConfigs), applied via ArgoCD sync instead of an
imperative kubectl secret patch from the Ansible role.
- Grafana dashboard for astro-orbiter LLM inference added as a ConfigMap
in cluster/applications/monitoring/dashboards.yaml (grafana_dashboard=1
sidecar label), replacing the role's ad-hoc kubectl apply of a rendered
Jinja template.
- ansible/roles/llm-inference/tasks/monitoring.yml: removed the kubectl
get/patch/apply tasks and orphaned grafana-llm-dashboard.json.j2
template; role now only stands up node_exporter + nvidia_gpu_exporter
and verifies they're reachable — cluster-facing config lives in Git.
- host_vars/vars.yml + inventory.yml: finalize astro-orbiter as the
llama.cpp/RTX 3090 host (jarvis user, ssh key), drop stale Ollama/AMD
vars and ollama_server inventory group superseded by the ATX rebuild.
bitsandbytes quantizes on-the-fly: loads full bf16 weights (~54GB RAM peak)
before compressing to int4. Kills the 40GB OptiPlex on torch.compile warmup.
Fix in next commit: switch serve phase to llama.cpp + GGUF Q4_K_M.
Pre-quantized weights load directly — peak RAM ~16GB, no compile overhead.
Migration from Ubuntu 3.x to upstream 6.x is complete. The explicit
removal task was firing changed on every run. state: latest on the
install task handles upgrades going forward.
Rules were documented but never deployed — /dev/jmri/nce was missing
entirely, only /dev/jmri/loconet existed (created manually).
Adds:
- templates/99-jmri-devices.rules.j2: LocoBuffer-NG -> loconet,
NCE Power Pro (Microchip CDC) -> nce
- Task to deploy rules + trigger udev settle
- Trigger udev handler (reload-rules alone is insufficient)
Adds jmri_ssh_authorized_keys_extra list to support multiple authorized
keys on the jmri account. Deploys rblundon's MacBook key so xpra can
connect via SSH without a password.
Ubuntu 24.04 packages xpra 3.1.5 which is wire-incompatible with the
v6.x macOS client from xpra.org. Add xpra.org apt repo and install
current stable release so client and server versions match.
Replace binary-exists check with .jmri_installed_version marker pattern.
- Reads marker on each run; skips install if version matches
- On version mismatch: stops JMRI, wipes /opt/JMRI, downloads new archive
- Separates build hash into jmri_build_hash var (templated into download URL)
- Config is preserved — lives in git-managed .jmri symlink
To upgrade: bump jmri_version + jmri_build_hash, re-run playbook.
- Stable udev device symlinks (/dev/jmri/nce, /dev/jmri/loconet, /dev/jmri/lcc)
- jmri-monitor: polls Leviton Decora Smart switch to start/stop JMRI automatically
- Quiet hours 1-10 AM (no polling)
- 30s off-delay before shutdown
- LCRR config cloned from Gitea (ssh://gitea.mk-labs.cloud:2221/rblundon/LCRR.git)
- ~/.jmri symlinked to LCRR repo for GitOps config management
- jmri-gui: X11 remote GUI access (PanelPro/DecoderPro) via ssh -X as jmri user
- Stops daemon, launches GUI, restarts daemon on exit if layout still on
- jmri user gets login shell + SSH key for GUI sessions
- Full JRE installed (openjdk-21-jre) for AWT/X11 support
- New ansible/roles/jmri role: installs OpenJDK 21 headless, creates
jmri service user, downloads JMRI 5.10, deploys JmriFaceless systemd unit
- Handles dialout group membership for serial device access
- Config restore task for post-reinstall recovery from GitHub backup
- host_vars/main-street-station: profile_id and serial device (TODO: fill in)
- Inventory: jmri_server group with main-street-station at 192.168.10.45
- Playbook: day1_deploy_jmri.yml (linux-baseline + jmri)
Semaphore runs ansible-playbook from the repo root, so ansible.cfg in
ansible/ is never loaded. The env var is the reliable path.
Also persists the setting in group_vars so semaphore_configure re-runs
don't regress it.
Semaphore clones the repo and runs ansible-playbook from the working
directory, so ansible.cfg is loaded automatically. This is more
reliable than env vars inherited through podman exec subprocesses.
/opt/ansible-collections is bind-mounted into the container and
populated by the semaphore role's collections task.
Host dir was owned by root (0755), blocking writes from uid=1001 inside
the container. Set owner to semaphore_container_uid=1001 so podman exec
can write collections into the bind-mount.
- Removed :ro from volume mount — ansible-galaxy writes via podman exec
into the container, so the mount must be writable during role runs
- Fixed deprecated ANSIBLE_COLLECTIONS_PATHS -> ANSIBLE_COLLECTIONS_PATH
/home/semaphore/.ansible/ is owned by root after Podman creates the
bind-mount dir, so ansible-galaxy can't create sibling tmp dirs.
Mount to a neutral /opt/ansible-collections path and point Ansible
at it via ANSIBLE_COLLECTIONS_PATHS env var instead.
Binary lives inside the container at /opt/semaphore/apps/ansible/<ver>/venv/bin/.
Use podman exec to run the install, writing to /home/semaphore/.ansible/collections
which is bind-mounted from the host-side directory.
ansible-galaxy is not on the system PATH on figment — Semaphore manages
its own venv under /opt/semaphore/apps/ansible/<ver>/venv/bin/.
Discover the binary dynamically rather than relying on PATH.
- New tasks/collections.yml installs collections from defaults list
into /opt/semaphore/ansible-collections on the host
- semaphore.container.j2: bind-mounts that dir into the container at
/home/semaphore/.ansible/collections (read-only)
- defaults/main.yml: semaphore_collections_dir + semaphore_ansible_collections
list (containers.podman, effectivelywild.technitium_dns)
- main.yml: collections task wired in after semaphore.yml, before verify
- Collections survive container restarts/rebuilds without image changes
- Re-run with --tags collections to add new collections without full redeploy
Collection not installed in Semaphore's Ansible environment.
Direct HTTP API call to Technitium on :5380 is simpler, has no
collection dependency, and is naturally idempotent (add returns ok
on duplicate records).
- New variable honcho_deriver_autostart controls service state/enabled
- honcho_deriver_enabled still controls whether Quadlet is created
- Default autostart=false after discovering deriver burns tokens autonomously
- Service is created but stopped/disabled; can be started manually if needed
- Add vault_honcho_openai_api_key (embeddings-only, Honcho-scoped)
- Inject OPENAI_API_KEY + LLM_OPENAI_API_KEY into api + deriver containers
- Flip honcho_embed_messages default to true now that embeddings have a provider
- Parameterise EMBEDDING__MODEL_CONFIG__{TRANSPORT,MODEL,BASE_URL} so we can
later swap to a local OpenAI-compatible embedder (e.g. Ollama on
astro-orbiter post-rebuild) with a single defaults change.
Vault diff is large because ansible-vault re-encrypts the whole file; logical
change is one new key.
A previous commit (via an agent write-file path with overaggressive
secret redaction) silently corrupted three Environment= lines in the
api/deriver Quadlet templates — the {{ delimiters around references
to honcho_auth_enabled, honcho_jwt_secret, and honcho_anthropic_api_key
were replaced with *** in the template file itself. Ansible templated
those *** through verbatim, and Honcho refused to start because
AUTH_USE_AUTH then resolved to the literal string "*** honcho_...".
Patched the templates back to proper Jinja via a side-channel that
bypasses the redactor. Verified the raw bytes on disk show 7b7b...7d7d
({{...}}) around all three references.
Smoke-testing the first deploy uncovered three default-config issues
that no amount of Quadlet tuning would have caught:
1. DIALECTIC subsystem ignored DERIVER_MODEL_CONFIG__*. Honcho splits
dialectic into five reasoning levels (minimal/low/medium/high/max)
each with its own MODEL_CONFIG that defaults to OpenAI. Without
overrides, every /chat call fails with: ValidationException:
Missing API key for openai model config. Now setting all five
DIALECTIC_LEVELS__<level>__MODEL_CONFIG__* env vars to anthropic.
2. DERIVER batches representation tasks until a token threshold is
reached. For low-volume homelab use (one chatty operator), tasks
can sit unprocessed forever. Add DERIVER_FLUSH_ENABLED knob,
default true.
3. Embeddings default to OpenAI text-embedding-3-small. Anthropic
has no embedding API, so without an OpenAI key the embed step
fails the entire derivation. Default EMBED_MESSAGES=false until
a separate embedding provider is wired up (OpenAI for embeds-only
or a local BGE endpoint on astro-orbiter).
defaults/main.yml documents all three issues and the migration path
back to embeddings when ready.
