Platform Knowledge Infrastructure pilot - context database for large
file trees, shared skills, and long-term logs to reduce agent token
consumption. Pilot scope: two corpora (hermes/ skills library,
personal/homelab/), two consumer profiles (Wong, Shuri) for before/after
token comparison.
- namespace.yaml: openviking namespace, sync-wave 8 (after Harbor wave 7)
- externalsecret.yaml: credentials from 1Password via onepassword-connect
ClusterSecretStore (Wong, t_32766900)
- values.yaml: Helm overrides - px-fa-direct-access storage (30Gi),
embedding (nomic-embed-text-v1.5) + VLM (Llama-3.1-8B) via astro-orbiter
router (:8002), internal-only ingress
- application.yaml: multi-source ArgoCD Application, Harbor pattern
(Peter Parker, t_eefdcc17 + reconciled in t_3e54efa8)
Prerequisites verified complete before this commit:
- nomic-embed-text-v1.5-Q4_K_M live on astro-orbiter router (War Machine,
t_34b96e83, commit ad70b34)
- All 3 1Password items provisioned (root/embedding/vlm api keys)
- Storage class corrected to px-fa-direct-access after live PV audit
showed pure-block/pure-file have zero provisioned volumes (t_77b3ff79)
- Dry-run validated against live cluster prior to commit
Constraint: vault (~/friday) remains canonical source of truth; OpenViking
index is a derived cache, rebuilt from vault source files.
Honcho/lincoln explicitly out of scope for this work.
/metrics?model=Qwen3.6 forces the router to attempt loading Qwen3.6 every
90s scrape cycle, triggering a CUDA OOM error since VRAM is already consumed
by the resident Llama3+Phi3.5 models. This produces real GPU power spikes
(~110W load-attempt), not a benign counter read like the Llama3/Phi3.5 jobs.
nvidia_gpu_exporter (:9835) already provides GPU power/VRAM/utilization at
zero wake cost. No Grafana dashboard panel references Qwen3.6 model-specific
llama-server metrics. Ryan approved full removal.
Job is commented out (not deleted) for easy revert if Qwen3.6 is ever
re-added as a resident model. Llama3/Phi3.5 scrape jobs untouched.
Each /metrics?model=<id> request on the llama.cpp router wakes the GPU
sub-server to P2 (~110W). At 15s with two active jobs (llama3 + phi35),
combined scrape frequency (~7-8s effective) keeps the GPU continuously at
P2 despite zero real inference requests.
At 90s: each scrape wakes GPU for ~5-10s then it drops to P8 (~20W) for
~80s. Verified via iptables block test on 2026-08-13 (t_e7d547ea):
Before block: 110-115W P2 continuously
After block: 19-21W P8 consistently
After unblock: returned to 110W P2 within seconds
qwen3 interval also set to 90s (model not loaded so moot, but consistent).
Ref: t_e7d547ea
Port 8000 (gemma-2-27b-it-GGUF) is dead after the day2 router cutover on
2026-08-12. Production inference now runs through llama-server-router on
port 8002.
The router exposes per-model Prometheus metrics via /metrics?model=<id>.
Since a single /metrics request without ?model returns HTTP 400, replaced
the single stale job with three per-model jobs — one per model registered
in the router per /v1/models:
- Qwen3.6-35B-A3B-UD-Q4_K_S (currently unloaded but registered)
- Meta-Llama-3.1-8B-Instruct-Q4_K_M (loaded)
- Phi-3.5-mini-instruct-Q8_0 (loaded)
Static 'model' label carries the canonical llama.cpp model id (not alias).
Added 'endpoint: astro-orbiter-router' to identify the scrape origin.
Removed dead :8000 target entirely.
- host_vars/astro-orbiter/vars.yml: add Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf
to llm_staged_models (size_bytes=8988111072, bartowski GGUF public repo).
Updated VRAM note to reflect 4-model roster and LRU eviction semantics.
- defaults/main.yml: add llm_router_coder_ctx_size=16384 and
llm_router_coder_flash_attn=true variables for per-model ctx tuning.
- templates/llama-server-router-preset.ini.j2: add [Qwen2.5-Coder-14B-Instruct-Q4_K_M]
section with alias=Qwen2.5-Coder-14B-Instruct-4bit, ctx-size=16384, flash-attn=true.
- playbooks/day2_add_coder_alias.yml: new playbook that downloads the GGUF (if
absent/mismatched), deploys updated preset INI and systemd unit, restarts
llama-server-router, and verifies all 4 models in /v1/models.
VRAM: Coder ~9GB. Full 4-model co-residency impossible on 24GB — LRU eviction
handles this automatically. Qwen3.6-35B <-> Coder switches incur ~30-60s cold load.
- Add Meta-Llama-3.1-8B-Instruct-4bit alias on Meta-Llama-3.1-8B-Instruct-Q4_K_M
entry in the preset INI (per Ryan dashboard note).
- Document known issue: Phi-3.5-mini json_schema grammar sampler incompatibility
in router mode (GH #23460 variant — chat template token format mismatch).
Meta-Llama works with json_schema response_format; confirmed via live test.
- Phi-3.5-mini-instruct-8bit alias remains working for model routing;
structured output (json_schema) fails due to the model's token format.
Changes:
- host_vars/astro-orbiter/vars.yml: add llm_router_models_max: 4 (overrides
conservative default of 1). Detailed VRAM OOM risk note included inline:
worst-case 3-model co-residency ~31GB > 24GB RTX 3090. LRU eviction
mitigates in single-user operation; flagged for Ryan's review.
- playbooks/day2_bump_router_models_max.yml: new targeted playbook; deploys
updated router unit, restarts the live service, verifies /health 200 and
/v1/models lists all 3 GGUFs post-restart.
- group_vars/all/semaphore.yml: add llm_router_update_unit template pointing
at the new playbook.
- roles/llm-inference-multimodel/defaults/main.yml: update comment to reflect
the var is now overridden in host_vars rather than 'hardcoded to 1'.
- roles/llm-inference-multimodel/templates/llama-server-router.service.j2:
correct stale 'HARDCODED TO 1' comment — value is variable-driven.
Constraints honored:
- --parallel 1 left untouched (not in scope, not modified anywhere)
- No ad-hoc SSH/systemctl/curl state mutation; all execution via Semaphore
- No installed/vendored code patched
Adds idempotent, data-driven GGUF staging for the two new router models on
astro-orbiter alongside the production Qwen3.6-35B-A3B-UD-Q4_K_S. Both files
were already staged live (byte-exact); this commit codifies them in Ansible so
future re-runs and any new model adds are version-controlled and audit-friendly.
Changes:
- roles/llm-inference-multimodel/tasks/stage_model.yml (NEW)
Idempotent per-model task: stat -> exact byte-size guard -> conditional
get_url -> ownership/mode ensure -> notify router restart handler only on
actual download. Loops from models.yml; nothing hardcoded.
- roles/llm-inference-multimodel/tasks/models.yml
Appends the stage_model.yml loop (tagged: models) after the existing Qwen3.6
download tasks. Data driven from host_vars/astro-orbiter/vars.yml.
- roles/llm-inference-multimodel/defaults/main.yml
Adds llm_staged_models: [] default (empty = safe no-op for hosts with no
staged model list defined).
- roles/llm-inference-multimodel/handlers/main.yml
Adds 'restart llama-server-router on new GGUF' handler. Only fires when
stage_model.yml performs an actual download or corrects ownership/mode.
Normal idempotent re-runs (files already correct) do NOT fire this handler.
- host_vars/astro-orbiter/vars.yml
Adds llm_staged_models list with the two new models:
* Phi-3.5-mini-instruct-Q8_0.gguf (4,061,222,688 bytes,
bartowski/Phi-3.5-mini-instruct-GGUF)
* Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf (4,920,739,232 bytes,
bartowski/Meta-Llama-3.1-8B-Instruct-GGUF)
- playbooks/day1_deploy_llm_inference_multimodel.yml
Updates header comment: removes stale 'Semaphore broken' note, documents
the correct execution channel (Semaphore template
llm_inference_multimodel_stage_models, --tags models).
- group_vars/all/semaphore.yml
Adds llm_inference_multimodel_stage_models template entry (config-as-code).
Template is scoped to --tags models explicitly. Phase 4 (verify) is
EXCLUDED: verify.yml starts llama-server-qwen on :8002, which would collide
with the production llama-server-router.service already running on :8002.
Semaphore template created via API: project 1 / template id 19.
Execution: triggered immediately after this commit via Semaphore REST API.
Promote llama-server-router to production on port 8002.
- Stops + disables llama-server-qwen (unit preserved as rollback target)
- Redeploys llama-server-router unit with --port 8002 (not 8003)
- PORT DECISION: rebind router to :8002 — zero Hermes profile config changes needed
- Re-runs validation gates 1-3 against production endpoint (hard gates)
- Gate 4: bundled SvelteKit UI check (HTTP 415 from ansible.builtin.uri is
a false negative — llama.cpp requires Accept-Encoding: gzip; real browsers work)
- Rollback via --tags cutover_rollback (uses 'never' Ansible tag to prevent
accidental execution during normal cutover run)
All 4 gates PASSED on cutover run 2026-08-12:
Gate 1: n_ctx=65536 >= 64000 PASS
Gate 2: finish_reason=tool_calls PASS
Gate 2b: finish_reason=stop (no spurious tool_calls) PASS
Gate 3: 20410 MiB / 23000 MiB ceiling PASS
Gate 4: SvelteKit HTML confirmed via curl + gunzip PASS
Production endpoint: llama-server-router on :8002 (enabled + running)
Rollback target: llama-server-qwen unit at /etc/systemd/system/ (stopped, disabled)
- Retire llama-server-aux (Phi-4, 8000) and llama-server-toolcall (Mistral-Small-24B, 8001): stopped, disabled, unit files removed from host and Ansible role
- Promote llama-server-qwen (Qwen2.5-14B-Instruct-1M, port 8002) to sole production model, serving both friday and war-machine Hermes profiles
- Verified live: n_ctx=65536/n_ctx_train=1010000, and tool_calls response via /v1/chat/completions probe (no hallucination)
- Deleted superseded GGUF weights (phi-4, mistral-small, orphaned base-Qwen, gemma-2-27b) from astro-orbiter, ~45GB reclaimed
- Updated friday and war-machine Hermes profile configs (model + compression + skills_hub aux) to point at 10.1.71.130:8002
- Ryan explicitly accepted single-model tradeoffs for both profiles
Mistral (llama-server-toolcall, 8001) stopped temporarily on astro-orbiter to
free ~6.2GB VRAM headroom for this test window per Ryan/JARVIS approval.
Not a permanent decommission of Mistral.
- New llama-server-qwen systemd unit template, gated by llm_qwen_service_enabled (default false)
- Idempotent GGUF download task (bartowski Qwen2.5-14B-Instruct-Q5_K_M, stat-guarded)
- Launch flags per local-llm-64k-context-recommendation.md: ctx-size 65536, flash-attn, q8_0 KV cache, batch 2048/ubatch 512, jinja, parallel 1
- verify.yml only starts/verifies the qwen unit when llm_qwen_service_enabled=true
- README: documents live VRAM gate finding (nvidia-smi 2026-08-06: Phi-4+Mistral already ~16.6/24GB, ~7.5GB free -- insufficient for Qwen weights concurrently) and options
- Does NOT touch llama-server-aux (8000) or llama-server-toolcall (8001) service state
discover.yml sets llm_existing_gemma_unit_found, but main.yml imports each
phase file with import_tasks + a distinct per-phase tag. Tags on
import_tasks apply to the whole file, so --tags verify (a supported,
documented way to re-run just this phase) skips discover.yml, leaving
the fact undefined. The stop task's 'default(false)' silently no-op'd,
so re-running verify alone against a host with Gemma still running would
start both new instances on top of it -- the OOM this task exists to
prevent.
Fix: gather service_facts and set the fact locally in verify.yml too,
only when not already defined, so the guard works regardless of which
tags were selected.
Phase 2 (systemd tag) notified per-service restart handlers and then
called meta: flush_handlers itself, so any run where either unit's
template content changed (including first apply) restarted BOTH
live services immediately in Phase 2 -- before Phase 3 firewall
scoping or Phase 4 smoke tests ran. This contradicted the phase's
documented purpose (units land on disk only, nothing starts/restarts
until Phase 4).
Fix: Phase 2 only reloads the systemd daemon and registers each
template task's changed result. Phase 4 (verify.yml) now decides
start vs restart per-service based on that recorded change, so
restarts remain independent per instance and never fire before
Phase 4.
- Prometheus scrape configs for node/gpu/llama-server exporters on
astro-orbiter now declared in cluster/applications/monitoring/values.yaml
(additionalScrapeConfigs), applied via ArgoCD sync instead of an
imperative kubectl secret patch from the Ansible role.
- Grafana dashboard for astro-orbiter LLM inference added as a ConfigMap
in cluster/applications/monitoring/dashboards.yaml (grafana_dashboard=1
sidecar label), replacing the role's ad-hoc kubectl apply of a rendered
Jinja template.
- ansible/roles/llm-inference/tasks/monitoring.yml: removed the kubectl
get/patch/apply tasks and orphaned grafana-llm-dashboard.json.j2
template; role now only stands up node_exporter + nvidia_gpu_exporter
and verifies they're reachable — cluster-facing config lives in Git.
- host_vars/vars.yml + inventory.yml: finalize astro-orbiter as the
llama.cpp/RTX 3090 host (jarvis user, ssh key), drop stale Ollama/AMD
vars and ollama_server inventory group superseded by the ATX rebuild.
bitsandbytes quantizes on-the-fly: loads full bf16 weights (~54GB RAM peak)
before compressing to int4. Kills the 40GB OptiPlex on torch.compile warmup.
Fix in next commit: switch serve phase to llama.cpp + GGUF Q4_K_M.
Pre-quantized weights load directly — peak RAM ~16GB, no compile overhead.
Migration from Ubuntu 3.x to upstream 6.x is complete. The explicit
removal task was firing changed on every run. state: latest on the
install task handles upgrades going forward.
Rules were documented but never deployed — /dev/jmri/nce was missing
entirely, only /dev/jmri/loconet existed (created manually).
Adds:
- templates/99-jmri-devices.rules.j2: LocoBuffer-NG -> loconet,
NCE Power Pro (Microchip CDC) -> nce
- Task to deploy rules + trigger udev settle
- Trigger udev handler (reload-rules alone is insufficient)
Adds jmri_ssh_authorized_keys_extra list to support multiple authorized
keys on the jmri account. Deploys rblundon's MacBook key so xpra can
connect via SSH without a password.
Ubuntu 24.04 packages xpra 3.1.5 which is wire-incompatible with the
v6.x macOS client from xpra.org. Add xpra.org apt repo and install
current stable release so client and server versions match.
Replace binary-exists check with .jmri_installed_version marker pattern.
- Reads marker on each run; skips install if version matches
- On version mismatch: stops JMRI, wipes /opt/JMRI, downloads new archive
- Separates build hash into jmri_build_hash var (templated into download URL)
- Config is preserved — lives in git-managed .jmri symlink
To upgrade: bump jmri_version + jmri_build_hash, re-run playbook.
- Stable udev device symlinks (/dev/jmri/nce, /dev/jmri/loconet, /dev/jmri/lcc)
- jmri-monitor: polls Leviton Decora Smart switch to start/stop JMRI automatically
- Quiet hours 1-10 AM (no polling)
- 30s off-delay before shutdown
- LCRR config cloned from Gitea (ssh://gitea.mk-labs.cloud:2221/rblundon/LCRR.git)
- ~/.jmri symlinked to LCRR repo for GitOps config management
- jmri-gui: X11 remote GUI access (PanelPro/DecoderPro) via ssh -X as jmri user
- Stops daemon, launches GUI, restarts daemon on exit if layout still on
- jmri user gets login shell + SSH key for GUI sessions
- Full JRE installed (openjdk-21-jre) for AWT/X11 support
