- Remove global --n-gpu-layers from router unit ExecStart in preset mode
(llama.cpp CLI arg outranked per-model INI n-gpu-layers=0; root cause from
War Machine's run 1). Flag now emitted only in --models-dir mode.
- All 5 preset INI sections carry explicit n-gpu-layers:
Qwen3.8=99, Phi=99, nomic=99, Coder=0, Llama=0.
- host_vars/astro-orbiter: llm_router_models_max 2 -> 4 so CPU-offloaded
models count as loaded without LRU-evicting Qwen3.8.
- defaults: llm_router_coder_gpu_layers / llm_router_llama_gpu_layers = 0.
- verify.yml: fix pre-existing .meta attribute crash in router mode.
- New playbook day2_cpu_offload_aux_models.yml.
Deployed + verified on astro-orbiter (gates A-E PASS): concurrent residency
achieved, Qwen3.8 stays GPU-resident. Measured CPU throughput Llama 9.0 /
Coder 4.7 tok/s. VRAM note: llama.cpp 6ea215d allocates ~1.4-1.7GB CUDA-context
per CPU model even at n-gpu-layers=0 -> ~24,004 MiB steady-state, below the
24,576 MiB physical limit. Comments corrected to match the measurement.
Report: friday/inbox/ryan/2026-08-17-llm-cpu-offload-coder-llama-deployed.md
- Retire llama-server-aux (Phi-4, 8000) and llama-server-toolcall (Mistral-Small-24B, 8001): stopped, disabled, unit files removed from host and Ansible role
- Promote llama-server-qwen (Qwen2.5-14B-Instruct-1M, port 8002) to sole production model, serving both friday and war-machine Hermes profiles
- Verified live: n_ctx=65536/n_ctx_train=1010000, and tool_calls response via /v1/chat/completions probe (no hallucination)
- Deleted superseded GGUF weights (phi-4, mistral-small, orphaned base-Qwen, gemma-2-27b) from astro-orbiter, ~45GB reclaimed
- Updated friday and war-machine Hermes profile configs (model + compression + skills_hub aux) to point at 10.1.71.130:8002
- Ryan explicitly accepted single-model tradeoffs for both profiles
- New llama-server-qwen systemd unit template, gated by llm_qwen_service_enabled (default false)
- Idempotent GGUF download task (bartowski Qwen2.5-14B-Instruct-Q5_K_M, stat-guarded)
- Launch flags per local-llm-64k-context-recommendation.md: ctx-size 65536, flash-attn, q8_0 KV cache, batch 2048/ubatch 512, jinja, parallel 1
- verify.yml only starts/verifies the qwen unit when llm_qwen_service_enabled=true
- README: documents live VRAM gate finding (nvidia-smi 2026-08-06: Phi-4+Mistral already ~16.6/24GB, ~7.5GB free -- insufficient for Qwen weights concurrently) and options
- Does NOT touch llama-server-aux (8000) or llama-server-toolcall (8001) service state
discover.yml sets llm_existing_gemma_unit_found, but main.yml imports each
phase file with import_tasks + a distinct per-phase tag. Tags on
import_tasks apply to the whole file, so --tags verify (a supported,
documented way to re-run just this phase) skips discover.yml, leaving
the fact undefined. The stop task's 'default(false)' silently no-op'd,
so re-running verify alone against a host with Gemma still running would
start both new instances on top of it -- the OOM this task exists to
prevent.
Fix: gather service_facts and set the fact locally in verify.yml too,
only when not already defined, so the guard works regardless of which
tags were selected.
Phase 2 (systemd tag) notified per-service restart handlers and then
called meta: flush_handlers itself, so any run where either unit's
template content changed (including first apply) restarted BOTH
live services immediately in Phase 2 -- before Phase 3 firewall
scoping or Phase 4 smoke tests ran. This contradicted the phase's
documented purpose (units land on disk only, nothing starts/restarts
until Phase 4).
Fix: Phase 2 only reloads the systemd daemon and registers each
template task's changed result. Phase 4 (verify.yml) now decides
start vs restart per-service based on that recorded change, so
restarts remain independent per instance and never fire before
Phase 4.
