Retired DeepSeek-R1-Distill-Qwen-32B after confirming its auto tool-choice
reliability is a known, documented DeepSeek-R1-distillation limitation
(trained on pure reasoning traces, no function-calling data — upstream
GitHub-confirmed, not a config gap). Model choice moved to Gemma 4 26B A4B
(Google, Apache 2.0, US-origin, matches Ryan's model-origin preference):
- cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit — MoE (25.2B total / 3.8B active),
chosen over the dense 31B variant for smaller on-disk footprint
(~17.2GB vs ~20.9GB), buying more KV-cache headroom on this 24GB card
- max_model_len=65536 (comfortably over Hermes's 64K floor; native
context is 256K, no extension trick needed)
- Native gemma4 tool-call-parser + gemma4 reasoning-parser (both
registered in this host's vLLM 0.28.0) — purpose-built for this
model's actual output format, not a same-family approximation
- kv_cache_dtype: int4_per_token_head from the outset (learned from the
DeepSeek swap's fp16->fp8->int4 trial-and-error escalation)
Bug found and fixed during deployment: the repo's config.json declares
quant_method 'compressed-tensors' (llm-compressor output) despite the
repo name saying 'AWQ-4bit'. Passing --quantization awq explicitly
caused a hard pydantic ValidationError on every startup attempt.
Fix: omit the quantization field entirely and let vLLM auto-detect from
the model's own config.json — confirmed clean single-attempt start,
NRestarts=0, once removed.
Verified live:
- /health 200, /v1/models confirms max_model_len=65536
- Live completion: correct answer, no unwanted reasoning trace by default
- tool_choice=auto with a clear trigger prompt: correct tool_calls
response with valid JSON args — the exact test DeepSeek-R1-Distill
failed (it either answered in plain text or burned tokens reasoning
about how to call the tool instead of calling it)
- tool_choice=auto with an irrelevant tool present: correctly answered
in plain text, did not over-trigger the tool
- Ansible idempotent re-run confirmed: changed=0, NRestarts=0, clean
journalctl (zero error/traceback lines) after a fresh restart
Known follow-up (not done here): Hindsight's HINDSIGHT_API_LLM_MODEL
cluster config still references the retired DeepSeek-R1-Distill-Qwen-32B
(itself a follow-up from the prior Qwen2.5-32B swap) — needs another
GitOps update to point at Gemma-4-26B-A4B-it-AWQ.
Single-model deployment per Ryan's direction:
- Primary model: casperhansen/deepseek-r1-distill-qwen-32b-awq, max_model_len=32768
- nomic-embed-text-v1.5 and Qwen3-8B-AWQ both disabled (single-model requirement)
- kv_cache_dtype: int4_per_token_head required to fit 32768 ctx on 24GB RTX 3090
(fp16 KV: 26GB needed, doesn't fit at any utilization; fp8 KV: OOM'd during
FlashInfer warmup with ~50-150MB margin; int4 KV: clean single-attempt start)
- Added kv_cache_dtype / kv_cache_memory_bytes as new optional per-model
template fields in vllm.service.j2 (guarded, no effect on other models)
Verified live: /health 200, /v1/models confirms max_model_len=32768,
live /v1/completions smoke test + manual chat completion both passed
(genuine <think> reasoning trace, correct arithmetic). NRestarts=0,
steady-state VRAM 23.2GB/24.576GB. Ansible idempotent re-run confirmed
changed=0.
Known follow-up (not done here): Hindsight's HINDSIGHT_API_LLM_MODEL
cluster config still references the retired Qwen2.5-32B-Instruct-AWQ —
needs separate GitOps update to point at the new model.
Dashboard decision (t_5508360a): 'stop and disable llama-swap and start
vLLM and its 3 models' -- explicit approval of a breaking change.
Applied:
- llama-swap stopped + disabled on astro-orbiter (systemd unit removed
from multi-user.target.wants, files left in place -- full teardown is
t_6dff1ecc, separate task)
- vllm_service_enabled/state flipped to true/started -- vLLM is now the
permanent, boot-persistent serving layer (was shadow-only/staged)
- Attempted enabling Qwen3-8B-AWQ (the 3rd model) -- does NOT fit.
Qwen2.5-32B-Instruct-AWQ (~18.6GB) + nomic-embed (~0.8GB) leaves only
~1.25GiB free on the 23.55GiB usable budget, below the 3.53GiB floor
Qwen3-8B-AWQ needs even at gpu_memory_utilization=0.15. Confirmed via
journalctl: identical ValueError on 7/7 consecutive restart attempts,
not a transient crash-loop. Reverted Qwen3-8B-AWQ to enabled: false.
2 of the 3 requested models fit permanently, not 3.
Verified live: /health 200 on both :8000 and :8020, live completion and
live embedding both returned correct real output, NRestarts=0 on both
services after a clean idempotent re-run (changed=0).
Critical finding: flipping vllm_service_enabled/state=true/started and
restarting llama-swap alongside it broke llama-swap's ability to load
ANY of its own generative models -- every /v1/chat/completions request
against Qwen3.8-27B-Q4_K_M or Qwen3-8B aux models failed with
'upstream command exited prematurely' (llama-server OOM at spawn,
~1.8GB free on this 24GB card once vLLM's ~22.8GB was claimed).
Confirmed by direct A/B: same request 500s with vLLM running, 200s
seconds after stopping it.
This breaks 21 Hermes agent profiles' aux-model tasks (skills_hub,
approval, mcp, title_generation, profile_describer, compression) plus
OpenViking's VLM -- a far larger blast radius than Hindsight's single
LLM endpoint. Reverted:
- vllm_service_enabled/state back to role defaults (false/stopped) --
vLLM stays staged, startable for a brief validated shadow window,
NOT safe to leave resident in production.
- Hindsight's HINDSIGHT_API_LLM_BASE_URL back to llama-swap
(astro-orbiter:8001, Qwen3.8-27B-Q4_K_M) and the API key secret
source back to the Nous fallback item (pre-task state) --
the vLLM cutover, while functionally validated in isolation
(health, /v1/chat/completions, and a live hindsight_retain+recall
round-trip all succeeded), requires continuous vLLM availability
which is now known to be unsafe on this card.
Comment posted on t_6dff1ecc: teardown remains correctly blocked --
full cutover is not achievable within this card's VRAM budget as
currently scoped. Needs a human decision on aux-model migration
strategy (see roles/deploy-vllm README's 'Critical architectural
finding' section) before any further progress.
- vllm.service.j2: branch on role==embedding for --runner pooling
--convert embed, --no-enable-prefix-caching, per-model
trust_remote_code toggle (needed for nomic-embed-text-v1.5's custom
NomicBertModel code), and enforce_eager toggle (needed to avoid CUDA
graph capture OOM when co-resident with another vLLM process on this
24GB card).
- tasks/verify.yml: split completions vs embedding smoke tests --
embedding-mode instances don't serve /v1/completions. Assert a
non-empty embedding vector, not just HTTP 200.
- host_vars/astro-orbiter: enable nomic-embed-text-v1.5 (port 8020),
lower primary model's gpu_memory_utilization 0.95->0.90 + add
enforce_eager after finding 0.95 crash-looped 6-7x before stabilizing
with co-resident nomic-embed (real fix, confirmed via NRestarts=0
after clean stop/start, not luck).
- Hindsight (values.yaml + externalsecret.yaml): cut LLM + embeddings
over to vLLM (:8000, :8020), wire the previously-unset
HINDSIGHT_API_EMBEDDINGS_* env vars for the first time, and swap the
API key secret source from the Nous fallback item to vllm/api-key
(vLLM enforces real auth, llama-swap did not).
- README: document the embedding-mode branch, VRAM findings, and a
genuine architecture gap -- vLLM's one-model-per-process design
cannot replace llama-swap's 5-model LRU roster on this 24GB card, so
21 Hermes profiles' aux-model consumers (Qwen3-8B-no_think,
Phi-3.5-mini, Meta-Llama-3.1-8B, Qwen2.5-Coder-14B) and OpenViking's
VLM stay on llama-swap. Full teardown (t_6dff1ecc) needs a human
decision on the aux-model strategy before it can proceed.
- Remove global --n-gpu-layers from router unit ExecStart in preset mode
(llama.cpp CLI arg outranked per-model INI n-gpu-layers=0; root cause from
War Machine's run 1). Flag now emitted only in --models-dir mode.
- All 5 preset INI sections carry explicit n-gpu-layers:
Qwen3.8=99, Phi=99, nomic=99, Coder=0, Llama=0.
- host_vars/astro-orbiter: llm_router_models_max 2 -> 4 so CPU-offloaded
models count as loaded without LRU-evicting Qwen3.8.
- defaults: llm_router_coder_gpu_layers / llm_router_llama_gpu_layers = 0.
- verify.yml: fix pre-existing .meta attribute crash in router mode.
- New playbook day2_cpu_offload_aux_models.yml.
Deployed + verified on astro-orbiter (gates A-E PASS): concurrent residency
achieved, Qwen3.8 stays GPU-resident. Measured CPU throughput Llama 9.0 /
Coder 4.7 tok/s. VRAM note: llama.cpp 6ea215d allocates ~1.4-1.7GB CUDA-context
per CPU model even at n-gpu-layers=0 -> ~24,004 MiB steady-state, below the
24,576 MiB physical limit. Comments corrected to match the measurement.
Report: friday/inbox/ryan/2026-08-17-llm-cpu-offload-coder-llama-deployed.md
Ryan-directed model swap (kanban t_f5f7e9ad, 2026-08-16).
Changes:
- Replace [Qwen3.6-35B-A3B-UD-Q4_K_S] with [Qwen3.8-27B-Q4_K_M] in
llama-server-router-preset.ini.j2 (production model slot).
- Qwen3.8-27B: dense 27B VLM, Apache-2.0, Alibaba Aug 2026.
Unsloth Dynamic V3.0 GGUF quantization.
Q4_K_M chosen: 17,106,775,008 bytes, 17.1GB. Measured VRAM: 17,068 MiB
at ctx=32768 (q4_0 KV cache).
- ctx-size set to 32768 (32K) via new variable llm_router_qwen38_ctx_size.
Native context is 262K; 32K chosen to maintain eviction headroom on 24GB RTX 3090.
- models-max reduced 4 -> 2 in host_vars. Qwen3.8 (17.6GB) + nomic-embed
(558MB) exhaust the 24GB card; no auxiliary model can co-reside with Qwen3.8.
LRU eviction handles model switching with ~30-60s cold-load latency.
- llm_router_expected_model_id updated to Qwen3.8-27B-Q4_K_M.
- Qwen3.6 GGUF retained at /opt/models/Qwen3.6-35B-A3B-UD-Q4_K_S.gguf
(not deleted — pending stable period and explicit cleanup task).
- day2_swap_qwen38.yml playbook added for Ansible idempotent redeployment.
Architecture note: Qwen3.8 uses Gated DeltaNet; llama.cpp 6ea215d logs
'fused Gated Delta Net (chunked) not supported, set to disabled'. Inference
works correctly on the non-fused fallback. A llama.cpp update may improve
throughput on the GDN layers.
Smoke test passed: model responded via router endpoint (http://10.1.71.130:8002).
VRAM: 17,630 MiB (Qwen3.8) + 5,928 MiB (Llama-8B concurrent) = 23,558 MiB.
Also commits accumulated but unpushed changes:
- nomic-embed batch-size/rope-scaling fix (t_openviking_embed_batch)
- per-model ctx-size day2 playbook (day2_per_model_ctx_size.yml)
- llama-server-router.service.j2 minor update
- host_vars/astro-orbiter/vars.yml: add Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf
to llm_staged_models (size_bytes=8988111072, bartowski GGUF public repo).
Updated VRAM note to reflect 4-model roster and LRU eviction semantics.
- defaults/main.yml: add llm_router_coder_ctx_size=16384 and
llm_router_coder_flash_attn=true variables for per-model ctx tuning.
- templates/llama-server-router-preset.ini.j2: add [Qwen2.5-Coder-14B-Instruct-Q4_K_M]
section with alias=Qwen2.5-Coder-14B-Instruct-4bit, ctx-size=16384, flash-attn=true.
- playbooks/day2_add_coder_alias.yml: new playbook that downloads the GGUF (if
absent/mismatched), deploys updated preset INI and systemd unit, restarts
llama-server-router, and verifies all 4 models in /v1/models.
VRAM: Coder ~9GB. Full 4-model co-residency impossible on 24GB — LRU eviction
handles this automatically. Qwen3.6-35B <-> Coder switches incur ~30-60s cold load.
Changes:
- host_vars/astro-orbiter/vars.yml: add llm_router_models_max: 4 (overrides
conservative default of 1). Detailed VRAM OOM risk note included inline:
worst-case 3-model co-residency ~31GB > 24GB RTX 3090. LRU eviction
mitigates in single-user operation; flagged for Ryan's review.
- playbooks/day2_bump_router_models_max.yml: new targeted playbook; deploys
updated router unit, restarts the live service, verifies /health 200 and
/v1/models lists all 3 GGUFs post-restart.
- group_vars/all/semaphore.yml: add llm_router_update_unit template pointing
at the new playbook.
- roles/llm-inference-multimodel/defaults/main.yml: update comment to reflect
the var is now overridden in host_vars rather than 'hardcoded to 1'.
- roles/llm-inference-multimodel/templates/llama-server-router.service.j2:
correct stale 'HARDCODED TO 1' comment — value is variable-driven.
Constraints honored:
- --parallel 1 left untouched (not in scope, not modified anywhere)
- No ad-hoc SSH/systemctl/curl state mutation; all execution via Semaphore
- No installed/vendored code patched
Adds idempotent, data-driven GGUF staging for the two new router models on
astro-orbiter alongside the production Qwen3.6-35B-A3B-UD-Q4_K_S. Both files
were already staged live (byte-exact); this commit codifies them in Ansible so
future re-runs and any new model adds are version-controlled and audit-friendly.
Changes:
- roles/llm-inference-multimodel/tasks/stage_model.yml (NEW)
Idempotent per-model task: stat -> exact byte-size guard -> conditional
get_url -> ownership/mode ensure -> notify router restart handler only on
actual download. Loops from models.yml; nothing hardcoded.
- roles/llm-inference-multimodel/tasks/models.yml
Appends the stage_model.yml loop (tagged: models) after the existing Qwen3.6
download tasks. Data driven from host_vars/astro-orbiter/vars.yml.
- roles/llm-inference-multimodel/defaults/main.yml
Adds llm_staged_models: [] default (empty = safe no-op for hosts with no
staged model list defined).
- roles/llm-inference-multimodel/handlers/main.yml
Adds 'restart llama-server-router on new GGUF' handler. Only fires when
stage_model.yml performs an actual download or corrects ownership/mode.
Normal idempotent re-runs (files already correct) do NOT fire this handler.
- host_vars/astro-orbiter/vars.yml
Adds llm_staged_models list with the two new models:
* Phi-3.5-mini-instruct-Q8_0.gguf (4,061,222,688 bytes,
bartowski/Phi-3.5-mini-instruct-GGUF)
* Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf (4,920,739,232 bytes,
bartowski/Meta-Llama-3.1-8B-Instruct-GGUF)
- playbooks/day1_deploy_llm_inference_multimodel.yml
Updates header comment: removes stale 'Semaphore broken' note, documents
the correct execution channel (Semaphore template
llm_inference_multimodel_stage_models, --tags models).
- group_vars/all/semaphore.yml
Adds llm_inference_multimodel_stage_models template entry (config-as-code).
Template is scoped to --tags models explicitly. Phase 4 (verify) is
EXCLUDED: verify.yml starts llama-server-qwen on :8002, which would collide
with the production llama-server-router.service already running on :8002.
Semaphore template created via API: project 1 / template id 19.
Execution: triggered immediately after this commit via Semaphore REST API.
- Prometheus scrape configs for node/gpu/llama-server exporters on
astro-orbiter now declared in cluster/applications/monitoring/values.yaml
(additionalScrapeConfigs), applied via ArgoCD sync instead of an
imperative kubectl secret patch from the Ansible role.
- Grafana dashboard for astro-orbiter LLM inference added as a ConfigMap
in cluster/applications/monitoring/dashboards.yaml (grafana_dashboard=1
sidecar label), replacing the role's ad-hoc kubectl apply of a rendered
Jinja template.
- ansible/roles/llm-inference/tasks/monitoring.yml: removed the kubectl
get/patch/apply tasks and orphaned grafana-llm-dashboard.json.j2
template; role now only stands up node_exporter + nvidia_gpu_exporter
and verifies they're reachable — cluster-facing config lives in Git.
- host_vars/vars.yml + inventory.yml: finalize astro-orbiter as the
llama.cpp/RTX 3090 host (jarvis user, ssh key), drop stale Ollama/AMD
vars and ollama_server inventory group superseded by the ATX rebuild.
Adds jmri_ssh_authorized_keys_extra list to support multiple authorized
keys on the jmri account. Deploys rblundon's MacBook key so xpra can
connect via SSH without a password.
- Stable udev device symlinks (/dev/jmri/nce, /dev/jmri/loconet, /dev/jmri/lcc)
- jmri-monitor: polls Leviton Decora Smart switch to start/stop JMRI automatically
- Quiet hours 1-10 AM (no polling)
- 30s off-delay before shutdown
- LCRR config cloned from Gitea (ssh://gitea.mk-labs.cloud:2221/rblundon/LCRR.git)
- ~/.jmri symlinked to LCRR repo for GitOps config management
- jmri-gui: X11 remote GUI access (PanelPro/DecoderPro) via ssh -X as jmri user
- Stops daemon, launches GUI, restarts daemon on exit if layout still on
- jmri user gets login shell + SSH key for GUI sessions
- Full JRE installed (openjdk-21-jre) for AWT/X11 support
- New ansible/roles/jmri role: installs OpenJDK 21 headless, creates
jmri service user, downloads JMRI 5.10, deploys JmriFaceless systemd unit
- Handles dialout group membership for serial device access
- Config restore task for post-reinstall recovery from GitHub backup
- host_vars/main-street-station: profile_id and serial device (TODO: fill in)
- Inventory: jmri_server group with main-street-station at 192.168.10.45
- Playbook: day1_deploy_jmri.yml (linux-baseline + jmri)
