- host_vars/astro-orbiter/vars.yml: add Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf
to llm_staged_models (size_bytes=8988111072, bartowski GGUF public repo).
Updated VRAM note to reflect 4-model roster and LRU eviction semantics.
- defaults/main.yml: add llm_router_coder_ctx_size=16384 and
llm_router_coder_flash_attn=true variables for per-model ctx tuning.
- templates/llama-server-router-preset.ini.j2: add [Qwen2.5-Coder-14B-Instruct-Q4_K_M]
section with alias=Qwen2.5-Coder-14B-Instruct-4bit, ctx-size=16384, flash-attn=true.
- playbooks/day2_add_coder_alias.yml: new playbook that downloads the GGUF (if
absent/mismatched), deploys updated preset INI and systemd unit, restarts
llama-server-router, and verifies all 4 models in /v1/models.
VRAM: Coder ~9GB. Full 4-model co-residency impossible on 24GB — LRU eviction
handles this automatically. Qwen3.6-35B <-> Coder switches incur ~30-60s cold load.
- Add Meta-Llama-3.1-8B-Instruct-4bit alias on Meta-Llama-3.1-8B-Instruct-Q4_K_M
entry in the preset INI (per Ryan dashboard note).
- Document known issue: Phi-3.5-mini json_schema grammar sampler incompatibility
in router mode (GH #23460 variant — chat template token format mismatch).
Meta-Llama works with json_schema response_format; confirmed via live test.
- Phi-3.5-mini-instruct-8bit alias remains working for model routing;
structured output (json_schema) fails due to the model's token format.
Changes:
- host_vars/astro-orbiter/vars.yml: add llm_router_models_max: 4 (overrides
conservative default of 1). Detailed VRAM OOM risk note included inline:
worst-case 3-model co-residency ~31GB > 24GB RTX 3090. LRU eviction
mitigates in single-user operation; flagged for Ryan's review.
- playbooks/day2_bump_router_models_max.yml: new targeted playbook; deploys
updated router unit, restarts the live service, verifies /health 200 and
/v1/models lists all 3 GGUFs post-restart.
- group_vars/all/semaphore.yml: add llm_router_update_unit template pointing
at the new playbook.
- roles/llm-inference-multimodel/defaults/main.yml: update comment to reflect
the var is now overridden in host_vars rather than 'hardcoded to 1'.
- roles/llm-inference-multimodel/templates/llama-server-router.service.j2:
correct stale 'HARDCODED TO 1' comment — value is variable-driven.
Constraints honored:
- --parallel 1 left untouched (not in scope, not modified anywhere)
- No ad-hoc SSH/systemctl/curl state mutation; all execution via Semaphore
- No installed/vendored code patched
- Retire llama-server-aux (Phi-4, 8000) and llama-server-toolcall (Mistral-Small-24B, 8001): stopped, disabled, unit files removed from host and Ansible role
- Promote llama-server-qwen (Qwen2.5-14B-Instruct-1M, port 8002) to sole production model, serving both friday and war-machine Hermes profiles
- Verified live: n_ctx=65536/n_ctx_train=1010000, and tool_calls response via /v1/chat/completions probe (no hallucination)
- Deleted superseded GGUF weights (phi-4, mistral-small, orphaned base-Qwen, gemma-2-27b) from astro-orbiter, ~45GB reclaimed
- Updated friday and war-machine Hermes profile configs (model + compression + skills_hub aux) to point at 10.1.71.130:8002
- Ryan explicitly accepted single-model tradeoffs for both profiles
- New llama-server-qwen systemd unit template, gated by llm_qwen_service_enabled (default false)
- Idempotent GGUF download task (bartowski Qwen2.5-14B-Instruct-Q5_K_M, stat-guarded)
- Launch flags per local-llm-64k-context-recommendation.md: ctx-size 65536, flash-attn, q8_0 KV cache, batch 2048/ubatch 512, jinja, parallel 1
- verify.yml only starts/verifies the qwen unit when llm_qwen_service_enabled=true
- README: documents live VRAM gate finding (nvidia-smi 2026-08-06: Phi-4+Mistral already ~16.6/24GB, ~7.5GB free -- insufficient for Qwen weights concurrently) and options
- Does NOT touch llama-server-aux (8000) or llama-server-toolcall (8001) service state
