rblundon/homelab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Qwen2.5-Coder-14B-Instruct-4bit to astro-orbiter router (t_55c164f5)

Browse Source 
- host_vars/astro-orbiter/vars.yml: add Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf
  to llm_staged_models (size_bytes=8988111072, bartowski GGUF public repo).
  Updated VRAM note to reflect 4-model roster and LRU eviction semantics.
- defaults/main.yml: add llm_router_coder_ctx_size=16384 and
  llm_router_coder_flash_attn=true variables for per-model ctx tuning.
- templates/llama-server-router-preset.ini.j2: add [Qwen2.5-Coder-14B-Instruct-Q4_K_M]
  section with alias=Qwen2.5-Coder-14B-Instruct-4bit, ctx-size=16384, flash-attn=true.
- playbooks/day2_add_coder_alias.yml: new playbook that downloads the GGUF (if
  absent/mismatched), deploys updated preset INI and systemd unit, restarts
  llama-server-router, and verifies all 4 models in /v1/models.

VRAM: Coder ~9GB. Full 4-model co-residency impossible on 24GB — LRU eviction
handles this automatically. Qwen3.6-35B <-> Coder switches incur ~30-60s cold load.
This commit is contained in:
Hermes Agent service account
2026-08-13 09:07:02 -05:00
parent 6455d22752
commit 7aea88724f
4 changed files with 329 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

39
ansible/roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2
View File

@@ -27,6 +27,19 @@
; Added 2026-08-12 (t_9adf0889): Phi alias — War Machine.
; All per-model settings carry over unchanged from the --models-dir baseline
; (ctx_size=65536, n_gpu_layers=99, cache=q4_0 for both K and V, models-max=4).
;
; UPDATED (t_ryan_per_model_ctx, per Ryan/JARVIS request): Llama-3.1-8B and
; Phi-3.5-mini now get PER-MODEL ctx-size/flash-attn matched to actual
; workload instead of the uniform 65536 used by every model previously:
; - Llama-3.1-8B-Instruct-Q4_K_M: ctx-size 8192 (tool-routing/micro-tasks)
; - Phi-3.5-mini-instruct-Q8_0: ctx-size 32768 (long web scrapes/logs)
; Both now request explicit flash-attn=true (was "auto"). Qwen3.6-35B is
; INTENTIONALLY left untouched at ctx-size 65536 / flash-attn auto — not part
; of this change. Existing aliases (Meta-Llama-3.1-8B-Instruct-4bit,
; Phi-3.5-mini-instruct-8bit) are PRESERVED unchanged to avoid breaking live
; Hermes custom_providers routing — see role README / deployment report for
; the alias-naming ambiguity flag (Ryan's pasted TOML used different alias
; strings: "llama-3.1-8b" / "phi-3.5-mini").
; ------------------------------------------------------------------------------
; --- Production model: Qwen3.6-35B-A3B-UD-Q4_K_S ----------------------------
@@ -67,7 +80,8 @@ parallel = {{ llm_router_parallel }}
model = {{ llm_models_dir }}/Phi-3.5-mini-instruct-Q8_0.gguf
alias = Phi-3.5-mini-instruct-8bit
n-gpu-layers = {{ llm_router_gpu_layers }}
ctx-size = {{ llm_router_ctx_size }}
ctx-size = {{ llm_router_phi_ctx_size }}
flash-attn = {{ llm_router_phi_flash_attn }}
cache-type-k = {{ llm_router_cache_type_k }}
cache-type-v = {{ llm_router_cache_type_v }}
batch-size = {{ llm_router_batch_size }}
@@ -83,7 +97,28 @@ parallel = {{ llm_router_parallel }}
model = {{ llm_models_dir }}/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
alias = Meta-Llama-3.1-8B-Instruct-4bit
n-gpu-layers = {{ llm_router_gpu_layers }}
ctx-size = {{ llm_router_ctx_size }}
ctx-size = {{ llm_router_llama_ctx_size }}
flash-attn = {{ llm_router_llama_flash_attn }}
cache-type-k = {{ llm_router_cache_type_k }}
cache-type-v = {{ llm_router_cache_type_v }}
batch-size = {{ llm_router_batch_size }}
ubatch-size = {{ llm_router_ubatch_size }}
parallel = {{ llm_router_parallel }}
; --- Coder model: Qwen2.5-Coder-14B-Instruct-Q4_K_M -------------------------
; Primary model ID: Qwen2.5-Coder-14B-Instruct-Q4_K_M (filename-derived)
; Alias: Qwen2.5-Coder-14B-Instruct-4bit (friendlier name)
; Both names resolve to this GGUF child process.
; ~8.4GB weights + ~0.6GB KV @ 16K ctx = ~9.0GB VRAM.
; ctx-size=16384, flash-attn=true per task t_55c164f5 / Ryan's request.
; Source: bartowski/Qwen2.5-Coder-14B-Instruct-GGUF (public, no auth)
; Added 2026-08-13 (t_55c164f5) — War Machine.
[Qwen2.5-Coder-14B-Instruct-Q4_K_M]
model = {{ llm_models_dir }}/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf
alias = Qwen2.5-Coder-14B-Instruct-4bit
n-gpu-layers = {{ llm_router_gpu_layers }}
ctx-size = {{ llm_router_coder_ctx_size }}
flash-attn = {{ llm_router_coder_flash_attn }}
cache-type-k = {{ llm_router_cache_type_k }}
cache-type-v = {{ llm_router_cache_type_v }}
batch-size = {{ llm_router_batch_size }}