Add Qwen2.5-Coder-14B-Instruct-4bit to astro-orbiter router (t_55c164f5)

- host_vars/astro-orbiter/vars.yml: add Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf
  to llm_staged_models (size_bytes=8988111072, bartowski GGUF public repo).
  Updated VRAM note to reflect 4-model roster and LRU eviction semantics.
- defaults/main.yml: add llm_router_coder_ctx_size=16384 and
  llm_router_coder_flash_attn=true variables for per-model ctx tuning.
- templates/llama-server-router-preset.ini.j2: add [Qwen2.5-Coder-14B-Instruct-Q4_K_M]
  section with alias=Qwen2.5-Coder-14B-Instruct-4bit, ctx-size=16384, flash-attn=true.
- playbooks/day2_add_coder_alias.yml: new playbook that downloads the GGUF (if
  absent/mismatched), deploys updated preset INI and systemd unit, restarts
  llama-server-router, and verifies all 4 models in /v1/models.

VRAM: Coder ~9GB. Full 4-model co-residency impossible on 24GB — LRU eviction
handles this automatically. Qwen3.6-35B <-> Coder switches incur ~30-60s cold load.
This commit is contained in:
Hermes Agent service account
2026-08-13 09:07:02 -05:00
parent 6455d22752
commit 7aea88724f
4 changed files with 329 additions and 15 deletions

View File

@@ -34,20 +34,25 @@ common_root_lv: ubuntu-lv
# (t_33acbb2e) so the router can keep more than one GGUF resident on-demand
# and LRU-evict when needed.
#
# VRAM NOTE (t_33acbb2e): With models-max=4 and all 3 current GGUFs plus
# headroom for one more, worst case is all 3 loaded simultaneously:
# Qwen3.6-35B-A3B Q4_K_S: ~21.5GB (weights ~19.5GB + KV ~2GB @ 64K ctx, q4_0)
# Phi-3.5-mini-instruct Q8_0: ~4.3GB (weights ~3.8GB + KV ~0.5GB @ 64K ctx)
# Meta-Llama-3.1-8B Q4_K_M: ~5.6GB (weights ~4.6GB + KV ~1.0GB @ 64K ctx)
# Total worst-case: ~31.4GB > 24GB RTX 3090
# VRAM NOTE (t_33acbb2e, updated t_55c164f5): With models-max=4 and all 4 GGUFs
# registered, worst case is all 4 loaded simultaneously:
# Qwen3.6-35B-A3B Q4_K_S: ~21.5GB (weights ~19.5GB + KV ~2GB @ 64K ctx, q4_0)
# Phi-3.5-mini-instruct Q8_0: ~4.3GB (weights ~3.8GB + KV ~0.5GB @ 32K ctx)
# Meta-Llama-3.1-8B Q4_K_M: ~5.6GB (weights ~4.6GB + KV ~0.2GB @ 8K ctx)
# Qwen2.5-Coder-14B Q4_K_M: ~9.0GB (weights ~8.4GB + KV ~0.6GB @ 16K ctx)
# Total worst-case: ~40.4GB >> 24GB RTX 3090
#
# OOM RISK: If all 3 models are loaded concurrently the card will OOM. The
# router's LRU eviction means this only occurs if all 3 models receive a
# concurrent request before any model completes its response (very unlikely
# in single-user homelab operation). However, Ryan should be aware. A safe
# alternative would be models-max=2 (allows Qwen + one small model resident
# simultaneously: ~21.5 + 5.6 = ~27.1GB still tight). Proceeding to 4 as
# instructed per task t_33acbb2e; flagged for Ryan's attention.
# OOM RISK: Full co-residency is impossible on 24GB. LRU eviction prevents this
# in practice: models-max=4 means the router can REGISTER 4 models but only keeps
# up to 4 LOADED simultaneously — the router will evict the LRU model when a new
# one is needed. In single-user homelab operation, only one model is active at a
# time. The realistic maximum co-residency is 2 models (whichever was last used).
# Qwen3.6-35B alone uses ~21.5GB; co-residency with Coder (~9GB) = ~30.5GB > 24GB.
# So effectively: Qwen3.6-35B + any second model will OOM IF both are held concurrently.
# LRU eviction handles this automatically — the router evicts the idle model before
# loading the new one. Ryan should be aware this means model-switching always incurs
# a ~30-60s cold-load latency when switching between Qwen3.6-35B and any other model.
# Proceeding to models-max=4 as instructed; flagged for Ryan's attention.
llm_router_models_max: 4
llm_staged_models:
@@ -59,4 +64,8 @@ llm_staged_models:
url: "https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
size_bytes: 4920739232
source_repo: "bartowski/Meta-Llama-3.1-8B-Instruct-GGUF"
- filename: "Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
url: "https://huggingface.co/bartowski/Qwen2.5-Coder-14B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
size_bytes: 8988111072
source_repo: "bartowski/Qwen2.5-Coder-14B-Instruct-GGUF"