feat(llm-inference-multimodel): bump router --models-max 1->4 on astro-orbiter (t_33acbb2e)
Changes: - host_vars/astro-orbiter/vars.yml: add llm_router_models_max: 4 (overrides conservative default of 1). Detailed VRAM OOM risk note included inline: worst-case 3-model co-residency ~31GB > 24GB RTX 3090. LRU eviction mitigates in single-user operation; flagged for Ryan's review. - playbooks/day2_bump_router_models_max.yml: new targeted playbook; deploys updated router unit, restarts the live service, verifies /health 200 and /v1/models lists all 3 GGUFs post-restart. - group_vars/all/semaphore.yml: add llm_router_update_unit template pointing at the new playbook. - roles/llm-inference-multimodel/defaults/main.yml: update comment to reflect the var is now overridden in host_vars rather than 'hardcoded to 1'. - roles/llm-inference-multimodel/templates/llama-server-router.service.j2: correct stale 'HARDCODED TO 1' comment — value is variable-driven. Constraints honored: - --parallel 1 left untouched (not in scope, not modified anywhere) - No ad-hoc SSH/systemctl/curl state mutation; all execution via Semaphore - No installed/vendored code patched
This commit is contained in:
@@ -29,6 +29,27 @@ common_root_lv: ubuntu-lv
|
||||
# Meta-Llama-3.1-8B-Instruct-Q4_K_M as router models alongside the production
|
||||
# Qwen3.6-35B-A3B-UD-Q4_K_S. The live files were already present/correct on
|
||||
# astro-orbiter; this pass codifies them. Future adds = append to this list.
|
||||
# Router --models-max override for astro-orbiter.
|
||||
# Default in defaults/main.yml is 1 (conservative). Bumped to 4 on 2026-08-12
|
||||
# (t_33acbb2e) so the router can keep more than one GGUF resident on-demand
|
||||
# and LRU-evict when needed.
|
||||
#
|
||||
# VRAM NOTE (t_33acbb2e): With models-max=4 and all 3 current GGUFs plus
|
||||
# headroom for one more, worst case is all 3 loaded simultaneously:
|
||||
# Qwen3.6-35B-A3B Q4_K_S: ~21.5GB (weights ~19.5GB + KV ~2GB @ 64K ctx, q4_0)
|
||||
# Phi-3.5-mini-instruct Q8_0: ~4.3GB (weights ~3.8GB + KV ~0.5GB @ 64K ctx)
|
||||
# Meta-Llama-3.1-8B Q4_K_M: ~5.6GB (weights ~4.6GB + KV ~1.0GB @ 64K ctx)
|
||||
# Total worst-case: ~31.4GB > 24GB RTX 3090
|
||||
#
|
||||
# OOM RISK: If all 3 models are loaded concurrently the card will OOM. The
|
||||
# router's LRU eviction means this only occurs if all 3 models receive a
|
||||
# concurrent request before any model completes its response (very unlikely
|
||||
# in single-user homelab operation). However, Ryan should be aware. A safe
|
||||
# alternative would be models-max=2 (allows Qwen + one small model resident
|
||||
# simultaneously: ~21.5 + 5.6 = ~27.1GB still tight). Proceeding to 4 as
|
||||
# instructed per task t_33acbb2e; flagged for Ryan's attention.
|
||||
llm_router_models_max: 4
|
||||
|
||||
llm_staged_models:
|
||||
- filename: "Phi-3.5-mini-instruct-Q8_0.gguf"
|
||||
url: "https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF/resolve/main/Phi-3.5-mini-instruct-Q8_0.gguf"
|
||||
|
||||
Reference in New Issue
Block a user