feat(llm-inference-multimodel): bump router --models-max 1->4 on astro-orbiter (t_33acbb2e)
Changes: - host_vars/astro-orbiter/vars.yml: add llm_router_models_max: 4 (overrides conservative default of 1). Detailed VRAM OOM risk note included inline: worst-case 3-model co-residency ~31GB > 24GB RTX 3090. LRU eviction mitigates in single-user operation; flagged for Ryan's review. - playbooks/day2_bump_router_models_max.yml: new targeted playbook; deploys updated router unit, restarts the live service, verifies /health 200 and /v1/models lists all 3 GGUFs post-restart. - group_vars/all/semaphore.yml: add llm_router_update_unit template pointing at the new playbook. - roles/llm-inference-multimodel/defaults/main.yml: update comment to reflect the var is now overridden in host_vars rather than 'hardcoded to 1'. - roles/llm-inference-multimodel/templates/llama-server-router.service.j2: correct stale 'HARDCODED TO 1' comment — value is variable-driven. Constraints honored: - --parallel 1 left untouched (not in scope, not modified anywhere) - No ad-hoc SSH/systemctl/curl state mutation; all execution via Semaphore - No installed/vendored code patched
This commit is contained in:
@@ -107,8 +107,9 @@ llm_existing_gemma_service_name_guess: llama-server
|
||||
# Default: llm_router_enabled: false — all router tasks are no-ops until you
|
||||
# flip this to true (either in host_vars, extra-vars, or the shadow playbook).
|
||||
#
|
||||
# CRITICAL: llm_router_models_max is hardcoded to 1 in the j2 template AND
|
||||
# listed here for documentation. DO NOT raise it without a VRAM budget review.
|
||||
# CRITICAL: llm_router_models_max default is 1 here for safety. It is
|
||||
# overridden to 4 in host_vars/astro-orbiter/vars.yml (t_33acbb2e) with
|
||||
# a full VRAM budget note. DO NOT raise it without a VRAM budget review.
|
||||
# Default llama-server cap is 4 simultaneous — that would OOM a 24GB card
|
||||
# immediately when Qwen3.6-35B (20GB) is the resident model.
|
||||
#
|
||||
|
||||
Reference in New Issue
Block a user