feat(llm-router): switch to --models-preset mode; add Phi-3.5-mini-instruct-8bit alias (t_9adf0889)
- Add templates/llama-server-router-preset.ini.j2: defines all 3 router GGUFs (Qwen3.6-35B, Phi-3.5-mini-Q8_0, Meta-Llama-3.1-8B) with explicit ctx-size, gpu-layers, cache settings carried over from --models-dir baseline. The Phi entry adds alias=Phi-3.5-mini-instruct-8bit (Ryan's aux.title_generation target). - Update templates/llama-server-router.service.j2: Jinja2 conditional emits --models-preset <path> when llm_router_preset_enabled=true, otherwise --models-dir (backward compat, default unchanged). - Add tasks/preset.yml: deploy preset INI, restart router on change, verify both Phi-3.5-mini-instruct-Q8_0 (primary) and Phi-3.5-mini-instruct-8bit (alias) appear in /v1/models, plus Qwen and Llama IDs unchanged. - Update defaults/main.yml: add llm_router_preset_enabled (default false) and llm_router_preset_path=/opt/llama-server-router-preset.ini. - Update tasks/main.yml: import preset.yml as Phase P (gated, no-op by default). - Update handlers/main.yml: add 'restart router' handler for preset changes. - Add playbooks/day2_add_phi_alias.yml: single-command deployment. GH #22364 note: --models-preset mode creates an extra 'default' entry in /v1/models — cosmetic, does not affect model selection by name.
This commit is contained in:
@@ -131,3 +131,21 @@ llm_router_bind_address: "{{ llm_bind_address }}" # 10.1.71.130
|
||||
llm_router_allowed_source_cidr: "{{ llm_allowed_source_cidr }}" # 10.1.70.0/24
|
||||
llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S" # verified at Gate 1
|
||||
llm_router_vram_max_mib: 23000 # Gate 3: fail if exceeded under load
|
||||
|
||||
# --- Router preset mode (--models-preset INI) ---------------------------------
|
||||
# Set llm_router_preset_enabled: true to switch from --models-dir to
|
||||
# --models-preset. Preset mode is REQUIRED to support model aliases.
|
||||
# The template at llama-server-router-preset.ini.j2 defines all 3 router models:
|
||||
# - Qwen3.6-35B-A3B-UD-Q4_K_S (no alias — primary ID unchanged)
|
||||
# - Phi-3.5-mini-instruct-Q8_0 (alias: Phi-3.5-mini-instruct-8bit) <-- t_9adf0889
|
||||
# - Meta-Llama-3.1-8B-Instruct-Q4_K_M (no alias — primary ID unchanged)
|
||||
#
|
||||
# llm_router_preset_path: on-disk path where the rendered INI is deployed.
|
||||
# Default: /opt/llama-server-router-preset.ini (owned by root, readable by all).
|
||||
#
|
||||
# GH #22364 note: --models-preset causes an extra "default" entry in /v1/models.
|
||||
# This is cosmetic and does not affect model selection by name. Accept it.
|
||||
#
|
||||
# Added 2026-08-12 (t_9adf0889) — War Machine.
|
||||
llm_router_preset_enabled: false # flip true to activate preset mode
|
||||
llm_router_preset_path: /opt/llama-server-router-preset.ini
|
||||
|
||||
Reference in New Issue
Block a user