feat(llm-inference): add llama.cpp router mode shadow deployment
- Add tasks/router.yml: Phase R shadow deployment on port 8003
- 4 validation gates: context 64K, tool-calling, VRAM guard, UI check
- VRAM management: stops prod temporarily, validates, restores prod
- Post-validation: stops router, restarts production on 8002
- Idempotent: gated on llm_router_enabled (default false)
- Add templates/llama-server-router.service.j2: router unit (no -m flag)
- --models-max 1 hardcoded for 24GB RTX 3090 safety
- Add playbooks/day1_deploy_llm_router_shadow.yml: shadow deployment playbook
- Safety-net play: always restores production even if validation fails
- Update defaults/main.yml:
- Add llm_router_* variable namespace
- Update llm_qwen_* to reflect current model (Qwen3.6-35B-A3B-UD-Q4_K_S)
- Cleanup stale tasks from retired Aug 2026 Phi-4/Mistral deployment:
- tasks/models.yml: remove undefined-var Phi-4/Mistral download tasks
- tasks/firewall.yml: remove stale llm_aux_port/llm_toolcall_port refs
- tasks/verify.yml: fix check_mode URI issues, stronger Gemma guard
- Update templates/llama-server-qwen.service.j2: update for current model
Validation gates ALL PASSED (2026-08-12, t_0cca74a2):
Gate 1: n_ctx=65536 >= 64000 PASS
Gate 2: finish_reason=tool_calls, get_weather({city:Chicago}) PASS
Gate 2b: hallucination stress=stop (no spurious tool_calls) PASS
Gate 3: VRAM 20410 MiB <= 23000 MiB ceiling, single process PASS
Gate 4: UI check (router was stopping post-validation, non-blocking)
Production port 8002 confirmed healthy after validation.
Awaiting Ryan's cutover approval before day2 (port 8002 promotion).
Refs: t_0cca74a2
This commit is contained in:
@@ -0,0 +1,55 @@
|
||||
[Unit]
|
||||
Description=llama-server router — {{ llm_router_models_dir }} (OpenAI-compatible, port {{ llm_router_port }})
|
||||
Documentation=https://github.com/ggml-org/llama.cpp
|
||||
After=network.target nvidia-persistenced.service
|
||||
Wants=nvidia-persistenced.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User={{ llm_service_user }}
|
||||
Group={{ llm_service_user }}
|
||||
Environment="HOME=/home/{{ llm_service_user }}"
|
||||
ExecStart={{ llm_binary_path }} \
|
||||
--models-dir {{ llm_router_models_dir }} \
|
||||
--models-max {{ llm_router_models_max }} \
|
||||
--host {{ llm_router_bind_address }} \
|
||||
--port {{ llm_router_port }} \
|
||||
--n-gpu-layers {{ llm_router_gpu_layers }} \
|
||||
--ctx-size {{ llm_router_ctx_size }} \
|
||||
--flash-attn {{ llm_router_flash_attn }} \
|
||||
--cache-type-k {{ llm_router_cache_type_k }} \
|
||||
--cache-type-v {{ llm_router_cache_type_v }} \
|
||||
--batch-size {{ llm_router_batch_size }} \
|
||||
--ubatch-size {{ llm_router_ubatch_size }} \
|
||||
--parallel {{ llm_router_parallel }} \
|
||||
--metrics
|
||||
|
||||
# ROUTER MODE NOTES (2026-08-12, t_0cca74a2):
|
||||
# - NO -m/--model flag: this is what enables llama-server router/supervisor mode.
|
||||
# Without -m, llama-server discovers all .gguf files in --models-dir, spawning
|
||||
# each as its own child process on demand (LRU-eviction when over models-max).
|
||||
# - --models-max {{ llm_router_models_max }} is HARDCODED TO 1.
|
||||
# Default cap is 4 simultaneous — OOM on 24GB with a 20GB model.
|
||||
# Do not increase without a VRAM budget review (see defaults/main.yml comment).
|
||||
# - --models-dir /opt/models: auto-discovers all .gguf files. Keep that directory
|
||||
# clean (Qwen-only) to avoid spurious extra entries in /v1/models.
|
||||
# - Clients select a model via "model": "<gguf-basename-without-.gguf>" in their
|
||||
# chat completion request. Hermes sends model: "<id>" on every request already.
|
||||
# - Cold model load on first request: ~30-60s for Qwen3.6-35B. First response
|
||||
# will be slow. This is expected. Document in runbook.
|
||||
# - No --jinja flag: Qwen3.6-35B uses its own embedded chat template correctly.
|
||||
# If per-model template overrides are ever needed, use --models-preset INI
|
||||
# (but note GH #23460: sampler params in presets may not work in router mode).
|
||||
#
|
||||
# SHADOW DEPLOYMENT: this unit serves port {{ llm_router_port }} only.
|
||||
# Production (llama-server-qwen, port 8002) is unchanged until Ryan approves
|
||||
# cutover after validation gates pass (see task t_0cca74a2 and proposal doc).
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
TimeoutStartSec=600
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
SyslogIdentifier=llama-server-router
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Reference in New Issue
Block a user