Hermes Agent service account
|
a47b29d49f
|
feat(llm-router): switch to --models-preset mode; add Phi-3.5-mini-instruct-8bit alias (t_9adf0889)
- Add templates/llama-server-router-preset.ini.j2: defines all 3 router GGUFs
(Qwen3.6-35B, Phi-3.5-mini-Q8_0, Meta-Llama-3.1-8B) with explicit ctx-size,
gpu-layers, cache settings carried over from --models-dir baseline. The Phi
entry adds alias=Phi-3.5-mini-instruct-8bit (Ryan's aux.title_generation target).
- Update templates/llama-server-router.service.j2: Jinja2 conditional emits
--models-preset <path> when llm_router_preset_enabled=true, otherwise
--models-dir (backward compat, default unchanged).
- Add tasks/preset.yml: deploy preset INI, restart router on change, verify
both Phi-3.5-mini-instruct-Q8_0 (primary) and Phi-3.5-mini-instruct-8bit
(alias) appear in /v1/models, plus Qwen and Llama IDs unchanged.
- Update defaults/main.yml: add llm_router_preset_enabled (default false) and
llm_router_preset_path=/opt/llama-server-router-preset.ini.
- Update tasks/main.yml: import preset.yml as Phase P (gated, no-op by default).
- Update handlers/main.yml: add 'restart router' handler for preset changes.
- Add playbooks/day2_add_phi_alias.yml: single-command deployment.
GH #22364 note: --models-preset mode creates an extra 'default' entry in
/v1/models — cosmetic, does not affect model selection by name.
|
2026-08-12 22:58:12 -05:00 |
|
Hermes Agent service account
|
ba311a3ec6
|
feat(llm-inference): add llama.cpp router mode shadow deployment
- Add tasks/router.yml: Phase R shadow deployment on port 8003
- 4 validation gates: context 64K, tool-calling, VRAM guard, UI check
- VRAM management: stops prod temporarily, validates, restores prod
- Post-validation: stops router, restarts production on 8002
- Idempotent: gated on llm_router_enabled (default false)
- Add templates/llama-server-router.service.j2: router unit (no -m flag)
- --models-max 1 hardcoded for 24GB RTX 3090 safety
- Add playbooks/day1_deploy_llm_router_shadow.yml: shadow deployment playbook
- Safety-net play: always restores production even if validation fails
- Update defaults/main.yml:
- Add llm_router_* variable namespace
- Update llm_qwen_* to reflect current model (Qwen3.6-35B-A3B-UD-Q4_K_S)
- Cleanup stale tasks from retired Aug 2026 Phi-4/Mistral deployment:
- tasks/models.yml: remove undefined-var Phi-4/Mistral download tasks
- tasks/firewall.yml: remove stale llm_aux_port/llm_toolcall_port refs
- tasks/verify.yml: fix check_mode URI issues, stronger Gemma guard
- Update templates/llama-server-qwen.service.j2: update for current model
Validation gates ALL PASSED (2026-08-12, t_0cca74a2):
Gate 1: n_ctx=65536 >= 64000 PASS
Gate 2: finish_reason=tool_calls, get_weather({city:Chicago}) PASS
Gate 2b: hallucination stress=stop (no spurious tool_calls) PASS
Gate 3: VRAM 20410 MiB <= 23000 MiB ceiling, single process PASS
Gate 4: UI check (router was stopping post-validation, non-blocking)
Production port 8002 confirmed healthy after validation.
Awaiting Ryan's cutover approval before day2 (port 8002 promotion).
Refs: t_0cca74a2
|
2026-08-12 20:23:00 -05:00 |
|