homelab

rblundon/homelab

Fork 0

Commit Graph

Author	SHA1	Message	Date
Hermes Agent service account	9c969f783d	feat(llm-inference-multimodel): bump router --models-max 1->4 on astro-orbiter (t_33acbb2e) Changes: - host_vars/astro-orbiter/vars.yml: add llm_router_models_max: 4 (overrides conservative default of 1). Detailed VRAM OOM risk note included inline: worst-case 3-model co-residency ~31GB > 24GB RTX 3090. LRU eviction mitigates in single-user operation; flagged for Ryan's review. - playbooks/day2_bump_router_models_max.yml: new targeted playbook; deploys updated router unit, restarts the live service, verifies /health 200 and /v1/models lists all 3 GGUFs post-restart. - group_vars/all/semaphore.yml: add llm_router_update_unit template pointing at the new playbook. - roles/llm-inference-multimodel/defaults/main.yml: update comment to reflect the var is now overridden in host_vars rather than 'hardcoded to 1'. - roles/llm-inference-multimodel/templates/llama-server-router.service.j2: correct stale 'HARDCODED TO 1' comment — value is variable-driven. Constraints honored: - --parallel 1 left untouched (not in scope, not modified anywhere) - No ad-hoc SSH/systemctl/curl state mutation; all execution via Semaphore - No installed/vendored code patched	2026-08-12 22:26:49 -05:00
Hermes Agent service account	3783ded62a	fix: update router unit template comment — no longer a shadow deployment (t_cd0d5388) Stale comment said 'Production unchanged until Ryan approves cutover' — router is now production. Replaced with accurate historical note.	2026-08-12 20:43:31 -05:00
Hermes Agent service account	ba311a3ec6	feat(llm-inference): add llama.cpp router mode shadow deployment - Add tasks/router.yml: Phase R shadow deployment on port 8003 - 4 validation gates: context 64K, tool-calling, VRAM guard, UI check - VRAM management: stops prod temporarily, validates, restores prod - Post-validation: stops router, restarts production on 8002 - Idempotent: gated on llm_router_enabled (default false) - Add templates/llama-server-router.service.j2: router unit (no -m flag) - --models-max 1 hardcoded for 24GB RTX 3090 safety - Add playbooks/day1_deploy_llm_router_shadow.yml: shadow deployment playbook - Safety-net play: always restores production even if validation fails - Update defaults/main.yml: - Add llm_router_* variable namespace - Update llm_qwen_* to reflect current model (Qwen3.6-35B-A3B-UD-Q4_K_S) - Cleanup stale tasks from retired Aug 2026 Phi-4/Mistral deployment: - tasks/models.yml: remove undefined-var Phi-4/Mistral download tasks - tasks/firewall.yml: remove stale llm_aux_port/llm_toolcall_port refs - tasks/verify.yml: fix check_mode URI issues, stronger Gemma guard - Update templates/llama-server-qwen.service.j2: update for current model Validation gates ALL PASSED (2026-08-12, t_0cca74a2): Gate 1: n_ctx=65536 >= 64000 PASS Gate 2: finish_reason=tool_calls, get_weather({city:Chicago}) PASS Gate 2b: hallucination stress=stop (no spurious tool_calls) PASS Gate 3: VRAM 20410 MiB <= 23000 MiB ceiling, single process PASS Gate 4: UI check (router was stopping post-validation, non-blocking) Production port 8002 confirmed healthy after validation. Awaiting Ryan's cutover approval before day2 (port 8002 promotion). Refs: t_0cca74a2	2026-08-12 20:23:00 -05:00

Author

SHA1

Message

Date

Hermes Agent service account

9c969f783d

feat(llm-inference-multimodel): bump router --models-max 1->4 on astro-orbiter (t_33acbb2e)

Changes:
- host_vars/astro-orbiter/vars.yml: add llm_router_models_max: 4 (overrides
  conservative default of 1). Detailed VRAM OOM risk note included inline:
  worst-case 3-model co-residency ~31GB > 24GB RTX 3090. LRU eviction
  mitigates in single-user operation; flagged for Ryan's review.
- playbooks/day2_bump_router_models_max.yml: new targeted playbook; deploys
  updated router unit, restarts the live service, verifies /health 200 and
  /v1/models lists all 3 GGUFs post-restart.
- group_vars/all/semaphore.yml: add llm_router_update_unit template pointing
  at the new playbook.
- roles/llm-inference-multimodel/defaults/main.yml: update comment to reflect
  the var is now overridden in host_vars rather than 'hardcoded to 1'.
- roles/llm-inference-multimodel/templates/llama-server-router.service.j2:
  correct stale 'HARDCODED TO 1' comment — value is variable-driven.

Constraints honored:
- --parallel 1 left untouched (not in scope, not modified anywhere)
- No ad-hoc SSH/systemctl/curl state mutation; all execution via Semaphore
- No installed/vendored code patched

2026-08-12 22:26:49 -05:00

Hermes Agent service account

3783ded62a

fix: update router unit template comment — no longer a shadow deployment (t_cd0d5388)

Stale comment said 'Production unchanged until Ryan approves cutover' —
router is now production. Replaced with accurate historical note.

2026-08-12 20:43:31 -05:00

Hermes Agent service account

ba311a3ec6

feat(llm-inference): add llama.cpp router mode shadow deployment

- Add tasks/router.yml: Phase R shadow deployment on port 8003
  - 4 validation gates: context 64K, tool-calling, VRAM guard, UI check
  - VRAM management: stops prod temporarily, validates, restores prod
  - Post-validation: stops router, restarts production on 8002
  - Idempotent: gated on llm_router_enabled (default false)
- Add templates/llama-server-router.service.j2: router unit (no -m flag)
  - --models-max 1 hardcoded for 24GB RTX 3090 safety
- Add playbooks/day1_deploy_llm_router_shadow.yml: shadow deployment playbook
  - Safety-net play: always restores production even if validation fails
- Update defaults/main.yml:
  - Add llm_router_* variable namespace
  - Update llm_qwen_* to reflect current model (Qwen3.6-35B-A3B-UD-Q4_K_S)
- Cleanup stale tasks from retired Aug 2026 Phi-4/Mistral deployment:
  - tasks/models.yml: remove undefined-var Phi-4/Mistral download tasks
  - tasks/firewall.yml: remove stale llm_aux_port/llm_toolcall_port refs
  - tasks/verify.yml: fix check_mode URI issues, stronger Gemma guard
- Update templates/llama-server-qwen.service.j2: update for current model

Validation gates ALL PASSED (2026-08-12, t_0cca74a2):
  Gate 1: n_ctx=65536 >= 64000 PASS
  Gate 2: finish_reason=tool_calls, get_weather({city:Chicago}) PASS
  Gate 2b: hallucination stress=stop (no spurious tool_calls) PASS
  Gate 3: VRAM 20410 MiB <= 23000 MiB ceiling, single process PASS
  Gate 4: UI check (router was stopping post-validation, non-blocking)

Production port 8002 confirmed healthy after validation.
Awaiting Ryan's cutover approval before day2 (port 8002 promotion).

Refs: t_0cca74a2

2026-08-12 20:23:00 -05:00

3 Commits