rblundon/homelab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
695 Commits 5 Branches 1 Tag
6455d2275237922fb62092324046a6ae50db0572
Commit Graph

11 Commits

Author SHA1 Message Date
Hermes Agent service account
6455d22752 feat(llm-router): add Meta-Llama-3.1-8B-Instruct-4bit alias; document Phi json_schema limitation (t_9adf0889) 
- Add Meta-Llama-3.1-8B-Instruct-4bit alias on Meta-Llama-3.1-8B-Instruct-Q4_K_M
  entry in the preset INI (per Ryan dashboard note).
- Document known issue: Phi-3.5-mini json_schema grammar sampler incompatibility
  in router mode (GH #23460 variant — chat template token format mismatch).
  Meta-Llama works with json_schema response_format; confirmed via live test.
- Phi-3.5-mini-instruct-8bit alias remains working for model routing;
  structured output (json_schema) fails due to the model's token format.
 2026-08-12 23:33:20 -05:00
Hermes Agent service account
a47b29d49f feat(llm-router): switch to --models-preset mode; add Phi-3.5-mini-instruct-8bit alias (t_9adf0889) 
- Add templates/llama-server-router-preset.ini.j2: defines all 3 router GGUFs
  (Qwen3.6-35B, Phi-3.5-mini-Q8_0, Meta-Llama-3.1-8B) with explicit ctx-size,
  gpu-layers, cache settings carried over from --models-dir baseline. The Phi
  entry adds alias=Phi-3.5-mini-instruct-8bit (Ryan's aux.title_generation target).

- Update templates/llama-server-router.service.j2: Jinja2 conditional emits
  --models-preset <path> when llm_router_preset_enabled=true, otherwise
  --models-dir (backward compat, default unchanged).

- Add tasks/preset.yml: deploy preset INI, restart router on change, verify
  both Phi-3.5-mini-instruct-Q8_0 (primary) and Phi-3.5-mini-instruct-8bit
  (alias) appear in /v1/models, plus Qwen and Llama IDs unchanged.

- Update defaults/main.yml: add llm_router_preset_enabled (default false) and
  llm_router_preset_path=/opt/llama-server-router-preset.ini.

- Update tasks/main.yml: import preset.yml as Phase P (gated, no-op by default).
- Update handlers/main.yml: add 'restart router' handler for preset changes.
- Add playbooks/day2_add_phi_alias.yml: single-command deployment.

GH #22364 note: --models-preset mode creates an extra 'default' entry in
/v1/models — cosmetic, does not affect model selection by name.
 2026-08-12 22:58:12 -05:00
Hermes Agent service account
9c969f783d feat(llm-inference-multimodel): bump router --models-max 1->4 on astro-orbiter (t_33acbb2e) 
Changes:
- host_vars/astro-orbiter/vars.yml: add llm_router_models_max: 4 (overrides
  conservative default of 1). Detailed VRAM OOM risk note included inline:
  worst-case 3-model co-residency ~31GB > 24GB RTX 3090. LRU eviction
  mitigates in single-user operation; flagged for Ryan's review.
- playbooks/day2_bump_router_models_max.yml: new targeted playbook; deploys
  updated router unit, restarts the live service, verifies /health 200 and
  /v1/models lists all 3 GGUFs post-restart.
- group_vars/all/semaphore.yml: add llm_router_update_unit template pointing
  at the new playbook.
- roles/llm-inference-multimodel/defaults/main.yml: update comment to reflect
  the var is now overridden in host_vars rather than 'hardcoded to 1'.
- roles/llm-inference-multimodel/templates/llama-server-router.service.j2:
  correct stale 'HARDCODED TO 1' comment — value is variable-driven.

Constraints honored:
- --parallel 1 left untouched (not in scope, not modified anywhere)
- No ad-hoc SSH/systemctl/curl state mutation; all execution via Semaphore
- No installed/vendored code patched
 2026-08-12 22:26:49 -05:00
Hermes Agent service account
3783ded62a fix: update router unit template comment — no longer a shadow deployment (t_cd0d5388) 
Stale comment said 'Production unchanged until Ryan approves cutover' —
router is now production. Replaced with accurate historical note.
 2026-08-12 20:43:31 -05:00
Hermes Agent service account
ba311a3ec6 feat(llm-inference): add llama.cpp router mode shadow deployment 
- Add tasks/router.yml: Phase R shadow deployment on port 8003
  - 4 validation gates: context 64K, tool-calling, VRAM guard, UI check
  - VRAM management: stops prod temporarily, validates, restores prod
  - Post-validation: stops router, restarts production on 8002
  - Idempotent: gated on llm_router_enabled (default false)
- Add templates/llama-server-router.service.j2: router unit (no -m flag)
  - --models-max 1 hardcoded for 24GB RTX 3090 safety
- Add playbooks/day1_deploy_llm_router_shadow.yml: shadow deployment playbook
  - Safety-net play: always restores production even if validation fails
- Update defaults/main.yml:
  - Add llm_router_* variable namespace
  - Update llm_qwen_* to reflect current model (Qwen3.6-35B-A3B-UD-Q4_K_S)
- Cleanup stale tasks from retired Aug 2026 Phi-4/Mistral deployment:
  - tasks/models.yml: remove undefined-var Phi-4/Mistral download tasks
  - tasks/firewall.yml: remove stale llm_aux_port/llm_toolcall_port refs
  - tasks/verify.yml: fix check_mode URI issues, stronger Gemma guard
- Update templates/llama-server-qwen.service.j2: update for current model

Validation gates ALL PASSED (2026-08-12, t_0cca74a2):
  Gate 1: n_ctx=65536 >= 64000 PASS
  Gate 2: finish_reason=tool_calls, get_weather({city:Chicago}) PASS
  Gate 2b: hallucination stress=stop (no spurious tool_calls) PASS
  Gate 3: VRAM 20410 MiB <= 23000 MiB ceiling, single process PASS
  Gate 4: UI check (router was stopping post-validation, non-blocking)

Production port 8002 confirmed healthy after validation.
Awaiting Ryan's cutover approval before day2 (port 8002 promotion).

Refs: t_0cca74a2
 2026-08-12 20:23:00 -05:00
Hermes Agent service account
d1f97ad5ac Phase 2 revised: consolidate astro-orbiter to single Qwen2.5-14B-1M model (port 8002) 
- Retire llama-server-aux (Phi-4, 8000) and llama-server-toolcall (Mistral-Small-24B, 8001): stopped, disabled, unit files removed from host and Ansible role
- Promote llama-server-qwen (Qwen2.5-14B-Instruct-1M, port 8002) to sole production model, serving both friday and war-machine Hermes profiles
- Verified live: n_ctx=65536/n_ctx_train=1010000, and tool_calls response via /v1/chat/completions probe (no hallucination)
- Deleted superseded GGUF weights (phi-4, mistral-small, orphaned base-Qwen, gemma-2-27b) from astro-orbiter, ~45GB reclaimed
- Updated friday and war-machine Hermes profile configs (model + compression + skills_hub aux) to point at 10.1.71.130:8002
- Ryan explicitly accepted single-model tradeoffs for both profiles
 2026-08-06 11:42:34 -05:00
Hermes Agent service account
b741f9b20b llm-inference-multimodel: repoint qwen shadow slot to Qwen2.5-14B-Instruct-1M (base Qwen disqualified, n_ctx_train=32768) 2026-08-06 10:39:45 -05:00
Hermes Agent service account
d4ff2681ac llm-inference-multimodel: fix qwen unit -- llama.cpp requires --flash-attn <on|off|auto>, not bare flag 2026-08-06 09:24:42 -05:00
Hermes Agent service account
d10255297c llm-inference-multimodel: add Qwen2.5-14B shadow instance (port 8002, gated off — VRAM co-residency not yet confirmed) 
- New llama-server-qwen systemd unit template, gated by llm_qwen_service_enabled (default false)
- Idempotent GGUF download task (bartowski Qwen2.5-14B-Instruct-Q5_K_M, stat-guarded)
- Launch flags per local-llm-64k-context-recommendation.md: ctx-size 65536, flash-attn, q8_0 KV cache, batch 2048/ubatch 512, jinja, parallel 1
- verify.yml only starts/verifies the qwen unit when llm_qwen_service_enabled=true
- README: documents live VRAM gate finding (nvidia-smi 2026-08-06: Phi-4+Mistral already ~16.6/24GB, ~7.5GB free -- insufficient for Qwen weights concurrently) and options
- Does NOT touch llama-server-aux (8000) or llama-server-toolcall (8001) service state
 2026-08-06 09:08:33 -05:00
Hermes Agent service account
5dc76a8348 llm-inference-multimodel: fix tool-calling support (jinja template + gpu-layers=20 for VRAM fit) 2026-08-05 17:13:32 -05:00
Hermes Agent service account
c3755aa29e llm-inference-multimodel: role + day1 playbook (phase 0 discover approved) 2026-08-05 15:53:31 -05:00