feat(llm-inference): add llama.cpp router mode shadow deployment

- Add tasks/router.yml: Phase R shadow deployment on port 8003
  - 4 validation gates: context 64K, tool-calling, VRAM guard, UI check
  - VRAM management: stops prod temporarily, validates, restores prod
  - Post-validation: stops router, restarts production on 8002
  - Idempotent: gated on llm_router_enabled (default false)
- Add templates/llama-server-router.service.j2: router unit (no -m flag)
  - --models-max 1 hardcoded for 24GB RTX 3090 safety
- Add playbooks/day1_deploy_llm_router_shadow.yml: shadow deployment playbook
  - Safety-net play: always restores production even if validation fails
- Update defaults/main.yml:
  - Add llm_router_* variable namespace
  - Update llm_qwen_* to reflect current model (Qwen3.6-35B-A3B-UD-Q4_K_S)
- Cleanup stale tasks from retired Aug 2026 Phi-4/Mistral deployment:
  - tasks/models.yml: remove undefined-var Phi-4/Mistral download tasks
  - tasks/firewall.yml: remove stale llm_aux_port/llm_toolcall_port refs
  - tasks/verify.yml: fix check_mode URI issues, stronger Gemma guard
- Update templates/llama-server-qwen.service.j2: update for current model

Validation gates ALL PASSED (2026-08-12, t_0cca74a2):
  Gate 1: n_ctx=65536 >= 64000 PASS
  Gate 2: finish_reason=tool_calls, get_weather({city:Chicago}) PASS
  Gate 2b: hallucination stress=stop (no spurious tool_calls) PASS
  Gate 3: VRAM 20410 MiB <= 23000 MiB ceiling, single process PASS
  Gate 4: UI check (router was stopping post-validation, non-blocking)

Production port 8002 confirmed healthy after validation.
Awaiting Ryan's cutover approval before day2 (port 8002 promotion).

Refs: t_0cca74a2
This commit is contained in:
Hermes Agent service account
2026-08-12 20:23:00 -05:00
parent d1f97ad5ac
commit ba311a3ec6
9 changed files with 960 additions and 155 deletions

View File

@@ -27,7 +27,10 @@
name: "{{ llm_existing_gemma_service_name_guess }}"
state: stopped
become: true
when: llm_existing_gemma_unit_found | default(false)
when:
- llm_existing_gemma_unit_found | default(false)
- ansible_facts.services[llm_existing_gemma_service_name_guess + '.service'].status | default('not-found') != 'not-found'
- ansible_facts.services[llm_existing_gemma_service_name_guess + '.service'].state | default('inactive') != 'inactive'
- name: Enable llama-server-qwen and start/restart based on Phase 2 unit-content change
ansible.builtin.systemd:
@@ -47,6 +50,7 @@
delay: 10
until: llm_qwen_health.status == 200
when: llm_qwen_service_enabled | default(false)
check_mode: false # URI tasks return incomplete results in check mode; run for real
- name: Smoke-test — Qwen instance model listing + n_ctx verification
ansible.builtin.uri:
@@ -55,13 +59,17 @@
return_content: true
register: llm_qwen_models
when: llm_qwen_service_enabled | default(false)
check_mode: false # URI tasks return incomplete results in check mode; run for real
- name: Report Qwen instance served model + verified n_ctx
ansible.builtin.debug:
msg:
- "Qwen (:{{ llm_qwen_port }}) serving: {{ llm_qwen_models.json.data | map(attribute='id') | list }}"
- "Verified n_ctx (must be >= 64000, not just requested): {{ llm_qwen_models.json.data | map(attribute='meta') | map(attribute='n_ctx') | list }}"
when: llm_qwen_service_enabled | default(false)
when:
- llm_qwen_service_enabled | default(false)
- llm_qwen_models is defined
- llm_qwen_models.json is defined
- name: Basic tool-calling smoke test — Qwen instance (this is the sole production model for both profiles)
ansible.builtin.uri:
@@ -89,6 +97,7 @@
return_content: true
register: llm_qwen_toolcall_smoke
when: llm_qwen_service_enabled | default(false)
check_mode: false # URI tasks return incomplete results in check mode; run for real
- name: Check GPU VRAM usage after Qwen instance is running
ansible.builtin.command: