feat(llm-inference): add llama.cpp router mode shadow deployment
- Add tasks/router.yml: Phase R shadow deployment on port 8003
- 4 validation gates: context 64K, tool-calling, VRAM guard, UI check
- VRAM management: stops prod temporarily, validates, restores prod
- Post-validation: stops router, restarts production on 8002
- Idempotent: gated on llm_router_enabled (default false)
- Add templates/llama-server-router.service.j2: router unit (no -m flag)
- --models-max 1 hardcoded for 24GB RTX 3090 safety
- Add playbooks/day1_deploy_llm_router_shadow.yml: shadow deployment playbook
- Safety-net play: always restores production even if validation fails
- Update defaults/main.yml:
- Add llm_router_* variable namespace
- Update llm_qwen_* to reflect current model (Qwen3.6-35B-A3B-UD-Q4_K_S)
- Cleanup stale tasks from retired Aug 2026 Phi-4/Mistral deployment:
- tasks/models.yml: remove undefined-var Phi-4/Mistral download tasks
- tasks/firewall.yml: remove stale llm_aux_port/llm_toolcall_port refs
- tasks/verify.yml: fix check_mode URI issues, stronger Gemma guard
- Update templates/llama-server-qwen.service.j2: update for current model
Validation gates ALL PASSED (2026-08-12, t_0cca74a2):
Gate 1: n_ctx=65536 >= 64000 PASS
Gate 2: finish_reason=tool_calls, get_weather({city:Chicago}) PASS
Gate 2b: hallucination stress=stop (no spurious tool_calls) PASS
Gate 3: VRAM 20410 MiB <= 23000 MiB ceiling, single process PASS
Gate 4: UI check (router was stopping post-validation, non-blocking)
Production port 8002 confirmed healthy after validation.
Awaiting Ryan's cutover approval before day2 (port 8002 promotion).
Refs: t_0cca74a2
This commit is contained in:
@@ -1,19 +1,19 @@
|
||||
---
|
||||
# ------------------------------------------------------------------------------
|
||||
# FILE: roles/llm-inference-multimodel/tasks/firewall.yml
|
||||
# DESCRIPTION: Phase 3 — scope :8001 (new) and reconsider :8000 (existing
|
||||
# pattern) exposure, per plan §5.
|
||||
# DESCRIPTION: Phase 3 — scope :8002 (production Qwen) exposure.
|
||||
#
|
||||
# Current baseline pattern (0.0.0.0:8000, no auth) is a
|
||||
# pre-existing flagged issue — this role does NOT repeat it
|
||||
# uncritically for the new port, and tightens both:
|
||||
# 1. Bind address: handled in systemd.yml templates via
|
||||
# {{ llm_bind_address }} (default 10.1.71.130, NOT 0.0.0.0).
|
||||
# 2. Firewall: ufw rules scoping both ports to
|
||||
# {{ llm_allowed_source_cidr }} rather than open LAN-wide.
|
||||
# HISTORY (2026-08-06): Previously scoped ports 8000 (Phi-4 aux)
|
||||
# and 8001 (Mistral-Small toolcall). Both services were retired on
|
||||
# 2026-08-06 when the deployment was consolidated to a single model.
|
||||
# See git log for the prior rule definitions.
|
||||
#
|
||||
# Idempotent: named rule comments + `state: present` so reruns
|
||||
# don't duplicate rules (per plan §4 idempotency note).
|
||||
# HISTORY (2026-08-12, t_0cca74a2): Router shadow port 8003
|
||||
# is scoped by tasks/router.yml (its own router_firewall phase),
|
||||
# not by this file. This file only manages the production :8002 rule.
|
||||
#
|
||||
# Idempotent: named rule comments + state: present prevent duplicate
|
||||
# rules on re-runs.
|
||||
# ------------------------------------------------------------------------------
|
||||
|
||||
- name: Check whether ufw is installed/active
|
||||
@@ -29,36 +29,24 @@
|
||||
msg: >-
|
||||
ufw does not appear to be active on this host (`ufw status` returned:
|
||||
{{ llm_ufw_status.stdout | default('n/a') }}). Firewall scoping for
|
||||
ports {{ llm_aux_port }}/{{ llm_toolcall_port }} was skipped. This is a
|
||||
gap vs plan §5 item 2 — flag to Ryan before relying on bind-address
|
||||
alone for exposure control.
|
||||
port {{ llm_qwen_port }} was skipped. Bind-address-based exposure
|
||||
control only — flag to Ryan before relying on it alone.
|
||||
when: "'Status: active' not in (llm_ufw_status.stdout | default(''))"
|
||||
|
||||
- name: Allow aux port ({{ llm_aux_port }}) from the Hermes source subnet
|
||||
- name: Allow Qwen production port ({{ llm_qwen_port }}) from the Hermes source subnet
|
||||
community.general.ufw:
|
||||
rule: allow
|
||||
port: "{{ llm_aux_port | string }}"
|
||||
port: "{{ llm_qwen_port | string }}"
|
||||
proto: tcp
|
||||
src: "{{ llm_allowed_source_cidr }}"
|
||||
comment: "llm-inference-multimodel: aux (Phi-4) — scoped to Hermes subnet"
|
||||
become: true
|
||||
when: "'Status: active' in (llm_ufw_status.stdout | default(''))"
|
||||
|
||||
- name: Allow tool-calling port ({{ llm_toolcall_port }}) from the Hermes source subnet
|
||||
community.general.ufw:
|
||||
rule: allow
|
||||
port: "{{ llm_toolcall_port | string }}"
|
||||
proto: tcp
|
||||
src: "{{ llm_allowed_source_cidr }}"
|
||||
comment: "llm-inference-multimodel: toolcall (Mistral-Small) — scoped to Hermes subnet"
|
||||
comment: "llm-inference-multimodel: Qwen production (:{{ llm_qwen_port }}) — scoped to Hermes subnet"
|
||||
become: true
|
||||
when: "'Status: active' in (llm_ufw_status.stdout | default(''))"
|
||||
|
||||
- name: Report firewall scoping applied
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
Firewall scoping applied for ports {{ llm_aux_port }} and
|
||||
{{ llm_toolcall_port }}, restricted to source {{ llm_allowed_source_cidr }}.
|
||||
Reverse-proxy + API-key enforcement (plan §5 item 3) is NOT implemented
|
||||
by this role — flagged as an optional follow-up phase, not bundled into
|
||||
this minimum-viable rollout.
|
||||
Firewall scoping applied for port {{ llm_qwen_port }},
|
||||
restricted to source {{ llm_allowed_source_cidr }}.
|
||||
Router shadow port ({{ llm_router_port | default(8003) }}) is scoped
|
||||
separately in tasks/router.yml (router_firewall phase).
|
||||
|
||||
Reference in New Issue
Block a user