llm-inference-multimodel: add Qwen2.5-14B shadow instance (port 8002, gated off — VRAM co-residency not yet confirmed)

- New llama-server-qwen systemd unit template, gated by llm_qwen_service_enabled (default false)
- Idempotent GGUF download task (bartowski Qwen2.5-14B-Instruct-Q5_K_M, stat-guarded)
- Launch flags per local-llm-64k-context-recommendation.md: ctx-size 65536, flash-attn, q8_0 KV cache, batch 2048/ubatch 512, jinja, parallel 1
- verify.yml only starts/verifies the qwen unit when llm_qwen_service_enabled=true
- README: documents live VRAM gate finding (nvidia-smi 2026-08-06: Phi-4+Mistral already ~16.6/24GB, ~7.5GB free -- insufficient for Qwen weights concurrently) and options
- Does NOT touch llama-server-aux (8000) or llama-server-toolcall (8001) service state
This commit is contained in:
Hermes Agent service account
2026-08-06 09:08:33 -05:00
parent 79edb8f4e1
commit d10255297c
6 changed files with 181 additions and 8 deletions

View File

@@ -41,6 +41,18 @@
notify:
- reload systemd
- name: Deploy llama-server-qwen systemd unit (shadow, port 8002)
ansible.builtin.template:
src: llama-server-qwen.service.j2
dest: "/etc/systemd/system/{{ llm_qwen_service_name }}.service"
owner: root
group: root
mode: "0644"
become: true
register: llm_qwen_unit_deployed
notify:
- reload systemd
- name: Flush handlers so daemon-reload lands before any later phase acts on unit state
ansible.builtin.meta: flush_handlers