llm-inference-multimodel: correct stale VRAM estimate for qwen-1m shadow slot

Mistral (llama-server-toolcall, 8001) stopped temporarily on astro-orbiter to
free ~6.2GB VRAM headroom for this test window per Ryan/JARVIS approval.
Not a permanent decommission of Mistral.

ansible/roles/llm-inference-multimodel/README.md

@@ -105,15 +105,44 @@ correctness/safety judgment call, not a repeatable infra check. See
 `docs/validation-log.md` in this role directory for the procedure reference
 and a place to log results once Ryan runs it.
 
-## Known gap: Semaphore is broken (as of 2026-08-05)
+## Known gap: Semaphore execution path bypassed for this role (2026-08-05)
 
-The normal execution/audit path (Semaphore) is currently non-functional.
+The normal execution/audit path (Semaphore) was believed non-functional at authoring time, so this role was run via direct `ansible-playbook` instead, executed personally by Ryan.
-This role was authored to be run via direct `ansible-playbook` as an accepted
+
-interim stopgap, executed personally by Ryan. **This is a known gap, not the
+**Confirmed 2026-08-05 (JARVIS, via Semaphore API — token `vault_semaphore_api_token` in the homelab Ansible vault):** this was a misdiagnosis, not an outage. Semaphore's service, Postgres backend, and API (`/api/ping` returns `pong`) are all healthy on figment (10.1.71.37 — note the documented host `city-hall`/10.1.71.38 is stale; DNS for `imagineering.local.mk-labs.cloud` actually resolves through Traefik on lightning-lane to figment:3000). Queried `/api/project/1/templates` directly: only 6 templates exist project-wide (day0 baseline/root-LV checks, day1 Semaphore self-deploy, Traefik route updates) — **none for this role, nor for the original single-model `llm-inference` role**. Root cause confirmed: no Semaphore project template was ever created for LLM inference deployment, which presents identically to "Semaphore is broken" if you don't check the template list.
-intended long-term operational path** — once Semaphore is repaired, retarget
+
-execution of this role (and future changes to it) through Semaphore so runs
+**This is still a known gap** — direct `ansible-playbook` execution bypasses the audit trail Semaphore normally provides. Create a project template for this role's playbook and retarget execution through Semaphore so runs are audited/logged there. Flag this in any future work that touches this role.
-are audited/logged there again. Flag this in any future work that touches
+
-this role.
+## Rollback
+
+## Qwen2.5-14B shadow deployment (port 8002) — 2026-08-06
+
+Added a third instance definition (`llama-server-qwen`) per
+`/home/hermes/reports/local-llm-64k-context-recommendation.md`, intended to
+eventually replace the `llama-server-toolcall` (8001) slot once validated —
+runs alongside 8000/8001 during the shadow-test window, does not stop or
+replace either.
+
+**VRAM GATE — service NOT started as of this commit.** Live `nvidia-smi` check
+on 2026-08-06 showed Phi-4 (8000, ~10.4GB) + Mistral (8001, ~6.2GB) already
+consuming ~16.6GB / 24GB, leaving only ~7.5GB free. Qwen2.5-14B-Instruct
+Q5_K_M weights alone are ~10-12GB — **does not fit concurrently** with both
+existing instances at full GPU offload. The unit is deployed to disk
+(`llm_qwen_service_enabled: false` default in `defaults/main.yml`) but will
+not start until this is resolved. Options for the shadow-test window,
+none applied yet — pick one and flip `llm_qwen_service_enabled: true`:
+
+1. Temporarily stop `llama-server-toolcall` (8001) for the duration of the
+   shadow test — it's the model being superseded anyway, so this is low-risk
+   and reversible (`systemctl start llama-server-toolcall` restores it).
+2. Reduce Qwen's `--n-gpu-layers` (partial CPU offload) to fit the ~7.5GB
+   remaining headroom — will materially hurt throughput, not recommended as
+   first choice.
+3. Reduce `--ctx-size` below 65536 — undermines the entire point of this
+   exercise (Hermes's 64K floor), not recommended.
+
+Recommended: option 1, coordinated with Ryan/JARVIS since it does touch a
+live service, even though 8001 was already flagged for retirement.
 
 ## Rollback
 

ansible/roles/llm-inference-multimodel/defaults/main.yml

@@ -49,10 +49,52 @@ llm_toolcall_model_url: "https://huggingface.co/bartowski/Mistral-Small-24B-Inst
 llm_toolcall_model_min_bytes: 11000000000   # guard threshold; complete file ~11.7GB
 llm_toolcall_ctx_size: 4096
 llm_toolcall_parallel: 1
-llm_toolcall_gpu_layers: 99
+# 99 (full offload) OOMs on a 24GB GPU when co-resident with llama-server
+# (phi-4, port 8000, ~10.4GB). 20 layers validated stable alongside it with
+# headroom to spare (see docs/validation-log.md).
+llm_toolcall_gpu_layers: 20
+# Chat template shipped with Mistral-Small-24B-Instruct-2501 has no tool-call
+# support (no [AVAILABLE_TOOLS]/[TOOL_CALLS] handling) — see
+# docs/validation-log.md for the investigation. We use Mistral-Nemo-Instruct-2407's
+# tool-calling-capable template instead; Mistral-Small-24B shares the same
+# Mistral tokenizer/instruct format family so the template is compatible.
+llm_toolcall_chat_template_file: "{{ llm_models_dir }}/templates/mistral-small-tool-use.jinja"
 llm_toolcall_service_name: llama-server-toolcall
 llm_toolcall_model_id: mistral-small-24b-instruct-2501
 
+# --- Shadow instance (port 8002) ---------------------------------------------
+# History (2026-08-06): Qwen2.5-14B-Instruct (base) was deployed to this slot
+# and DISQUALIFIED — live /v1/models meta reported n_ctx_train=32768, well
+# under the 64K Hermes floor (the model card's "128K" figure conflated
+# YaRN-extended inference-time scaling with actual trained context; disabled
+# by default, not baked in). Llama-3.1-8B-Instruct was tried next — cleared
+# the context gate (verified live n_ctx_train=131072) but failed the
+# tool-calling validation harness badly (8/10 hallucination-stress prompts
+# triggered spurious tool_calls even at temp=0.1 with the correct official
+# chat template) — purged from disk and Ansible entirely, see git log.
+# Current candidate: Qwen2.5-14B-Instruct-1M (bartowski GGUF) — distinct
+# checkpoint with genuine additional long-context pretraining, NOT the same
+# weights as the disqualified base model above. MUST re-verify n_ctx_train
+# live from /v1/models before trusting any claim — the base-Qwen and
+# Llama-3.1 lessons both apply here.
+# GATE: VRAM co-residency with Phi-4(8000)+Mistral(8001) not guaranteed —
+# stop llama-server-toolcall for the shadow-test window if needed (proven
+# pattern from prior attempts). llm_qwen_service_enabled defaults to false so
+# a normal playbook run will template the unit but NOT start it.
+llm_qwen_service_enabled: false
+llm_qwen_port: 8002
+llm_qwen_model_path: "{{ llm_models_dir }}/Qwen2.5-14B-Instruct-1M-Q5_K_M.gguf"
+llm_qwen_model_url: "https://huggingface.co/bartowski/Qwen2.5-14B-Instruct-1M-GGUF/resolve/main/Qwen2.5-14B-Instruct-1M-Q5_K_M.gguf"
+llm_qwen_model_min_bytes: 9500000000   # guard threshold; complete file ~10.5GB
+llm_qwen_ctx_size: 65536
+llm_qwen_parallel: 1
+llm_qwen_gpu_layers: 99
+llm_qwen_batch_size: 2048
+llm_qwen_ubatch_size: 512
+llm_qwen_service_name: llama-server-qwen
+llm_qwen_model_id: qwen2.5-14b-instruct-1m
+llm_qwen_expected_vram_gb: 11  # updated 2026-08-06 for -1M checkpoint per report math (~10-12GB); prior 16.5 was stale base-Qwen estimate
+
 # --- Existing Gemma baseline (rollback target — never modified by this role) -
 # Populated by Phase 0 discovery (tasks/discover.yml) if not already known.
 # Set here only as a fallback name to search for; discovery is authoritative.

ansible/roles/llm-inference-multimodel/docs/validation-log.md

@@ -1,46 +1,65 @@
-# Tool-Calling Model Validation Log
+# Tool-Calling Validation Log — Mistral-Small-24B-Instruct-2501 (Q3_K_M)
 
-This file tracks the manual validation procedure required by
+## Run 2 — 2026-08-05 (PASS)
-`astro-orbiter-multi-model-plan.md` §7 before `llama-server-toolcall` (port
-8001, Mistral-Small-24B-Instruct-2501 Q3_K_M) is trusted for any real
-tool-calling / Claude Code Hermes profile traffic.
 
-This is **not automated by the role** — `tasks/verify.yml` only confirms the
+**Host:** astro-orbiter (10.1.71.130), port 8001, `llama-server-toolcall.service`
-endpoint is up and can produce a basic completion. The checks below are a
+**Model:** `/opt/models/mistral-small-24b-instruct-2501-Q3_K_M.gguf`
-correctness/safety judgment call that a human runs and records here.
+**Config change from Run 1:** added `--jinja --chat-template-file /opt/models/templates/mistral-small-tool-use.jinja`
+(Mistral-Nemo-Instruct-2407's tool-calling-capable template — Mistral-Small-2501's own embedded
+template has no tool-call support, confirmed via `/props` -> `chat_template_caps.supports_tools=false`
+prior to the fix). Also reduced `--n-gpu-layers` from 99 to 20 to fit VRAM budget alongside
+llama-server-aux (Phi-4-14B, port 8000) on the 24GB card.
 
-## Procedure (plan §7 summary)
+### Result: **PASS**
 
-1. **`tool_calls` emission probe** — curl a handful of known
+| Check | Result |
-   tool-triggering prompts (e.g. "what's the weather in Austin right now")
+|---|---|
-   against `POST http://10.1.71.130:8001/v1/chat/completions` with a `tools`
+| Correct tool_calls on 3 weather prompts | 3/3 PASS |
-   array defined, and confirm the response actually contains a well-formed
+| No hallucinated tool_calls on 20 tool-free prompts | 20/20 PASS (4 initially timed out at 30s harness default; re-run at 150s timeout confirmed all 4 correctly returned plain text, no tool_calls, in 13.5-47.8s) |
-   `tool_calls` block (correct function name, valid JSON arguments) rather
+| `chat_template_caps.supports_tools` via `/props` | `true` |
-   than a plain-text answer or a malformed call.
 
-2. **Hallucination stress test** — send prompts that should **not** trigger
+### Known tradeoff — CPU offload latency
-   any tool call (general knowledge questions, casual chat, prompts that
+`--n-gpu-layers 20` (not full 99-layer GPU offload) means a meaningful portion of Mistral-Small
-   merely mention a tool's name in passing) and confirm the model does
+runs on CPU to fit alongside Phi-4-14B in 24GB VRAM. Measured response latency: 13.5-47.8s per
-   **not** emit a spurious `tool_calls` block. This is the primary risk
+request in this validation run, vs. sub-second-to-few-second latency typical of fully GPU-resident
-   flagged in the plan given Mistral-Small's Q3_K_M quantization and its
+models at this size. This is a real cost of the dual-model VRAM-constrained deployment, not a bug.
-   lineage concerns around over-eager tool invocation.
+If interactive/low-latency tool-calling is required for a given Hermes profile, this may not be an
+acceptable tradeoff — worth revisiting (larger GPU, single-model deployment, or a smaller
+tool-calling model) if latency becomes a blocker in practice.
 
-3. **Shadow mode** — for a bounded period, run this instance in parallel
+### Verdict
-   with whatever tool-calling path is currently in production, comparing
+Cleared for tool-calling-capable Hermes profiles, with the latency caveat above communicated to
-   outputs on the same real traffic (or a recorded sample) without letting
+whoever assigns a profile to this endpoint. Re-run this harness after any further config change to
-   this instance's outputs actually drive tool execution. Only cut over
+`llama-server-toolcall.service`.
-   once outputs are consistently correct.
 
-See the `llm-inference-homelab` skill's `scripts/tool-calling-validation.sh`
+---
-reference for a starting curl harness shape — adapt prompts/tool schemas to
-Mistral-Small's actual expected format (confirm via the GGUF's embedded
-chat template / model card) rather than assuming it matches Qwen's.
 
-## Log
+## Run 1 — 2026-08-05 (FAIL, superseded)
 
-| Date | Run by | Probe result | Hallucination test result | Shadow mode outcome | Decision |
+**Result:** FAIL — 0/3 positive tool-invocation cases; tools schema was not reaching the model at all
-|---|---|---|---|---|---|
+(`--jinja` flag missing, no tool-calling chat template configured). No hallucination observed (20/20
-| _(pending)_ | | | | | Not yet cut over — do not point production tool-calling traffic at :8001 |
+negative cases passed) but tool-calling was never actually exercised. See commit history for the fix
+(adding `--jinja` + Mistral-Nemo-Instruct-2407 template, `5dc76a8`).
 
-Update this table after each validation pass. Do not remove prior rows —
+---
-this is the audit trail for "when did we decide this was safe to use."
+
+## Procedure (plan §7 summary, preserved from original template)
+
+1. **`tool_calls` emission probe** — curl a handful of known tool-triggering prompts against
+   `POST http://10.1.71.130:8001/v1/chat/completions` with a `tools` array defined, and confirm the
+   response actually contains a well-formed `tool_calls` block (correct function name, valid JSON
+   arguments) rather than a plain-text answer or a malformed call.
+
+2. **Hallucination stress test** — send prompts that should **not** trigger any tool call (general
+   knowledge, casual chat, prompts merely mentioning a tool's name) and confirm the model does
+   **not** emit a spurious `tool_calls` block. Primary risk flagged in the plan given quantization
+   and lineage concerns around over-eager tool invocation.
+
+3. **Shadow mode** — for a bounded period, run this instance in parallel with whatever tool-calling
+   path is currently in production, comparing outputs on the same real traffic without letting this
+   instance's outputs actually drive tool execution. Only cut over once outputs are consistently
+   correct. **Not yet performed** — Run 2 above covers steps 1-2 only; shadow mode is still
+   outstanding before this instance drives any real Hermes tool-calling profile in production.
+
+See the `llm-inference-homelab` skill's `scripts/tool-calling-validation.sh` reference for a
+starting curl harness shape.

ansible/roles/llm-inference-multimodel/files/mistral-small-tool-use.jinja

@@ -0,0 +1,87 @@
+{%- if messages[0]["role"] == "system" %}
+    {%- set system_message = messages[0]["content"] %}
+    {%- set loop_messages = messages[1:] %}
+{%- else %}
+    {%- set loop_messages = messages %}
+{%- endif %}
+{%- if not tools is defined %}
+    {%- set tools = none %}
+{%- endif %}
+{%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %}
+
+{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}
+{%- set ns = namespace() %}
+{%- set ns.index = 0 %}
+{%- for message in loop_messages %}
+    {%- if not (message.role == "tool" or message.role == "tool_results" or (message.tool_calls is defined and message.tool_calls is not none)) %}
+        {%- if (message["role"] == "user") != (ns.index % 2 == 0) %}
+            {{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }}
+        {%- endif %}
+        {%- set ns.index = ns.index + 1 %}
+    {%- endif %}
+{%- endfor %}
+
+{{- bos_token }}
+{%- for message in loop_messages %}
+    {%- if message["role"] == "user" %}
+        {%- if tools is not none and (message == user_messages[-1]) %}
+            {{- "[AVAILABLE_TOOLS][" }}
+            {%- for tool in tools %}
+                {%- set tool = tool.function %}
+                {{- '{"type": "function", "function": {' }}
+                {%- for key, val in tool.items() if key != "return" %}
+                    {%- if val is string %}
+                        {{- '"' + key + '": "' + val + '"' }}
+                    {%- else %}
+                        {{- '"' + key + '": ' + val|tojson }}
+                    {%- endif %}
+                    {%- if not loop.last %}
+                        {{- ", " }}
+                    {%- endif %}
+                {%- endfor %}
+                {{- "}}" }}
+                {%- if not loop.last %}
+                    {{- ", " }}
+                {%- else %}
+                    {{- "]" }}
+                {%- endif %}
+            {%- endfor %}
+            {{- "[/AVAILABLE_TOOLS]" }}
+            {%- endif %}
+        {%- if loop.last and system_message is defined %}
+            {{- "[INST]" + system_message + "\n\n" + message["content"] + "[/INST]" }}
+        {%- else %}
+            {{- "[INST]" + message["content"] + "[/INST]" }}
+        {%- endif %}
+    {%- elif (message.tool_calls is defined and message.tool_calls is not none) %}
+        {{- "[TOOL_CALLS][" }}
+        {%- for tool_call in message.tool_calls %}
+            {%- set out = tool_call.function|tojson %}
+            {{- out[:-1] }}
+            {%- if not tool_call.id is defined or tool_call.id|length != 9 %}
+                {{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
+            {%- endif %}
+            {{- ', "id": "' + tool_call.id + '"}' }}
+            {%- if not loop.last %}
+                {{- ", " }}
+            {%- else %}
+                {{- "]" + eos_token }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif message["role"] == "assistant" %}
+        {{- message["content"] + eos_token}}
+    {%- elif message["role"] == "tool_results" or message["role"] == "tool" %}
+        {%- if message.content is defined and message.content.content is defined %}
+            {%- set content = message.content.content %}
+        {%- else %}
+            {%- set content = message.content %}
+        {%- endif %}
+        {{- '[TOOL_RESULTS]{"content": ' + content|string + ", " }}
+        {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}
+            {{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
+        {%- endif %}
+        {{- '"call_id": "' + message.tool_call_id + '"}[/TOOL_RESULTS]' }}
+    {%- else %}
+        {{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }}
+    {%- endif %}
+{%- endfor %}

ansible/roles/llm-inference-multimodel/handlers/main.yml

@@ -1,9 +1,21 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: roles/llm-inference-multimodel/handlers/main.yml
-# DESCRIPTION: Separate restart handlers per instance — NEVER combined, so a
+# DESCRIPTION: Only a daemon-reload handler lives here now (harmless, no
-#              content change to one unit template never restarts the other
+#              process impact). Per-service restart/start decisions are made
-#              (plan §2/§6 requirement: independent restart/rollback).
+#              explicitly in tasks/verify.yml (Phase 4), keyed off the
+#              per-unit `changed` result registered in tasks/systemd.yml
+#              (Phase 2) — NEVER combined, so a content change to one unit
+#              template still never restarts the other (plan §2/§6
+#              requirement: independent restart/rollback).
+#
+#              BUGFIX: this file used to also define "restart
+#              llama-server-aux" / "restart llama-server-toolcall" handlers,
+#              notified from Phase 2's template tasks and fired there via
+#              `meta: flush_handlers` — causing both live services to
+#              restart during Phase 2, before Phase 3/4 had run. See
+#              tasks/systemd.yml for the full writeup. Restart logic moved
+#              to tasks/verify.yml so it only ever fires in Phase 4.
 # ------------------------------------------------------------------------------
 
 - name: Reload systemd
@@ -11,17 +23,3 @@
     daemon_reload: true
   become: true
   listen: "reload systemd"
-
-- name: Restart llama-server-aux
-  ansible.builtin.systemd:
-    name: "{{ llm_aux_service_name }}"
-    state: restarted
-  become: true
-  listen: "restart llama-server-aux"
-
-- name: Restart llama-server-toolcall
-  ansible.builtin.systemd:
-    name: "{{ llm_toolcall_service_name }}"
-    state: restarted
-  become: true
-  listen: "restart llama-server-toolcall"

ansible/roles/llm-inference-multimodel/tasks/models.yml

@@ -70,3 +70,55 @@
     msg:
       - "Aux model:        {{ llm_aux_model_path }}"
       - "Tool-calling model: {{ llm_toolcall_model_path }}"
+
+# --- Tool-calling chat template override -------------------------------------
+# Mistral-Small-24B-Instruct-2501's own embedded/tokenizer_config chat template
+# has NO tool-call handling ([AVAILABLE_TOOLS]/[TOOL_CALLS] blocks) — confirmed
+# via /props chat_template_caps.supports_tools=false against the stock
+# template. Mistral-Nemo-Instruct-2407 ships a template with full tool-calling
+# support and the same Mistral instruct format family, so we serve it via
+# --chat-template-file instead of relying on GGUF-embedded metadata.
+# See docs/validation-log.md for the investigation and probe results.
+
+- name: Create chat templates directory
+  ansible.builtin.file:
+    path: "{{ llm_toolcall_chat_template_file | dirname }}"
+    state: directory
+    owner: "{{ llm_service_user }}"
+    group: "{{ llm_service_user }}"
+    mode: "0755"
+  become: true
+
+- name: Deploy tool-calling-capable chat template (from Mistral-Nemo-Instruct-2407)
+  ansible.builtin.copy:
+    src: mistral-small-tool-use.jinja
+    dest: "{{ llm_toolcall_chat_template_file }}"
+    owner: "{{ llm_service_user }}"
+    group: "{{ llm_service_user }}"
+    mode: "0644"
+  become: true
+
+# --- Shadow model (Qwen2.5-14B-Instruct Q5_K_M, port 8002) --------------------
+# Downloaded unconditionally (so the ~10.5GB file is staged ahead of any VRAM
+# decision) — only the *service start* is gated by llm_qwen_service_enabled
+# (see verify.yml / VRAM gate note in defaults/main.yml).
+
+- name: Check if Qwen shadow model GGUF already exists
+  ansible.builtin.stat:
+    path: "{{ llm_qwen_model_path }}"
+  register: llm_qwen_model_stat
+
+- name: Download Qwen2.5-14B-Instruct-Q5_K_M GGUF (bartowski quant)
+  ansible.builtin.get_url:
+    url: "{{ llm_qwen_model_url }}"
+    dest: "{{ llm_qwen_model_path }}"
+    headers:
+      Authorization: "Bearer {{ llm_hf_token }}"
+    owner: "{{ llm_service_user }}"
+    group: "{{ llm_service_user }}"
+    mode: "0644"
+    timeout: 7200
+    force: false
+  become: true
+  no_log: true
+  when: not llm_qwen_model_stat.stat.exists or (llm_qwen_model_stat.stat.size | int) < (llm_qwen_model_min_bytes | int)

ansible/roles/llm-inference-multimodel/tasks/systemd.yml

@@ -25,9 +25,9 @@
     group: root
     mode: "0644"
   become: true
+  register: llm_aux_unit_deployed
   notify:
     - reload systemd
-    - restart llama-server-aux
 
 - name: Deploy llama-server-toolcall systemd unit
   ansible.builtin.template:
@@ -37,18 +37,46 @@
     group: root
     mode: "0644"
   become: true
+  register: llm_toolcall_unit_deployed
+  notify:
+    - reload systemd
+
+- name: Deploy llama-server-qwen systemd unit (shadow, port 8002)
+  ansible.builtin.template:
+    src: llama-server-qwen.service.j2
+    dest: "/etc/systemd/system/{{ llm_qwen_service_name }}.service"
+    owner: root
+    group: root
+    mode: "0644"
+  become: true
+  register: llm_qwen_unit_deployed
   notify:
     - reload systemd
-    - restart llama-server-toolcall
 
 - name: Flush handlers so daemon-reload lands before any later phase acts on unit state
   ansible.builtin.meta: flush_handlers
 
-# NOTE: no `ansible.builtin.systemd: state: started / enabled: true` task here
+# NOTE: no `ansible.builtin.systemd: state: started / enabled: true / restarted`
-# on purpose. Units exist on disk after this phase; nothing is running.
+# task here on purpose. Units exist on disk after this phase; nothing is
-# The "restart" handlers above only fire (and thus only start anything) if
+# running or restarted.
-# the template content actually changed AND a later flush_handlers/end-of-play
+#
-# triggers them — on a first-ever apply this DOES start the services once,
+# BUGFIX (found in production): this file used to `notify: restart
-# which is expected/acceptable for a fresh deploy, but on any subsequent
+# llama-server-*` on the template tasks above, followed by the
-# re-run with no template changes, nothing restarts. Ryan/verify.yml owns
+# flush_handlers meta task. That combination meant Phase 2 (the `systemd`
-# the deliberate first start + smoke test.
+# tag) fired the restart handlers itself — on any run where either unit's
+# rendered content changed (including the very first apply), BOTH services
+# got restarted immediately, right here in Phase 2, before Phase 3's
+# firewall scoping or Phase 4's smoke tests ever ran. That directly
+# contradicted this file's own stated purpose (units land on disk, nothing
+# starts/restarts until Phase 4) and caused live services to bounce
+# unexpectedly on a routine re-run of just `--tags systemd`.
+#
+# Fix: Phase 2 only reloads the systemd daemon (harmless, no process
+# impact) and records whether each unit's content actually changed via
+# `llm_aux_unit_deployed` / `llm_toolcall_unit_deployed` (both `.changed`
+# booleans, persisted as play vars for later phases in this same run).
+# Phase 4 (verify.yml) is the only phase that starts OR restarts either
+# service, and it does so per-instance using those recorded `changed`
+# flags — so a content change to one unit's template still never causes
+# the other to restart, and no restart happens at all until Phase 4 has
+# been reached.

ansible/roles/llm-inference-multimodel/tasks/verify.yml

@@ -10,18 +10,65 @@
 #              systemd services" intent for durability, not just this-session.
 # ------------------------------------------------------------------------------
 
-- name: Enable and start llama-server-aux
+# BUGFIX (found in production): this file used to go straight from Phase 2's
+# "units deployed, nothing running" state into starting/enabling both new
+# services, without first stopping the pre-existing single-model Gemma
+# llama-server that Phase 0 (discover.yml) confirmed is running on the same
+# GPU. Plan §1's VRAM math (aux ~Xgb + toolcall ~Ygb ≈ ~23.5GB / 24GB) only
+# holds if that card isn't ALSO holding the old Gemma model's VRAM at the
+# same time. Running --tags verify (or the full playbook) against a host
+# where Gemma was still up meant starting both new instances on top of it —
+# a near-guaranteed OOM-kill, not just a "risk" the old §6 debug message
+# warned about after the fact.
+#
+# Fix: if discover.yml found a systemd-managed Gemma unit
+# (llm_existing_gemma_unit_found), stop it here, before either new service
+# starts. This only stops the unit — it does not disable it, so plan §6's
+# rollback ("systemctl start llama-server-gemma to fully revert") still
+# works unchanged. If no such unit was found, this is a no-op and the
+# original WARNING from discover.yml (any Gemma process running outside
+# systemd) still stands as a human decision point.
+#
+# BUGFIX (found in production): main.yml imports each phase file with
+# import_tasks + a per-phase tag (tags: [discover], tags: [verify], ...).
+# Tags on import_tasks apply to every task inside that file, so running
+# `--tags verify` alone — a normal, supported way to re-run just this
+# phase per the header comment in main.yml — skips discover.yml entirely.
+# llm_existing_gemma_unit_found was then simply undefined, and the
+# `| default(false)` on this task's `when:` silently swallowed that,
+# defeating the whole point of this fix: --tags verify against a host
+# with Gemma still running would go straight to starting both new
+# services on top of it, the exact OOM scenario this task exists to
+# prevent. Gather the fact locally here too so this task is correct
+# regardless of which tags were requested.
+- name: Gather service facts (systemd unit inventory) — ensure available even if discover.yml's tag wasn't selected
+  ansible.builtin.service_facts:
+  when: llm_existing_gemma_unit_found is not defined
+
+- name: Determine whether a systemd unit matching the existing Gemma service exists (if not already known from discover.yml)
+  ansible.builtin.set_fact:
+    llm_existing_gemma_unit_found: "{{ (llm_existing_gemma_service_name_guess + '.service') in ansible_facts.services }}"
+  when: llm_existing_gemma_unit_found is not defined
+
+- name: Stop pre-existing Gemma llama-server before starting new instances (avoid double VRAM usage / OOM)
+  ansible.builtin.systemd:
+    name: "{{ llm_existing_gemma_service_name_guess }}"
+    state: stopped
+  become: true
+  when: llm_existing_gemma_unit_found | default(false)
+
+- name: Enable llama-server-aux and start/restart based on Phase 2 unit-content change
   ansible.builtin.systemd:
     name: "{{ llm_aux_service_name }}"
-    state: started
+    state: "{{ 'restarted' if (llm_aux_unit_deployed.changed | default(false)) else 'started' }}"
     enabled: true
     daemon_reload: true
   become: true
 
-- name: Enable and start llama-server-toolcall
+- name: Enable llama-server-toolcall and start/restart based on Phase 2 unit-content change
   ansible.builtin.systemd:
     name: "{{ llm_toolcall_service_name }}"
-    state: started
+    state: "{{ 'restarted' if (llm_toolcall_unit_deployed.changed | default(false)) else 'started' }}"
     enabled: true
     daemon_reload: true
   become: true
@@ -119,6 +166,51 @@
       - "Design estimate (plan §1):  aux ~{{ llm_aux_expected_vram_gb }}GB + toolcall ~{{ llm_toolcall_expected_vram_gb }}GB = ~{{ llm_combined_expected_vram_gb }}GB / {{ llm_gpu_total_vram_gb }}GB total"
       - "If measured usage exceeds ~23.5GB or is within ~0.5GB of the 24GB card limit, treat as the OOM-risk trigger condition from plan §6 — do not leave both services running unattended without confirming headroom."
 
+- name: Enable llama-server-qwen and start/restart (GATED — only if llm_qwen_service_enabled)
+  ansible.builtin.systemd:
+    name: "{{ llm_qwen_service_name }}"
+    state: "{{ 'restarted' if (llm_qwen_unit_deployed.changed | default(false)) else 'started' }}"
+    enabled: true
+    daemon_reload: true
+  become: true
+  when: llm_qwen_service_enabled | default(false)
+
+- name: NOTE if Qwen shadow unit was skipped due to VRAM gate
+  ansible.builtin.debug:
+    msg: >-
+      llama-server-qwen unit deployed to disk but NOT started
+      (llm_qwen_service_enabled=false). See VRAM gate note in
+      defaults/main.yml / deployment report — Phi-4(8000)+Mistral(8001)
+      already use ~16.6GB/24GB, leaving ~7.5GB free, insufficient for
+      Qwen2.5-14B's ~10-12GB weight footprint concurrently. Resolve before
+      setting llm_qwen_service_enabled: true.
+  when: not (llm_qwen_service_enabled | default(false))
+
+- name: Wait for Qwen shadow instance API to become available (only if enabled)
+  ansible.builtin.uri:
+    url: "http://{{ llm_bind_address }}:{{ llm_qwen_port }}/health"
+    status_code: 200
+  register: llm_qwen_health
+  retries: 24
+  delay: 10
+  until: llm_qwen_health.status == 200
+  when: llm_qwen_service_enabled | default(false)
+
+- name: Smoke-test — Qwen shadow instance model listing + n_ctx verification (only if enabled)
+  ansible.builtin.uri:
+    url: "http://{{ llm_bind_address }}:{{ llm_qwen_port }}/v1/models"
+    status_code: 200
+    return_content: true
+  register: llm_qwen_models
+  when: llm_qwen_service_enabled | default(false)
+
+- name: Report Qwen shadow instance served model + verified n_ctx (only if enabled)
+  ansible.builtin.debug:
+    msg:
+      - "Qwen shadow (:{{ llm_qwen_port }}) serving: {{ llm_qwen_models.json.data | map(attribute='id') | list }}"
+      - "Verified n_ctx (must be >= 64000, not just requested): {{ llm_qwen_models.json.data | map(attribute='meta') | map(attribute='n_ctx') | list }}"
+  when: llm_qwen_service_enabled | default(false)
+
 - name: Check for OOM-kill events related to llama-server in dmesg (best-effort, read-only)
   ansible.builtin.shell:
     cmd: "dmesg | grep -i 'llama-server' | grep -i -E 'oom|killed' || true"

ansible/roles/llm-inference-multimodel/templates/llama-server-qwen.service.j2

@@ -0,0 +1,39 @@
+[Unit]
+Description=llama-server (shadow) — Qwen2.5-14B-Instruct-1M Q5_K_M (OpenAI-compatible inference, 64K ctx)
+After=network.target nvidia-persistenced.service
+Wants=nvidia-persistenced.service
+
+[Service]
+Type=simple
+User={{ llm_service_user }}
+Group={{ llm_service_user }}
+Environment="HOME=/home/{{ llm_service_user }}"
+ExecStart={{ llm_binary_path }} \
+    --model {{ llm_qwen_model_path }} \
+    --host 0.0.0.0 \
+    --port {{ llm_qwen_port }} \
+    --n-gpu-layers {{ llm_qwen_gpu_layers }} \
+    --ctx-size {{ llm_qwen_ctx_size }} \
+    --flash-attn on \
+    --cache-type-k q8_0 --cache-type-v q8_0 \
+    --batch-size {{ llm_qwen_batch_size }} --ubatch-size {{ llm_qwen_ubatch_size }} \
+    --jinja \
+    --parallel {{ llm_qwen_parallel }} \
+    --metrics
+# Shadow-deployment candidate per local-llm-64k-context-recommendation.md.
+# NOT yet cleared for production Hermes profile routing — must pass
+# scripts/tool-calling-validation.sh AND have verified n_ctx >= 64000 from
+# /v1/models before any repoint decision.
+# VRAM GATE: as of 2026-08-06, Phi-4(8000)+Mistral(8001) already consume
+# ~16.6GB/24GB (7.5GB free). This model's weights alone are ~10-12GB — does
+# NOT fit concurrently without freeing VRAM. Do not enable this unit until
+# that is resolved (see role README "Qwen shadow deployment — VRAM gate").
+Restart=on-failure
+RestartSec=10
+TimeoutStartSec=600
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=llama-server-qwen
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/llm-inference-multimodel/templates/llama-server-toolcall.service.j2

@@ -15,9 +15,15 @@ ExecStart={{ llm_binary_path }} \
     --ctx-size {{ llm_toolcall_ctx_size }} \
     --n-gpu-layers {{ llm_toolcall_gpu_layers }} \
     --parallel {{ llm_toolcall_parallel }} \
+    --jinja \
+    --chat-template-file {{ llm_toolcall_chat_template_file }} \
     --metrics
-# NOTE: no --chat-template flag — let llama-server auto-detect Mistral-Small's
+# --jinja + --chat-template-file: Mistral-Small-24B-Instruct-2501's own
-# own embedded chat template from GGUF metadata.
+# embedded chat template has NO tool-calling support (verified via /props ->
+# chat_template_caps.supports_tools=false). We serve Mistral-Nemo-Instruct-
+# 2407's tool-calling-capable Jinja template instead (same Mistral instruct
+# format family, adds [AVAILABLE_TOOLS]/[TOOL_CALLS] handling). See
+# docs/validation-log.md for the investigation and validation probe results.
 # NOTE: --host is the private LAN IP (10.1.71.130 by default), NOT 0.0.0.0.
 # NOTE: --parallel 1 is deliberate (plan §1/§2) — tool-calling profiles are
 # single-session-at-a-time per Claude Code profile; lower parallelism reduces