# Tool-Calling Validation Log — Mistral-Small-24B-Instruct-2501 (Q3_K_M)

## Run 2 — 2026-08-05 (PASS)

**Host:** astro-orbiter (10.1.71.130), port 8001, `llama-server-toolcall.service`
**Model:** `/opt/models/mistral-small-24b-instruct-2501-Q3_K_M.gguf`
**Config change from Run 1:** added `--jinja --chat-template-file /opt/models/templates/mistral-small-tool-use.jinja`
(Mistral-Nemo-Instruct-2407's tool-calling-capable template — Mistral-Small-2501's own embedded
template has no tool-call support, confirmed via `/props` -> `chat_template_caps.supports_tools=false`
prior to the fix). Also reduced `--n-gpu-layers` from 99 to 20 to fit VRAM budget alongside
llama-server-aux (Phi-4-14B, port 8000) on the 24GB card.

### Result: **PASS**

| Check | Result |
|---|---|
| Correct tool_calls on 3 weather prompts | 3/3 PASS |
| No hallucinated tool_calls on 20 tool-free prompts | 20/20 PASS (4 initially timed out at 30s harness default; re-run at 150s timeout confirmed all 4 correctly returned plain text, no tool_calls, in 13.5-47.8s) |
| `chat_template_caps.supports_tools` via `/props` | `true` |

### Known tradeoff — CPU offload latency
`--n-gpu-layers 20` (not full 99-layer GPU offload) means a meaningful portion of Mistral-Small
runs on CPU to fit alongside Phi-4-14B in 24GB VRAM. Measured response latency: 13.5-47.8s per
request in this validation run, vs. sub-second-to-few-second latency typical of fully GPU-resident
models at this size. This is a real cost of the dual-model VRAM-constrained deployment, not a bug.
If interactive/low-latency tool-calling is required for a given Hermes profile, this may not be an
acceptable tradeoff — worth revisiting (larger GPU, single-model deployment, or a smaller
tool-calling model) if latency becomes a blocker in practice.

### Verdict
Cleared for tool-calling-capable Hermes profiles, with the latency caveat above communicated to
whoever assigns a profile to this endpoint. Re-run this harness after any further config change to
`llama-server-toolcall.service`.

---

## Run 1 — 2026-08-05 (FAIL, superseded)

**Result:** FAIL — 0/3 positive tool-invocation cases; tools schema was not reaching the model at all
(`--jinja` flag missing, no tool-calling chat template configured). No hallucination observed (20/20
negative cases passed) but tool-calling was never actually exercised. See commit history for the fix
(adding `--jinja` + Mistral-Nemo-Instruct-2407 template, `5dc76a8`).

---

## Procedure (plan §7 summary, preserved from original template)

1. **`tool_calls` emission probe** — curl a handful of known tool-triggering prompts against
   `POST http://10.1.71.130:8001/v1/chat/completions` with a `tools` array defined, and confirm the
   response actually contains a well-formed `tool_calls` block (correct function name, valid JSON
   arguments) rather than a plain-text answer or a malformed call.

2. **Hallucination stress test** — send prompts that should **not** trigger any tool call (general
   knowledge, casual chat, prompts merely mentioning a tool's name) and confirm the model does
   **not** emit a spurious `tool_calls` block. Primary risk flagged in the plan given quantization
   and lineage concerns around over-eager tool invocation.

3. **Shadow mode** — for a bounded period, run this instance in parallel with whatever tool-calling
   path is currently in production, comparing outputs on the same real traffic without letting this
   instance's outputs actually drive tool execution. Only cut over once outputs are consistently
   correct. **Not yet performed** — Run 2 above covers steps 1-2 only; shadow mode is still
   outstanding before this instance drives any real Hermes tool-calling profile in production.

See the `llm-inference-homelab` skill's `scripts/tool-calling-validation.sh` reference for a
starting curl harness shape.