Tool-Calling Validation Log — Mistral-Small-24B-Instruct-2501 (Q3_K_M)

Run 2 — 2026-08-05 (PASS)

Host: astro-orbiter (10.1.71.130), port 8001, llama-server-toolcall.service Model: /opt/models/mistral-small-24b-instruct-2501-Q3_K_M.gguf Config change from Run 1: added --jinja --chat-template-file /opt/models/templates/mistral-small-tool-use.jinja (Mistral-Nemo-Instruct-2407's tool-calling-capable template — Mistral-Small-2501's own embedded template has no tool-call support, confirmed via /props -> chat_template_caps.supports_tools=false prior to the fix). Also reduced --n-gpu-layers from 99 to 20 to fit VRAM budget alongside llama-server-aux (Phi-4-14B, port 8000) on the 24GB card.

Result: PASS

Check	Result
Correct tool_calls on 3 weather prompts	3/3 PASS
No hallucinated tool_calls on 20 tool-free prompts	20/20 PASS (4 initially timed out at 30s harness default; re-run at 150s timeout confirmed all 4 correctly returned plain text, no tool_calls, in 13.5-47.8s)
`chat_template_caps.supports_tools` via `/props`	`true`

Known tradeoff — CPU offload latency

--n-gpu-layers 20 (not full 99-layer GPU offload) means a meaningful portion of Mistral-Small runs on CPU to fit alongside Phi-4-14B in 24GB VRAM. Measured response latency: 13.5-47.8s per request in this validation run, vs. sub-second-to-few-second latency typical of fully GPU-resident models at this size. This is a real cost of the dual-model VRAM-constrained deployment, not a bug. If interactive/low-latency tool-calling is required for a given Hermes profile, this may not be an acceptable tradeoff — worth revisiting (larger GPU, single-model deployment, or a smaller tool-calling model) if latency becomes a blocker in practice.

Verdict

Cleared for tool-calling-capable Hermes profiles, with the latency caveat above communicated to whoever assigns a profile to this endpoint. Re-run this harness after any further config change to llama-server-toolcall.service.

Run 1 — 2026-08-05 (FAIL, superseded)

Result: FAIL — 0/3 positive tool-invocation cases; tools schema was not reaching the model at all (--jinja flag missing, no tool-calling chat template configured). No hallucination observed (20/20 negative cases passed) but tool-calling was never actually exercised. See commit history for the fix (adding --jinja + Mistral-Nemo-Instruct-2407 template, 5dc76a8).

Procedure (plan §7 summary, preserved from original template)

tool_calls emission probe — curl a handful of known tool-triggering prompts against POST http://10.1.71.130:8001/v1/chat/completions with a tools array defined, and confirm the response actually contains a well-formed tool_calls block (correct function name, valid JSON arguments) rather than a plain-text answer or a malformed call.
Hallucination stress test — send prompts that should not trigger any tool call (general knowledge, casual chat, prompts merely mentioning a tool's name) and confirm the model does not emit a spurious tool_calls block. Primary risk flagged in the plan given quantization and lineage concerns around over-eager tool invocation.
Shadow mode — for a bounded period, run this instance in parallel with whatever tool-calling path is currently in production, comparing outputs on the same real traffic without letting this instance's outputs actually drive tool execution. Only cut over once outputs are consistently correct. Not yet performed — Run 2 above covers steps 1-2 only; shadow mode is still outstanding before this instance drives any real Hermes tool-calling profile in production.

See the llm-inference-homelab skill's scripts/tool-calling-validation.sh reference for a starting curl harness shape.

3.8 KiB Raw Blame History