66 lines
3.8 KiB
Markdown
66 lines
3.8 KiB
Markdown
# Tool-Calling Validation Log — Mistral-Small-24B-Instruct-2501 (Q3_K_M)
|
|
|
|
## Run 2 — 2026-08-05 (PASS)
|
|
|
|
**Host:** astro-orbiter (10.1.71.130), port 8001, `llama-server-toolcall.service`
|
|
**Model:** `/opt/models/mistral-small-24b-instruct-2501-Q3_K_M.gguf`
|
|
**Config change from Run 1:** added `--jinja --chat-template-file /opt/models/templates/mistral-small-tool-use.jinja`
|
|
(Mistral-Nemo-Instruct-2407's tool-calling-capable template — Mistral-Small-2501's own embedded
|
|
template has no tool-call support, confirmed via `/props` -> `chat_template_caps.supports_tools=false`
|
|
prior to the fix). Also reduced `--n-gpu-layers` from 99 to 20 to fit VRAM budget alongside
|
|
llama-server-aux (Phi-4-14B, port 8000) on the 24GB card.
|
|
|
|
### Result: **PASS**
|
|
|
|
| Check | Result |
|
|
|---|---|
|
|
| Correct tool_calls on 3 weather prompts | 3/3 PASS |
|
|
| No hallucinated tool_calls on 20 tool-free prompts | 20/20 PASS (4 initially timed out at 30s harness default; re-run at 150s timeout confirmed all 4 correctly returned plain text, no tool_calls, in 13.5-47.8s) |
|
|
| `chat_template_caps.supports_tools` via `/props` | `true` |
|
|
|
|
### Known tradeoff — CPU offload latency
|
|
`--n-gpu-layers 20` (not full 99-layer GPU offload) means a meaningful portion of Mistral-Small
|
|
runs on CPU to fit alongside Phi-4-14B in 24GB VRAM. Measured response latency: 13.5-47.8s per
|
|
request in this validation run, vs. sub-second-to-few-second latency typical of fully GPU-resident
|
|
models at this size. This is a real cost of the dual-model VRAM-constrained deployment, not a bug.
|
|
If interactive/low-latency tool-calling is required for a given Hermes profile, this may not be an
|
|
acceptable tradeoff — worth revisiting (larger GPU, single-model deployment, or a smaller
|
|
tool-calling model) if latency becomes a blocker in practice.
|
|
|
|
### Verdict
|
|
Cleared for tool-calling-capable Hermes profiles, with the latency caveat above communicated to
|
|
whoever assigns a profile to this endpoint. Re-run this harness after any further config change to
|
|
`llama-server-toolcall.service`.
|
|
|
|
---
|
|
|
|
## Run 1 — 2026-08-05 (FAIL, superseded)
|
|
|
|
**Result:** FAIL — 0/3 positive tool-invocation cases; tools schema was not reaching the model at all
|
|
(`--jinja` flag missing, no tool-calling chat template configured). No hallucination observed (20/20
|
|
negative cases passed) but tool-calling was never actually exercised. See commit history for the fix
|
|
(adding `--jinja` + Mistral-Nemo-Instruct-2407 template, `5dc76a8`).
|
|
|
|
---
|
|
|
|
## Procedure (plan §7 summary, preserved from original template)
|
|
|
|
1. **`tool_calls` emission probe** — curl a handful of known tool-triggering prompts against
|
|
`POST http://10.1.71.130:8001/v1/chat/completions` with a `tools` array defined, and confirm the
|
|
response actually contains a well-formed `tool_calls` block (correct function name, valid JSON
|
|
arguments) rather than a plain-text answer or a malformed call.
|
|
|
|
2. **Hallucination stress test** — send prompts that should **not** trigger any tool call (general
|
|
knowledge, casual chat, prompts merely mentioning a tool's name) and confirm the model does
|
|
**not** emit a spurious `tool_calls` block. Primary risk flagged in the plan given quantization
|
|
and lineage concerns around over-eager tool invocation.
|
|
|
|
3. **Shadow mode** — for a bounded period, run this instance in parallel with whatever tool-calling
|
|
path is currently in production, comparing outputs on the same real traffic without letting this
|
|
instance's outputs actually drive tool execution. Only cut over once outputs are consistently
|
|
correct. **Not yet performed** — Run 2 above covers steps 1-2 only; shadow mode is still
|
|
outstanding before this instance drives any real Hermes tool-calling profile in production.
|
|
|
|
See the `llm-inference-homelab` skill's `scripts/tool-calling-validation.sh` reference for a
|
|
starting curl harness shape.
|