rblundon/homelab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

llm-inference-multimodel: log Run 2 validation PASS (tool-calling + hallucination), preserve procedure doc

Browse Source
This commit is contained in:
Hermes Agent service account
2026-08-05 17:25:24 -05:00
parent 5dc76a8348
commit 79edb8f4e1
1 changed files with 55 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files

91
ansible/roles/llm-inference-multimodel/docs/validation-log.md
View File

@@ -1,46 +1,65 @@
# Tool-Calling Model Validation Log
# Tool-Calling Validation Log — Mistral-Small-24B-Instruct-2501 (Q3_K_M)
This file tracks the manual validation procedure required by
`astro-orbiter-multi-model-plan.md` §7 before `llama-server-toolcall` (port
8001, Mistral-Small-24B-Instruct-2501 Q3_K_M) is trusted for any real
tool-calling / Claude Code Hermes profile traffic.
## Run 2 — 2026-08-05 (PASS)
This is **not automated by the role** — `tasks/verify.yml` only confirms the
endpoint is up and can produce a basic completion. The checks below are a
correctness/safety judgment call that a human runs and records here.
**Host:** astro-orbiter (10.1.71.130), port 8001, `llama-server-toolcall.service`
**Model:** `/opt/models/mistral-small-24b-instruct-2501-Q3_K_M.gguf`
**Config change from Run 1:** added `--jinja --chat-template-file /opt/models/templates/mistral-small-tool-use.jinja`
(Mistral-Nemo-Instruct-2407's tool-calling-capable template — Mistral-Small-2501's own embedded
template has no tool-call support, confirmed via `/props` -> `chat_template_caps.supports_tools=false`
prior to the fix). Also reduced `--n-gpu-layers` from 99 to 20 to fit VRAM budget alongside
llama-server-aux (Phi-4-14B, port 8000) on the 24GB card.
## Procedure (plan §7 summary)
### Result: **PASS**
1. **`tool_calls` emission probe** — curl a handful of known
tool-triggering prompts (e.g. "what's the weather in Austin right now")
against `POST http://10.1.71.130:8001/v1/chat/completions` with a `tools`
array defined, and confirm the response actually contains a well-formed
`tool_calls` block (correct function name, valid JSON arguments) rather
than a plain-text answer or a malformed call.
| Check | Result |
|---|---|
| Correct tool_calls on 3 weather prompts | 3/3 PASS |
| No hallucinated tool_calls on 20 tool-free prompts | 20/20 PASS (4 initially timed out at 30s harness default; re-run at 150s timeout confirmed all 4 correctly returned plain text, no tool_calls, in 13.5-47.8s) |
| `chat_template_caps.supports_tools` via `/props` | `true` |
2. **Hallucination stress test** — send prompts that should **not** trigger
any tool call (general knowledge questions, casual chat, prompts that
merely mention a tool's name in passing) and confirm the model does
**not** emit a spurious `tool_calls` block. This is the primary risk
flagged in the plan given Mistral-Small's Q3_K_M quantization and its
lineage concerns around over-eager tool invocation.
### Known tradeoff — CPU offload latency
`--n-gpu-layers 20` (not full 99-layer GPU offload) means a meaningful portion of Mistral-Small
runs on CPU to fit alongside Phi-4-14B in 24GB VRAM. Measured response latency: 13.5-47.8s per
request in this validation run, vs. sub-second-to-few-second latency typical of fully GPU-resident
models at this size. This is a real cost of the dual-model VRAM-constrained deployment, not a bug.
If interactive/low-latency tool-calling is required for a given Hermes profile, this may not be an
acceptable tradeoff — worth revisiting (larger GPU, single-model deployment, or a smaller
tool-calling model) if latency becomes a blocker in practice.
3. **Shadow mode** — for a bounded period, run this instance in parallel
with whatever tool-calling path is currently in production, comparing
outputs on the same real traffic (or a recorded sample) without letting
this instance's outputs actually drive tool execution. Only cut over
once outputs are consistently correct.
### Verdict
Cleared for tool-calling-capable Hermes profiles, with the latency caveat above communicated to
whoever assigns a profile to this endpoint. Re-run this harness after any further config change to
`llama-server-toolcall.service`.
See the `llm-inference-homelab` skill's `scripts/tool-calling-validation.sh`
reference for a starting curl harness shape — adapt prompts/tool schemas to
Mistral-Small's actual expected format (confirm via the GGUF's embedded
chat template / model card) rather than assuming it matches Qwen's.
---
## Log
## Run 1 — 2026-08-05 (FAIL, superseded)
| Date | Run by | Probe result | Hallucination test result | Shadow mode outcome | Decision |
|---|---|---|---|---|---|
| _(pending)_ | | | | | Not yet cut over — do not point production tool-calling traffic at :8001 |
**Result:** FAIL — 0/3 positive tool-invocation cases; tools schema was not reaching the model at all
(`--jinja` flag missing, no tool-calling chat template configured). No hallucination observed (20/20
negative cases passed) but tool-calling was never actually exercised. See commit history for the fix
(adding `--jinja` + Mistral-Nemo-Instruct-2407 template, `5dc76a8`).
Update this table after each validation pass. Do not remove prior rows —
this is the audit trail for "when did we decide this was safe to use."
---
## Procedure (plan §7 summary, preserved from original template)
1. **`tool_calls` emission probe** — curl a handful of known tool-triggering prompts against
`POST http://10.1.71.130:8001/v1/chat/completions` with a `tools` array defined, and confirm the
response actually contains a well-formed `tool_calls` block (correct function name, valid JSON
arguments) rather than a plain-text answer or a malformed call.
2. **Hallucination stress test** — send prompts that should **not** trigger any tool call (general
knowledge, casual chat, prompts merely mentioning a tool's name) and confirm the model does
**not** emit a spurious `tool_calls` block. Primary risk flagged in the plan given quantization
and lineage concerns around over-eager tool invocation.
3. **Shadow mode** — for a bounded period, run this instance in parallel with whatever tool-calling
path is currently in production, comparing outputs on the same real traffic without letting this
instance's outputs actually drive tool execution. Only cut over once outputs are consistently
correct. **Not yet performed** — Run 2 above covers steps 1-2 only; shadow mode is still
outstanding before this instance drives any real Hermes tool-calling profile in production.
See the `llm-inference-homelab` skill's `scripts/tool-calling-validation.sh` reference for a
starting curl harness shape.