diff --git a/ansible/roles/llm-inference-multimodel/docs/validation-log.md b/ansible/roles/llm-inference-multimodel/docs/validation-log.md index 4efd474..7f13ebf 100644 --- a/ansible/roles/llm-inference-multimodel/docs/validation-log.md +++ b/ansible/roles/llm-inference-multimodel/docs/validation-log.md @@ -1,46 +1,65 @@ -# Tool-Calling Model Validation Log +# Tool-Calling Validation Log — Mistral-Small-24B-Instruct-2501 (Q3_K_M) -This file tracks the manual validation procedure required by -`astro-orbiter-multi-model-plan.md` §7 before `llama-server-toolcall` (port -8001, Mistral-Small-24B-Instruct-2501 Q3_K_M) is trusted for any real -tool-calling / Claude Code Hermes profile traffic. +## Run 2 — 2026-08-05 (PASS) -This is **not automated by the role** — `tasks/verify.yml` only confirms the -endpoint is up and can produce a basic completion. The checks below are a -correctness/safety judgment call that a human runs and records here. +**Host:** astro-orbiter (10.1.71.130), port 8001, `llama-server-toolcall.service` +**Model:** `/opt/models/mistral-small-24b-instruct-2501-Q3_K_M.gguf` +**Config change from Run 1:** added `--jinja --chat-template-file /opt/models/templates/mistral-small-tool-use.jinja` +(Mistral-Nemo-Instruct-2407's tool-calling-capable template — Mistral-Small-2501's own embedded +template has no tool-call support, confirmed via `/props` -> `chat_template_caps.supports_tools=false` +prior to the fix). Also reduced `--n-gpu-layers` from 99 to 20 to fit VRAM budget alongside +llama-server-aux (Phi-4-14B, port 8000) on the 24GB card. -## Procedure (plan §7 summary) +### Result: **PASS** -1. **`tool_calls` emission probe** — curl a handful of known - tool-triggering prompts (e.g. "what's the weather in Austin right now") - against `POST http://10.1.71.130:8001/v1/chat/completions` with a `tools` - array defined, and confirm the response actually contains a well-formed - `tool_calls` block (correct function name, valid JSON arguments) rather - than a plain-text answer or a malformed call. +| Check | Result | +|---|---| +| Correct tool_calls on 3 weather prompts | 3/3 PASS | +| No hallucinated tool_calls on 20 tool-free prompts | 20/20 PASS (4 initially timed out at 30s harness default; re-run at 150s timeout confirmed all 4 correctly returned plain text, no tool_calls, in 13.5-47.8s) | +| `chat_template_caps.supports_tools` via `/props` | `true` | -2. **Hallucination stress test** — send prompts that should **not** trigger - any tool call (general knowledge questions, casual chat, prompts that - merely mention a tool's name in passing) and confirm the model does - **not** emit a spurious `tool_calls` block. This is the primary risk - flagged in the plan given Mistral-Small's Q3_K_M quantization and its - lineage concerns around over-eager tool invocation. +### Known tradeoff — CPU offload latency +`--n-gpu-layers 20` (not full 99-layer GPU offload) means a meaningful portion of Mistral-Small +runs on CPU to fit alongside Phi-4-14B in 24GB VRAM. Measured response latency: 13.5-47.8s per +request in this validation run, vs. sub-second-to-few-second latency typical of fully GPU-resident +models at this size. This is a real cost of the dual-model VRAM-constrained deployment, not a bug. +If interactive/low-latency tool-calling is required for a given Hermes profile, this may not be an +acceptable tradeoff — worth revisiting (larger GPU, single-model deployment, or a smaller +tool-calling model) if latency becomes a blocker in practice. -3. **Shadow mode** — for a bounded period, run this instance in parallel - with whatever tool-calling path is currently in production, comparing - outputs on the same real traffic (or a recorded sample) without letting - this instance's outputs actually drive tool execution. Only cut over - once outputs are consistently correct. +### Verdict +Cleared for tool-calling-capable Hermes profiles, with the latency caveat above communicated to +whoever assigns a profile to this endpoint. Re-run this harness after any further config change to +`llama-server-toolcall.service`. -See the `llm-inference-homelab` skill's `scripts/tool-calling-validation.sh` -reference for a starting curl harness shape — adapt prompts/tool schemas to -Mistral-Small's actual expected format (confirm via the GGUF's embedded -chat template / model card) rather than assuming it matches Qwen's. +--- -## Log +## Run 1 — 2026-08-05 (FAIL, superseded) -| Date | Run by | Probe result | Hallucination test result | Shadow mode outcome | Decision | -|---|---|---|---|---|---| -| _(pending)_ | | | | | Not yet cut over — do not point production tool-calling traffic at :8001 | +**Result:** FAIL — 0/3 positive tool-invocation cases; tools schema was not reaching the model at all +(`--jinja` flag missing, no tool-calling chat template configured). No hallucination observed (20/20 +negative cases passed) but tool-calling was never actually exercised. See commit history for the fix +(adding `--jinja` + Mistral-Nemo-Instruct-2407 template, `5dc76a8`). -Update this table after each validation pass. Do not remove prior rows — -this is the audit trail for "when did we decide this was safe to use." +--- + +## Procedure (plan §7 summary, preserved from original template) + +1. **`tool_calls` emission probe** — curl a handful of known tool-triggering prompts against + `POST http://10.1.71.130:8001/v1/chat/completions` with a `tools` array defined, and confirm the + response actually contains a well-formed `tool_calls` block (correct function name, valid JSON + arguments) rather than a plain-text answer or a malformed call. + +2. **Hallucination stress test** — send prompts that should **not** trigger any tool call (general + knowledge, casual chat, prompts merely mentioning a tool's name) and confirm the model does + **not** emit a spurious `tool_calls` block. Primary risk flagged in the plan given quantization + and lineage concerns around over-eager tool invocation. + +3. **Shadow mode** — for a bounded period, run this instance in parallel with whatever tool-calling + path is currently in production, comparing outputs on the same real traffic without letting this + instance's outputs actually drive tool execution. Only cut over once outputs are consistently + correct. **Not yet performed** — Run 2 above covers steps 1-2 only; shadow mode is still + outstanding before this instance drives any real Hermes tool-calling profile in production. + +See the `llm-inference-homelab` skill's `scripts/tool-calling-validation.sh` reference for a +starting curl harness shape.