47 lines
2.3 KiB
Markdown
47 lines
2.3 KiB
Markdown
# Tool-Calling Model Validation Log
|
|
|
|
This file tracks the manual validation procedure required by
|
|
`astro-orbiter-multi-model-plan.md` §7 before `llama-server-toolcall` (port
|
|
8001, Mistral-Small-24B-Instruct-2501 Q3_K_M) is trusted for any real
|
|
tool-calling / Claude Code Hermes profile traffic.
|
|
|
|
This is **not automated by the role** — `tasks/verify.yml` only confirms the
|
|
endpoint is up and can produce a basic completion. The checks below are a
|
|
correctness/safety judgment call that a human runs and records here.
|
|
|
|
## Procedure (plan §7 summary)
|
|
|
|
1. **`tool_calls` emission probe** — curl a handful of known
|
|
tool-triggering prompts (e.g. "what's the weather in Austin right now")
|
|
against `POST http://10.1.71.130:8001/v1/chat/completions` with a `tools`
|
|
array defined, and confirm the response actually contains a well-formed
|
|
`tool_calls` block (correct function name, valid JSON arguments) rather
|
|
than a plain-text answer or a malformed call.
|
|
|
|
2. **Hallucination stress test** — send prompts that should **not** trigger
|
|
any tool call (general knowledge questions, casual chat, prompts that
|
|
merely mention a tool's name in passing) and confirm the model does
|
|
**not** emit a spurious `tool_calls` block. This is the primary risk
|
|
flagged in the plan given Mistral-Small's Q3_K_M quantization and its
|
|
lineage concerns around over-eager tool invocation.
|
|
|
|
3. **Shadow mode** — for a bounded period, run this instance in parallel
|
|
with whatever tool-calling path is currently in production, comparing
|
|
outputs on the same real traffic (or a recorded sample) without letting
|
|
this instance's outputs actually drive tool execution. Only cut over
|
|
once outputs are consistently correct.
|
|
|
|
See the `llm-inference-homelab` skill's `scripts/tool-calling-validation.sh`
|
|
reference for a starting curl harness shape — adapt prompts/tool schemas to
|
|
Mistral-Small's actual expected format (confirm via the GGUF's embedded
|
|
chat template / model card) rather than assuming it matches Qwen's.
|
|
|
|
## Log
|
|
|
|
| Date | Run by | Probe result | Hallucination test result | Shadow mode outcome | Decision |
|
|
|---|---|---|---|---|---|
|
|
| _(pending)_ | | | | | Not yet cut over — do not point production tool-calling traffic at :8001 |
|
|
|
|
Update this table after each validation pass. Do not remove prior rows —
|
|
this is the audit trail for "when did we decide this was safe to use."
|