# Tool-Calling Model Validation Log

This file tracks the manual validation procedure required by
`astro-orbiter-multi-model-plan.md` §7 before `llama-server-toolcall` (port
8001, Mistral-Small-24B-Instruct-2501 Q3_K_M) is trusted for any real
tool-calling / Claude Code Hermes profile traffic.

This is **not automated by the role** — `tasks/verify.yml` only confirms the
endpoint is up and can produce a basic completion. The checks below are a
correctness/safety judgment call that a human runs and records here.

## Procedure (plan §7 summary)

1. **`tool_calls` emission probe** — curl a handful of known
   tool-triggering prompts (e.g. "what's the weather in Austin right now")
   against `POST http://10.1.71.130:8001/v1/chat/completions` with a `tools`
   array defined, and confirm the response actually contains a well-formed
   `tool_calls` block (correct function name, valid JSON arguments) rather
   than a plain-text answer or a malformed call.

2. **Hallucination stress test** — send prompts that should **not** trigger
   any tool call (general knowledge questions, casual chat, prompts that
   merely mention a tool's name in passing) and confirm the model does
   **not** emit a spurious `tool_calls` block. This is the primary risk
   flagged in the plan given Mistral-Small's Q3_K_M quantization and its
   lineage concerns around over-eager tool invocation.

3. **Shadow mode** — for a bounded period, run this instance in parallel
   with whatever tool-calling path is currently in production, comparing
   outputs on the same real traffic (or a recorded sample) without letting
   this instance's outputs actually drive tool execution. Only cut over
   once outputs are consistently correct.

See the `llm-inference-homelab` skill's `scripts/tool-calling-validation.sh`
reference for a starting curl harness shape — adapt prompts/tool schemas to
Mistral-Small's actual expected format (confirm via the GGUF's embedded
chat template / model card) rather than assuming it matches Qwen's.

## Log

| Date | Run by | Probe result | Hallucination test result | Shadow mode outcome | Decision |
|---|---|---|---|---|---|
| _(pending)_ | | | | | Not yet cut over — do not point production tool-calling traffic at :8001 |

Update this table after each validation pass. Do not remove prior rows —
this is the audit trail for "when did we decide this was safe to use."