homelab/ansible/roles/llm-inference-multimodel/docs/validation-log.md

Tool-Calling Model Validation Log

This file tracks the manual validation procedure required by astro-orbiter-multi-model-plan.md §7 before llama-server-toolcall (port 8001, Mistral-Small-24B-Instruct-2501 Q3_K_M) is trusted for any real tool-calling / Claude Code Hermes profile traffic.

This is not automated by the role — tasks/verify.yml only confirms the endpoint is up and can produce a basic completion. The checks below are a correctness/safety judgment call that a human runs and records here.

Procedure (plan §7 summary)

1. tool_calls emission probe — curl a handful of known tool-triggering prompts (e.g. "what's the weather in Austin right now") against POST http://10.1.71.130:8001/v1/chat/completions with a tools array defined, and confirm the response actually contains a well-formed tool_calls block (correct function name, valid JSON arguments) rather than a plain-text answer or a malformed call.

2. Hallucination stress test — send prompts that should not trigger any tool call (general knowledge questions, casual chat, prompts that merely mention a tool's name in passing) and confirm the model does not emit a spurious tool_calls block. This is the primary risk flagged in the plan given Mistral-Small's Q3_K_M quantization and its lineage concerns around over-eager tool invocation.

3. Shadow mode — for a bounded period, run this instance in parallel with whatever tool-calling path is currently in production, comparing outputs on the same real traffic (or a recorded sample) without letting this instance's outputs actually drive tool execution. Only cut over once outputs are consistently correct.

See the llm-inference-homelab skill's scripts/tool-calling-validation.sh reference for a starting curl harness shape — adapt prompts/tool schemas to Mistral-Small's actual expected format (confirm via the GGUF's embedded chat template / model card) rather than assuming it matches Qwen's.

Log

Date	Run by	Probe result	Hallucination test result	Shadow mode outcome	Decision
(pending)					Not yet cut over — do not point production tool-calling traffic at :8001

Update this table after each validation pass. Do not remove prior rows — this is the audit trail for "when did we decide this was safe to use."