Retired DeepSeek-R1-Distill-Qwen-32B after confirming its auto tool-choice
reliability is a known, documented DeepSeek-R1-distillation limitation
(trained on pure reasoning traces, no function-calling data — upstream
GitHub-confirmed, not a config gap). Model choice moved to Gemma 4 26B A4B
(Google, Apache 2.0, US-origin, matches Ryan's model-origin preference):
- cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit — MoE (25.2B total / 3.8B active),
chosen over the dense 31B variant for smaller on-disk footprint
(~17.2GB vs ~20.9GB), buying more KV-cache headroom on this 24GB card
- max_model_len=65536 (comfortably over Hermes's 64K floor; native
context is 256K, no extension trick needed)
- Native gemma4 tool-call-parser + gemma4 reasoning-parser (both
registered in this host's vLLM 0.28.0) — purpose-built for this
model's actual output format, not a same-family approximation
- kv_cache_dtype: int4_per_token_head from the outset (learned from the
DeepSeek swap's fp16->fp8->int4 trial-and-error escalation)
Bug found and fixed during deployment: the repo's config.json declares
quant_method 'compressed-tensors' (llm-compressor output) despite the
repo name saying 'AWQ-4bit'. Passing --quantization awq explicitly
caused a hard pydantic ValidationError on every startup attempt.
Fix: omit the quantization field entirely and let vLLM auto-detect from
the model's own config.json — confirmed clean single-attempt start,
NRestarts=0, once removed.
Verified live:
- /health 200, /v1/models confirms max_model_len=65536
- Live completion: correct answer, no unwanted reasoning trace by default
- tool_choice=auto with a clear trigger prompt: correct tool_calls
response with valid JSON args — the exact test DeepSeek-R1-Distill
failed (it either answered in plain text or burned tokens reasoning
about how to call the tool instead of calling it)
- tool_choice=auto with an irrelevant tool present: correctly answered
in plain text, did not over-trigger the tool
- Ansible idempotent re-run confirmed: changed=0, NRestarts=0, clean
journalctl (zero error/traceback lines) after a fresh restart
Known follow-up (not done here): Hindsight's HINDSIGHT_API_LLM_MODEL
cluster config still references the retired DeepSeek-R1-Distill-Qwen-32B
(itself a follow-up from the prior Qwen2.5-32B swap) — needs another
GitOps update to point at Gemma-4-26B-A4B-it-AWQ.
a3c92f70bf