a3c92f70bf
Retired DeepSeek-R1-Distill-Qwen-32B after confirming its auto tool-choice reliability is a known, documented DeepSeek-R1-distillation limitation (trained on pure reasoning traces, no function-calling data — upstream GitHub-confirmed, not a config gap). Model choice moved to Gemma 4 26B A4B (Google, Apache 2.0, US-origin, matches Ryan's model-origin preference): - cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit — MoE (25.2B total / 3.8B active), chosen over the dense 31B variant for smaller on-disk footprint (~17.2GB vs ~20.9GB), buying more KV-cache headroom on this 24GB card - max_model_len=65536 (comfortably over Hermes's 64K floor; native context is 256K, no extension trick needed) - Native gemma4 tool-call-parser + gemma4 reasoning-parser (both registered in this host's vLLM 0.28.0) — purpose-built for this model's actual output format, not a same-family approximation - kv_cache_dtype: int4_per_token_head from the outset (learned from the DeepSeek swap's fp16->fp8->int4 trial-and-error escalation) Bug found and fixed during deployment: the repo's config.json declares quant_method 'compressed-tensors' (llm-compressor output) despite the repo name saying 'AWQ-4bit'. Passing --quantization awq explicitly caused a hard pydantic ValidationError on every startup attempt. Fix: omit the quantization field entirely and let vLLM auto-detect from the model's own config.json — confirmed clean single-attempt start, NRestarts=0, once removed. Verified live: - /health 200, /v1/models confirms max_model_len=65536 - Live completion: correct answer, no unwanted reasoning trace by default - tool_choice=auto with a clear trigger prompt: correct tool_calls response with valid JSON args — the exact test DeepSeek-R1-Distill failed (it either answered in plain text or burned tokens reasoning about how to call the tool instead of calling it) - tool_choice=auto with an irrelevant tool present: correctly answered in plain text, did not over-trigger the tool - Ansible idempotent re-run confirmed: changed=0, NRestarts=0, clean journalctl (zero error/traceback lines) after a fresh restart Known follow-up (not done here): Hindsight's HINDSIGHT_API_LLM_MODEL cluster config still references the retired DeepSeek-R1-Distill-Qwen-32B (itself a follow-up from the prior Qwen2.5-32B swap) — needs another GitOps update to point at Gemma-4-26B-A4B-it-AWQ.
14 KiB
14 KiB