Hermes Agent service account 53a55e7317 feat(deploy-vllm): swap Qwen2.5-32B for DeepSeek-R1-Distill-Qwen-32B-AWQ (t_r1d32b_swap) ...

Single-model deployment per Ryan's direction:
- Primary model: casperhansen/deepseek-r1-distill-qwen-32b-awq, max_model_len=32768
- nomic-embed-text-v1.5 and Qwen3-8B-AWQ both disabled (single-model requirement)
- kv_cache_dtype: int4_per_token_head required to fit 32768 ctx on 24GB RTX 3090
  (fp16 KV: 26GB needed, doesn't fit at any utilization; fp8 KV: OOM'd during
  FlashInfer warmup with ~50-150MB margin; int4 KV: clean single-attempt start)
- Added kv_cache_dtype / kv_cache_memory_bytes as new optional per-model
  template fields in vllm.service.j2 (guarded, no effect on other models)

Verified live: /health 200, /v1/models confirms max_model_len=32768,
live /v1/completions smoke test + manual chat completion both passed
(genuine <think> reasoning trace, correct arithmetic). NRestarts=0,
steady-state VRAM 23.2GB/24.576GB. Ansible idempotent re-run confirmed
changed=0.

Known follow-up (not done here): Hindsight's HINDSIGHT_API_LLM_MODEL
cluster config still references the retired Qwen2.5-32B-Instruct-AWQ —
needs separate GitOps update to point at the new model.

2026-08-31 20:21:27 -05:00

12 KiB

Raw Blame History

View Raw