feat(deploy-vllm): swap Qwen2.5-32B for DeepSeek-R1-Distill-Qwen-32B-AWQ (t_r1d32b_swap)
Single-model deployment per Ryan's direction: - Primary model: casperhansen/deepseek-r1-distill-qwen-32b-awq, max_model_len=32768 - nomic-embed-text-v1.5 and Qwen3-8B-AWQ both disabled (single-model requirement) - kv_cache_dtype: int4_per_token_head required to fit 32768 ctx on 24GB RTX 3090 (fp16 KV: 26GB needed, doesn't fit at any utilization; fp8 KV: OOM'd during FlashInfer warmup with ~50-150MB margin; int4 KV: clean single-attempt start) - Added kv_cache_dtype / kv_cache_memory_bytes as new optional per-model template fields in vllm.service.j2 (guarded, no effect on other models) Verified live: /health 200, /v1/models confirms max_model_len=32768, live /v1/completions smoke test + manual chat completion both passed (genuine <think> reasoning trace, correct arithmetic). NRestarts=0, steady-state VRAM 23.2GB/24.576GB. Ansible idempotent re-run confirmed changed=0. Known follow-up (not done here): Hindsight's HINDSIGHT_API_LLM_MODEL cluster config still references the retired Qwen2.5-32B-Instruct-AWQ — needs separate GitOps update to point at the new model.
This commit is contained in:
Hermes Agent service account
@@ -109,40 +109,83 @@ llm_staged_models:
|
||||
size_bytes: 5027784224
|
||||
source_repo: "bartowski/Qwen_Qwen3-8B-GGUF"
|
||||
|
||||
# --- deploy-vllm role: vllm_models override (t_e6facb19, 2026-08-31) --------
|
||||
# --- deploy-vllm role: vllm_models override (t_r1d32b_swap, 2026-09-01) -----
|
||||
# Ansible's hash_behaviour is "replace" (see ansible.cfg) — a host_vars list
|
||||
# variable REPLACES the role default list wholesale, it does not deep-merge.
|
||||
# This is therefore a full copy of roles/deploy-vllm/defaults/main.yml's
|
||||
# vllm_models with ONE change: nomic-embed-text-v1.5.enabled flipped to true,
|
||||
# now that vllm.service.j2 has an embedding-mode branch (--runner pooling
|
||||
# --convert embed --trust-remote-code) tested end-to-end in a shadow window.
|
||||
# Primary (Qwen2.5-32B-Instruct-AWQ) and aux (Qwen3-8B-AWQ) entries are
|
||||
# unchanged from role defaults — reproduced here only because the whole list
|
||||
# must be redefined together. Keep this in sync with defaults/main.yml if the
|
||||
# role's model roster changes.
|
||||
#
|
||||
# SWAP (Ryan direction, 2026-09-01): Qwen2.5-32B-Instruct-AWQ retired,
|
||||
# replaced with DeepSeek-R1-Distill-Qwen-32B-AWQ, max_model_len=32768.
|
||||
# "Single model only" — nomic-embed-text-v1.5 (embedding, :8020) and
|
||||
# Qwen3-8B-AWQ (aux, :8010, already disabled) are BOTH disabled here.
|
||||
# DeepSeek gets the full 24GB card to itself. Nothing in production
|
||||
# consumed nomic-embed at the time of this swap (Hindsight uses its own
|
||||
# bundled 384-dim embedder; OpenViking pointed at the old llama-swap
|
||||
# endpoint, already stopped) — confirmed with Ryan before disabling.
|
||||
#
|
||||
# Model choice: casperhansen/deepseek-r1-distill-qwen-32b-awq — same
|
||||
# quantizer/toolchain (AutoAWQ) as the outgoing Qwen2.5-32B-Instruct-AWQ,
|
||||
# widely used, 4-bit GEMM AWQ, ~19.3GB on disk (4 safetensors shards).
|
||||
# Architecture: Qwen2ForCausalLM (DeepSeek-R1 distilled onto Qwen2.5-32B
|
||||
# base) — same vLLM code path as the outgoing model, no new serving
|
||||
# support needed. Native max_position_embeddings=131072; we cap at 32768
|
||||
# per the task's explicit max-model-len requirement.
|
||||
#
|
||||
# VRAM math: ~19.3GB weights (4-bit AWQ) + KV cache at 32768 ctx (GQA,
|
||||
# 8 KV heads, 128 head_dim, 64 layers, fp16 KV by default) ≈ 19.3GB +
|
||||
# ~4GB KV+overhead ≈ 23.3GB — tight but the FULL 24GB card is now
|
||||
# available (no co-resident nomic-embed/Qwen3-8B taking a share, unlike
|
||||
# the outgoing Qwen2.5-32B config). gpu_memory_utilization=0.95 (role
|
||||
# default) + enforce_eager retained as the proven-stable mitigation from
|
||||
# t_e6facb19/t_ca1af9fb (avoids CUDA graph capture VRAM spike; this host's
|
||||
# only validated way to avoid crash-loop-to-stabilize behavior on this
|
||||
# card). If 0.95 OOMs at 32768 ctx once tested live, drop to 0.90 next
|
||||
# (documented fallback, same pattern as the outgoing model).
|
||||
#
|
||||
# DeepSeek-R1 output note: reasoning traces stream in <think> tags before
|
||||
# the final answer — this is expected R1-distill behavior, not a bug.
|
||||
# Model card recommends temperature 0.5-0.7 (not 0, not vLLM's greedy
|
||||
# default) to avoid repetition/incoherence; not set here (server-side
|
||||
# default), left to be set client-side per the model card's guidance —
|
||||
# flagging for whoever wires this into Hermes profile configs next.
|
||||
vllm_models:
|
||||
- id: "Qwen2.5-32B-Instruct-AWQ"
|
||||
hf_repo: "Qwen/Qwen2.5-32B-Instruct-AWQ"
|
||||
- id: "DeepSeek-R1-Distill-Qwen-32B-AWQ"
|
||||
hf_repo: "casperhansen/deepseek-r1-distill-qwen-32b-awq"
|
||||
role: primary
|
||||
quantization: awq
|
||||
port: 8000
|
||||
max_model_len: 8192
|
||||
# 0.95 (role default) OOM'd during CUDA graph capture once nomic-embed
|
||||
# (role: embedding, ~814MiB actual, not the nominal 300MB) is co-resident
|
||||
# on the same 24GB card (t_e6facb19, 2026-08-31): KV cache allocation
|
||||
# succeeded (14,720 tokens) but graph capture needed ~20MiB more than the
|
||||
# 0.95 budget left after nomic's share. Two independent, permanent
|
||||
# co-residents (unlike t_ca1af9fb's shadow-window test, which had the
|
||||
# whole 24GB free) need either a lower utilization ceiling or no graph
|
||||
# capture. enforce_eager avoids the whole cudagraph capture memory spike
|
||||
# entirely — small throughput cost, no OOM risk, safer for a fixed
|
||||
# multi-process VRAM budget than tuning utilization percentages by hand.
|
||||
# Even WITH enforce_eager, 0.95 left only ~847MiB genuinely free out of
|
||||
# 24576MiB total (23,729MiB used) and both services crash-looped 6-7x
|
||||
# during warmup/KV-cache sizing before stabilizing — too fragile for a
|
||||
# permanent two-process co-residency. Lowered to 0.90 for real headroom
|
||||
# (~1.6GiB free), confirmed clean single-attempt start with no retries.
|
||||
gpu_memory_utilization: 0.90
|
||||
max_model_len: 32768
|
||||
# VRAM correction (2026-09-01, live test): weights alone load at 18.17
|
||||
# GiB (confirmed via journalctl "Model loading took 18.17 GiB memory").
|
||||
# fp16 KV cache at 32768 ctx needs 8.0 GiB per vLLM's own error message
|
||||
# ("To serve at least one request with the model's max seq len (32768),
|
||||
# 8.0 GiB KV cache is needed") — 18.17 + 8.0 = 26.17GB, DOES NOT FIT a
|
||||
# 24GB card even at gpu_memory_utilization=1.0. Confirmed via 3 live
|
||||
# crash-loop attempts at 0.95 (usable ~23.35GiB budget): vLLM's own
|
||||
# ValueError reported "estimated maximum model length is 14528" at
|
||||
# whatever KV budget was actually available — nowhere close to 32768.
|
||||
# FIX: --kv-cache-dtype fp8 halves KV cache memory (~4.0GiB instead of
|
||||
# 8.0GiB) — 18.17 + 4.0 = ~22.2GB, fits with ~1.4GB headroom at 0.95.
|
||||
# fp8 KV cache is a standard vLLM feature (not experimental for this
|
||||
# vLLM version), minor precision loss in attention scores, no known
|
||||
# material quality impact for a distilled reasoning model at this scale.
|
||||
# VRAM correction round 3 (2026-09-01, live test): manual
|
||||
# kv_cache_memory_bytes did NOT fix it either — same OOM pattern
|
||||
# (small ~150MB alloc failing with only ~52MB actually free), meaning
|
||||
# real GPU usage at warmup exceeds what profiling/reservation account
|
||||
# for by roughly ~1GB (unaccounted FlashInfer warmup workspace buffers,
|
||||
# not sized during profiling or reservation). Rather than keep guessing
|
||||
# at a KV byte budget with ~0 margin, switched KV dtype from fp8 (8-bit)
|
||||
# to int4_per_token_head (4-bit) — roughly HALVES KV cache footprint
|
||||
# for the same 32768 ctx (~2.0GiB instead of ~4.0-4.3GiB), buying back
|
||||
# ~2GiB of real headroom to absorb whatever the unaccounted warmup
|
||||
# overhead actually is. Reverted to standard percentage-based
|
||||
# gpu_memory_utilization (no kv_cache_memory_bytes override) so vLLM's
|
||||
# own profiling determines the exact KV budget again, now with much
|
||||
# more slack in play. Some precision loss vs fp8 is expected for KV
|
||||
# cache in int4 — acceptable tradeoff for a homelab box; revisit if
|
||||
# DeepSeek's output quality visibly degrades once running.
|
||||
kv_cache_dtype: int4_per_token_head
|
||||
gpu_memory_utilization: 0.95
|
||||
enforce_eager: true
|
||||
enabled: true
|
||||
- id: "Qwen3-8B-AWQ"
|
||||
@@ -153,20 +196,7 @@ vllm_models:
|
||||
max_model_len: 32768
|
||||
gpu_memory_utilization: 0.15
|
||||
enforce_eager: true
|
||||
# ATTEMPTED enabled: true (t_5508360a, 2026-08-31) per dashboard's "start
|
||||
# vLLM and its 3 models" instruction -- does NOT fit. With the primary
|
||||
# Qwen2.5-32B-Instruct-AWQ (~18.6GB weights) + nomic-embed (~0.8GB)
|
||||
# resident, only ~1.25GiB free remained on the 23.55GiB usable budget --
|
||||
# short of the 3.53GiB gpu_memory_utilization=0.15 floor requires. This
|
||||
# is not the CUDA-graph-capture transient crash-loop t_e6facb19 saw and
|
||||
# solved with enforce_eager+lower utilization -- confirmed via
|
||||
# `journalctl`: "ValueError: Free memory on device cuda:0 (1.25/23.55
|
||||
# GiB) on startup is less than desired GPU memory utilization" on EVERY
|
||||
# one of 7 consecutive restart attempts, no stabilization. Reverted to
|
||||
# false. The 24GB card fits Qwen2.5-32B-Instruct-AWQ + nomic-embed
|
||||
# (22.6GB) with no room left for a third resident process -- 2 of the
|
||||
# "3 models" fit, not 3. Flagged in kanban t_5508360a completion.
|
||||
enabled: false
|
||||
enabled: false # single-model deployment — see swap note above
|
||||
- id: "nomic-embed-text-v1.5"
|
||||
hf_repo: "nomic-ai/nomic-embed-text-v1.5"
|
||||
role: embedding
|
||||
@@ -175,25 +205,11 @@ vllm_models:
|
||||
max_model_len: 2048
|
||||
gpu_memory_utilization: 0.05
|
||||
trust_remote_code: true
|
||||
enabled: true
|
||||
enabled: false # single-model deployment — see swap note above
|
||||
|
||||
# --- deploy-vllm role: boot persistence ENABLED (t_5508360a, 2026-08-31) -----
|
||||
# Human decision (dashboard, t_5508360a comment thread): "stop and disable
|
||||
# llama-swap and start vLLM and its 3 models" / "I understand this is a
|
||||
# breaking change" -- explicit authorization to give vLLM permanent
|
||||
# residency and retire llama-swap, accepting that the 21 Hermes profiles'
|
||||
# aux-model tasks that used to have 4 llama-swap aux models to choose from
|
||||
# (Qwen3-8B, Phi-3.5-mini, Meta-Llama-3.1-8B, Qwen2.5-Coder-14B) now have
|
||||
# exactly ONE local aux model (Qwen3-8B-AWQ via vLLM :8010) -- see
|
||||
# README.md "Critical architectural finding" section for the full VRAM
|
||||
# collision history this decision resolves.
|
||||
# Qwen3-8B-AWQ flipped enabled: true + enforce_eager: true (same crash-loop
|
||||
# mitigation applied to the primary model in t_e6facb19 -- avoids CUDA graph
|
||||
# capture's VRAM spike now that 3 processes must share the budget instead of
|
||||
# 2). vllm_service_enabled/state flipped to true/started -- this is now the
|
||||
# permanent, boot-persistent serving layer. llama-swap is being stopped +
|
||||
# disabled (not removed yet -- full unit/file teardown tracked separately in
|
||||
# t_6dff1ecc once this is verified stable).
|
||||
# --- deploy-vllm role: boot persistence (unchanged) -------------------------
|
||||
# Still permanent/boot-persistent — same policy as the outgoing Qwen2.5-32B
|
||||
# deployment (t_5508360a), just now serving one model instead of two.
|
||||
vllm_service_enabled: true
|
||||
vllm_service_state: started
|
||||
|
||||
|
||||
Reference in New Issue
Block a user