diff --git a/ansible/host_vars/astro-orbiter/vars.yml b/ansible/host_vars/astro-orbiter/vars.yml index 27bf806..330a0b6 100644 --- a/ansible/host_vars/astro-orbiter/vars.yml +++ b/ansible/host_vars/astro-orbiter/vars.yml @@ -109,40 +109,83 @@ llm_staged_models: size_bytes: 5027784224 source_repo: "bartowski/Qwen_Qwen3-8B-GGUF" -# --- deploy-vllm role: vllm_models override (t_e6facb19, 2026-08-31) -------- +# --- deploy-vllm role: vllm_models override (t_r1d32b_swap, 2026-09-01) ----- # Ansible's hash_behaviour is "replace" (see ansible.cfg) — a host_vars list # variable REPLACES the role default list wholesale, it does not deep-merge. -# This is therefore a full copy of roles/deploy-vllm/defaults/main.yml's -# vllm_models with ONE change: nomic-embed-text-v1.5.enabled flipped to true, -# now that vllm.service.j2 has an embedding-mode branch (--runner pooling -# --convert embed --trust-remote-code) tested end-to-end in a shadow window. -# Primary (Qwen2.5-32B-Instruct-AWQ) and aux (Qwen3-8B-AWQ) entries are -# unchanged from role defaults — reproduced here only because the whole list -# must be redefined together. Keep this in sync with defaults/main.yml if the -# role's model roster changes. +# +# SWAP (Ryan direction, 2026-09-01): Qwen2.5-32B-Instruct-AWQ retired, +# replaced with DeepSeek-R1-Distill-Qwen-32B-AWQ, max_model_len=32768. +# "Single model only" — nomic-embed-text-v1.5 (embedding, :8020) and +# Qwen3-8B-AWQ (aux, :8010, already disabled) are BOTH disabled here. +# DeepSeek gets the full 24GB card to itself. Nothing in production +# consumed nomic-embed at the time of this swap (Hindsight uses its own +# bundled 384-dim embedder; OpenViking pointed at the old llama-swap +# endpoint, already stopped) — confirmed with Ryan before disabling. +# +# Model choice: casperhansen/deepseek-r1-distill-qwen-32b-awq — same +# quantizer/toolchain (AutoAWQ) as the outgoing Qwen2.5-32B-Instruct-AWQ, +# widely used, 4-bit GEMM AWQ, ~19.3GB on disk (4 safetensors shards). +# Architecture: Qwen2ForCausalLM (DeepSeek-R1 distilled onto Qwen2.5-32B +# base) — same vLLM code path as the outgoing model, no new serving +# support needed. Native max_position_embeddings=131072; we cap at 32768 +# per the task's explicit max-model-len requirement. +# +# VRAM math: ~19.3GB weights (4-bit AWQ) + KV cache at 32768 ctx (GQA, +# 8 KV heads, 128 head_dim, 64 layers, fp16 KV by default) ≈ 19.3GB + +# ~4GB KV+overhead ≈ 23.3GB — tight but the FULL 24GB card is now +# available (no co-resident nomic-embed/Qwen3-8B taking a share, unlike +# the outgoing Qwen2.5-32B config). gpu_memory_utilization=0.95 (role +# default) + enforce_eager retained as the proven-stable mitigation from +# t_e6facb19/t_ca1af9fb (avoids CUDA graph capture VRAM spike; this host's +# only validated way to avoid crash-loop-to-stabilize behavior on this +# card). If 0.95 OOMs at 32768 ctx once tested live, drop to 0.90 next +# (documented fallback, same pattern as the outgoing model). +# +# DeepSeek-R1 output note: reasoning traces stream in tags before +# the final answer — this is expected R1-distill behavior, not a bug. +# Model card recommends temperature 0.5-0.7 (not 0, not vLLM's greedy +# default) to avoid repetition/incoherence; not set here (server-side +# default), left to be set client-side per the model card's guidance — +# flagging for whoever wires this into Hermes profile configs next. vllm_models: - - id: "Qwen2.5-32B-Instruct-AWQ" - hf_repo: "Qwen/Qwen2.5-32B-Instruct-AWQ" + - id: "DeepSeek-R1-Distill-Qwen-32B-AWQ" + hf_repo: "casperhansen/deepseek-r1-distill-qwen-32b-awq" role: primary quantization: awq port: 8000 - max_model_len: 8192 - # 0.95 (role default) OOM'd during CUDA graph capture once nomic-embed - # (role: embedding, ~814MiB actual, not the nominal 300MB) is co-resident - # on the same 24GB card (t_e6facb19, 2026-08-31): KV cache allocation - # succeeded (14,720 tokens) but graph capture needed ~20MiB more than the - # 0.95 budget left after nomic's share. Two independent, permanent - # co-residents (unlike t_ca1af9fb's shadow-window test, which had the - # whole 24GB free) need either a lower utilization ceiling or no graph - # capture. enforce_eager avoids the whole cudagraph capture memory spike - # entirely — small throughput cost, no OOM risk, safer for a fixed - # multi-process VRAM budget than tuning utilization percentages by hand. - # Even WITH enforce_eager, 0.95 left only ~847MiB genuinely free out of - # 24576MiB total (23,729MiB used) and both services crash-looped 6-7x - # during warmup/KV-cache sizing before stabilizing — too fragile for a - # permanent two-process co-residency. Lowered to 0.90 for real headroom - # (~1.6GiB free), confirmed clean single-attempt start with no retries. - gpu_memory_utilization: 0.90 + max_model_len: 32768 + # VRAM correction (2026-09-01, live test): weights alone load at 18.17 + # GiB (confirmed via journalctl "Model loading took 18.17 GiB memory"). + # fp16 KV cache at 32768 ctx needs 8.0 GiB per vLLM's own error message + # ("To serve at least one request with the model's max seq len (32768), + # 8.0 GiB KV cache is needed") — 18.17 + 8.0 = 26.17GB, DOES NOT FIT a + # 24GB card even at gpu_memory_utilization=1.0. Confirmed via 3 live + # crash-loop attempts at 0.95 (usable ~23.35GiB budget): vLLM's own + # ValueError reported "estimated maximum model length is 14528" at + # whatever KV budget was actually available — nowhere close to 32768. + # FIX: --kv-cache-dtype fp8 halves KV cache memory (~4.0GiB instead of + # 8.0GiB) — 18.17 + 4.0 = ~22.2GB, fits with ~1.4GB headroom at 0.95. + # fp8 KV cache is a standard vLLM feature (not experimental for this + # vLLM version), minor precision loss in attention scores, no known + # material quality impact for a distilled reasoning model at this scale. + # VRAM correction round 3 (2026-09-01, live test): manual + # kv_cache_memory_bytes did NOT fix it either — same OOM pattern + # (small ~150MB alloc failing with only ~52MB actually free), meaning + # real GPU usage at warmup exceeds what profiling/reservation account + # for by roughly ~1GB (unaccounted FlashInfer warmup workspace buffers, + # not sized during profiling or reservation). Rather than keep guessing + # at a KV byte budget with ~0 margin, switched KV dtype from fp8 (8-bit) + # to int4_per_token_head (4-bit) — roughly HALVES KV cache footprint + # for the same 32768 ctx (~2.0GiB instead of ~4.0-4.3GiB), buying back + # ~2GiB of real headroom to absorb whatever the unaccounted warmup + # overhead actually is. Reverted to standard percentage-based + # gpu_memory_utilization (no kv_cache_memory_bytes override) so vLLM's + # own profiling determines the exact KV budget again, now with much + # more slack in play. Some precision loss vs fp8 is expected for KV + # cache in int4 — acceptable tradeoff for a homelab box; revisit if + # DeepSeek's output quality visibly degrades once running. + kv_cache_dtype: int4_per_token_head + gpu_memory_utilization: 0.95 enforce_eager: true enabled: true - id: "Qwen3-8B-AWQ" @@ -153,20 +196,7 @@ vllm_models: max_model_len: 32768 gpu_memory_utilization: 0.15 enforce_eager: true - # ATTEMPTED enabled: true (t_5508360a, 2026-08-31) per dashboard's "start - # vLLM and its 3 models" instruction -- does NOT fit. With the primary - # Qwen2.5-32B-Instruct-AWQ (~18.6GB weights) + nomic-embed (~0.8GB) - # resident, only ~1.25GiB free remained on the 23.55GiB usable budget -- - # short of the 3.53GiB gpu_memory_utilization=0.15 floor requires. This - # is not the CUDA-graph-capture transient crash-loop t_e6facb19 saw and - # solved with enforce_eager+lower utilization -- confirmed via - # `journalctl`: "ValueError: Free memory on device cuda:0 (1.25/23.55 - # GiB) on startup is less than desired GPU memory utilization" on EVERY - # one of 7 consecutive restart attempts, no stabilization. Reverted to - # false. The 24GB card fits Qwen2.5-32B-Instruct-AWQ + nomic-embed - # (22.6GB) with no room left for a third resident process -- 2 of the - # "3 models" fit, not 3. Flagged in kanban t_5508360a completion. - enabled: false + enabled: false # single-model deployment — see swap note above - id: "nomic-embed-text-v1.5" hf_repo: "nomic-ai/nomic-embed-text-v1.5" role: embedding @@ -175,25 +205,11 @@ vllm_models: max_model_len: 2048 gpu_memory_utilization: 0.05 trust_remote_code: true - enabled: true + enabled: false # single-model deployment — see swap note above -# --- deploy-vllm role: boot persistence ENABLED (t_5508360a, 2026-08-31) ----- -# Human decision (dashboard, t_5508360a comment thread): "stop and disable -# llama-swap and start vLLM and its 3 models" / "I understand this is a -# breaking change" -- explicit authorization to give vLLM permanent -# residency and retire llama-swap, accepting that the 21 Hermes profiles' -# aux-model tasks that used to have 4 llama-swap aux models to choose from -# (Qwen3-8B, Phi-3.5-mini, Meta-Llama-3.1-8B, Qwen2.5-Coder-14B) now have -# exactly ONE local aux model (Qwen3-8B-AWQ via vLLM :8010) -- see -# README.md "Critical architectural finding" section for the full VRAM -# collision history this decision resolves. -# Qwen3-8B-AWQ flipped enabled: true + enforce_eager: true (same crash-loop -# mitigation applied to the primary model in t_e6facb19 -- avoids CUDA graph -# capture's VRAM spike now that 3 processes must share the budget instead of -# 2). vllm_service_enabled/state flipped to true/started -- this is now the -# permanent, boot-persistent serving layer. llama-swap is being stopped + -# disabled (not removed yet -- full unit/file teardown tracked separately in -# t_6dff1ecc once this is verified stable). +# --- deploy-vllm role: boot persistence (unchanged) ------------------------- +# Still permanent/boot-persistent — same policy as the outgoing Qwen2.5-32B +# deployment (t_5508360a), just now serving one model instead of two. vllm_service_enabled: true vllm_service_state: started diff --git a/ansible/roles/deploy-vllm/README.md b/ansible/roles/deploy-vllm/README.md index a66d1b2..9c2416c 100644 --- a/ansible/roles/deploy-vllm/README.md +++ b/ansible/roles/deploy-vllm/README.md @@ -57,6 +57,11 @@ systemd units on distinct ports, not a single multiplexed server (unlike llama-swap's matrix DSL). Today's phase enables only the primary model; flip `enabled: true` on the others as VRAM allows (see "Phased Strategy"): +**⚠️ Table below reflects the ORIGINAL Qwen2.5-32B deployment. As of +2026-09-01 (t_r1d32b_swap) the primary model is +`DeepSeek-R1-Distill-Qwen-32B-AWQ`, single-model only (nomic-embed also +disabled) — see the "SUPERSEDED" section further down for current state.** + | id | hf_repo | role | port | quant | enabled | |---|---|---|---|---|---| | Qwen2.5-32B-Instruct-AWQ | Qwen/Qwen2.5-32B-Instruct-AWQ | primary | 8000 | awq | **true** | @@ -309,6 +314,95 @@ aux-model migration was authorized or attempted in this task. If those 21 profiles need a replacement aux-model path, that is separate, new, explicitly-scoped follow-up work, not implied by this decision. +## SUPERSEDED (t_r1d32b_swap, 2026-09-01): Qwen2.5-32B-Instruct-AWQ retired, replaced with DeepSeek-R1-Distill-Qwen-32B-AWQ, single-model deployment + +Ryan direction: "Swap Qwen2.5-32B for DeepSeek-R1-Distill-Qwen-32B, +max-model-len 32768. Single model only." Confirmed with Ryan that "single +model only" includes disabling `nomic-embed-text-v1.5` (:8020) as well — +nothing in production consumed it (Hindsight uses its own bundled 384-dim +embedder; OpenViking pointed at the retired llama-swap endpoint). DeepSeek +gets the entire 24GB card. + +**Model choice:** `casperhansen/deepseek-r1-distill-qwen-32b-awq` — same +AutoAWQ toolchain/quant style as the outgoing Qwen2.5-32B-Instruct-AWQ, +widely-used community quant, `Qwen2ForCausalLM` architecture (DeepSeek-R1 +reasoning distilled onto a Qwen2.5-32B base) — no new vLLM code path +required. Native `max_position_embeddings: 131072`; capped at 32768 per +the task's explicit requirement. + +**Executed:** +1. Stopped + disabled `vllm-nomic-embed-text-v1.5.service` (single-model + requirement), freed its ~19GB Qwen2.5-32B model cache on disk (30GB + free → 48GB free) to make room for DeepSeek's ~19.3GB download. +2. Replaced `vllm_models` in `host_vars/astro-orbiter/vars.yml`: primary + entry now `DeepSeek-R1-Distill-Qwen-32B-AWQ`, aux (`Qwen3-8B-AWQ`) and + embedding (`nomic-embed-text-v1.5`) both `enabled: false`. +3. Staged the model via `--tags vllm-models` (idempotent `hf download`, + ~19GB, confirmed via `du -sh` and snapshot-dir stat). +4. **Three rounds of live VRAM-fit debugging** before a stable config was + found (documented inline in host_vars comments) — worth recording here + since the failure mode is non-obvious and will recur for future + 32B-class models at high context on this 24GB card: + - **Round 1 (fp16 KV, gpu_memory_utilization 0.90/0.95/0.98):** vLLM's + own pre-flight check reported 18.17GiB weights + 8.0GiB KV cache + needed at 32768 ctx fp16 = 26.17GB — mathematically impossible on a + 24GB card at ANY utilization percentage. Crash-looped every attempt. + - **Round 2 (`--kv-cache-dtype fp8`):** halved nominal KV cache to + ~4.0-4.3GiB, should fit with ~1GB margin. Still OOM'd — small + (~50-150MB) `cudaMalloc` failures during FlashInfer kernel warmup, + consistently, even when vLLM's own pre-flight math said it should + fit. Root cause: real GPU usage during warmup kernel compilation + exceeds what upfront profiling/reservation accounts for by roughly + ~1GB (unaccounted FlashInfer/sampler warmup workspace buffers). + Tried both the percentage knob AND vLLM's own suggested + `--kv-cache-memory-bytes` exact value — same failure either way, + confirming the gap wasn't a rounding/estimation error in the + percentage math, it was a real missing ~1GB of margin. + - **Round 3 (`--kv-cache-dtype int4_per_token_head`, fixed): SUCCESS.** + Switching from 8-bit to 4-bit KV cache roughly halves the KV + footprint again (~2GiB instead of ~4-4.3GiB), buying back enough + real headroom to absorb the unaccounted warmup overhead. Clean + single-attempt start, `NRestarts=0`, steady-state VRAM 23.2GB/24.576GB. +5. **Full Ansible verify phase (`--tags vllm-api-key,vllm-verify`)** + passed: systemd unit active, `/health` 200, `/v1/models` returns + `DeepSeek-R1-Distill-Qwen-32B-AWQ` with `max_model_len: 32768`, live + `/v1/completions` smoke test HTTP 200, clean restart + re-run of + `--tags vllm-systemd` confirmed idempotent (`changed=0`, + `NRestarts=0`, same `ActiveEnterTimestamp` — no unnecessary restart). +6. **Manual end-to-end generation test**, not inference: a real + `/v1/chat/completions` call ("What is 12*8?") returned a genuine + DeepSeek-R1 reasoning trace in `` tags followed by the correct + answer (96) with correct step-by-step arithmetic shown — confirms the + model is not just health-check-alive but actually reasoning correctly. + +**Role/template changes (reusable for future models on this host):** +- Added `kv_cache_dtype` (renders `--kv-cache-dtype`) and + `kv_cache_memory_bytes` (renders `--kv-cache-memory-bytes`) as new + optional per-model fields in `vllm.service.j2` — both are `{% if + ... is defined %}` guarded, no effect on models that don't set them. + +**Final production state on astro-orbiter (verified live, 2026-09-01):** +- `vllm.service` (DeepSeek-R1-Distill-Qwen-32B-AWQ, :8000, `max_model_len: + 32768`, `kv_cache_dtype: int4_per_token_head`): active, enabled, + boot-persistent, single model on the card +- `vllm-nomic-embed-text-v1.5.service` (:8020): inactive, disabled +- `vllm-Qwen3-8B-AWQ.service` (:8010): inactive, disabled (unchanged from prior state) +- `llama-swap.service`: inactive, disabled (unchanged from prior state) +- VRAM: ~23.2GB/24.576GB steady-state, no crash-looping, `NRestarts=0` + +**Not done in this task (flagging, not implied by this swap):** +- Hindsight's `HINDSIGHT_API_LLM_MODEL` / `HINDSIGHT_API_LLM_BASE_URL` + cluster config still references `Qwen2.5-32B-Instruct-AWQ` — that model + is now gone from the card. Hindsight's LLM calls to astro-orbiter will + fail model-not-found until that GitOps config is updated to point at + `DeepSeek-R1-Distill-Qwen-32B-AWQ`. Not touched here — task scope was + the astro-orbiter model swap itself, cluster consumer cutover is a + separate, explicit follow-up (same boundary respected in the prior + t_5508360a section: this role does not own cluster-side config). +- DeepSeek-R1's reasoning output uses `` tags and the model card + recommends temperature 0.5-0.7 (not greedy/0) — neither is enforced + server-side; any consumer wiring this model into a Hermes profile or + application should account for both when parsing responses. ## Validation Log (2026-08-31, t_ca1af9fb) diff --git a/ansible/roles/deploy-vllm/templates/vllm.service.j2 b/ansible/roles/deploy-vllm/templates/vllm.service.j2 index 4d23bb3..8fdc36e 100644 --- a/ansible/roles/deploy-vllm/templates/vllm.service.j2 +++ b/ansible/roles/deploy-vllm/templates/vllm.service.j2 @@ -46,6 +46,12 @@ ExecStart={{ vllm_venv_path }}/bin/python -m vllm.entrypoints.openai.api_server {% endif %} {% if item.quantization is defined and item.quantization != 'none' %} --quantization {{ item.quantization }} \ +{% endif %} +{% if item.kv_cache_dtype is defined %} + --kv-cache-dtype {{ item.kv_cache_dtype }} \ +{% endif %} +{% if item.kv_cache_memory_bytes is defined %} + --kv-cache-memory-bytes {{ item.kv_cache_memory_bytes }} \ {% endif %} --gpu-memory-utilization {{ item.gpu_memory_utilization }} \ --max-model-len {{ item.max_model_len }} \