feat(deploy-vllm): swap Qwen2.5-32B for DeepSeek-R1-Distill-Qwen-32B-AWQ (t_r1d32b_swap)

Single-model deployment per Ryan's direction:
- Primary model: casperhansen/deepseek-r1-distill-qwen-32b-awq, max_model_len=32768
- nomic-embed-text-v1.5 and Qwen3-8B-AWQ both disabled (single-model requirement)
- kv_cache_dtype: int4_per_token_head required to fit 32768 ctx on 24GB RTX 3090
  (fp16 KV: 26GB needed, doesn't fit at any utilization; fp8 KV: OOM'd during
  FlashInfer warmup with ~50-150MB margin; int4 KV: clean single-attempt start)
- Added kv_cache_dtype / kv_cache_memory_bytes as new optional per-model
  template fields in vllm.service.j2 (guarded, no effect on other models)

Verified live: /health 200, /v1/models confirms max_model_len=32768,
live /v1/completions smoke test + manual chat completion both passed
(genuine <think> reasoning trace, correct arithmetic). NRestarts=0,
steady-state VRAM 23.2GB/24.576GB. Ansible idempotent re-run confirmed
changed=0.

Known follow-up (not done here): Hindsight's HINDSIGHT_API_LLM_MODEL
cluster config still references the retired Qwen2.5-32B-Instruct-AWQ —
needs separate GitOps update to point at the new model.

ansible/host_vars/astro-orbiter/vars.yml

@@ -109,40 +109,83 @@ llm_staged_models:
     size_bytes: 5027784224
     source_repo: "bartowski/Qwen_Qwen3-8B-GGUF"
 
-# --- deploy-vllm role: vllm_models override (t_e6facb19, 2026-08-31) --------
+# --- deploy-vllm role: vllm_models override (t_r1d32b_swap, 2026-09-01) -----
 # Ansible's hash_behaviour is "replace" (see ansible.cfg) — a host_vars list
 # variable REPLACES the role default list wholesale, it does not deep-merge.
-# This is therefore a full copy of roles/deploy-vllm/defaults/main.yml's
+#
-# vllm_models with ONE change: nomic-embed-text-v1.5.enabled flipped to true,
+# SWAP (Ryan direction, 2026-09-01): Qwen2.5-32B-Instruct-AWQ retired,
-# now that vllm.service.j2 has an embedding-mode branch (--runner pooling
+# replaced with DeepSeek-R1-Distill-Qwen-32B-AWQ, max_model_len=32768.
-# --convert embed --trust-remote-code) tested end-to-end in a shadow window.
+# "Single model only" — nomic-embed-text-v1.5 (embedding, :8020) and
-# Primary (Qwen2.5-32B-Instruct-AWQ) and aux (Qwen3-8B-AWQ) entries are
+# Qwen3-8B-AWQ (aux, :8010, already disabled) are BOTH disabled here.
-# unchanged from role defaults — reproduced here only because the whole list
+# DeepSeek gets the full 24GB card to itself. Nothing in production
-# must be redefined together. Keep this in sync with defaults/main.yml if the
+# consumed nomic-embed at the time of this swap (Hindsight uses its own
-# role's model roster changes.
+# bundled 384-dim embedder; OpenViking pointed at the old llama-swap
+# endpoint, already stopped) — confirmed with Ryan before disabling.
+#
+# Model choice: casperhansen/deepseek-r1-distill-qwen-32b-awq — same
+# quantizer/toolchain (AutoAWQ) as the outgoing Qwen2.5-32B-Instruct-AWQ,
+# widely used, 4-bit GEMM AWQ, ~19.3GB on disk (4 safetensors shards).
+# Architecture: Qwen2ForCausalLM (DeepSeek-R1 distilled onto Qwen2.5-32B
+# base) — same vLLM code path as the outgoing model, no new serving
+# support needed. Native max_position_embeddings=131072; we cap at 32768
+# per the task's explicit max-model-len requirement.
+#
+# VRAM math: ~19.3GB weights (4-bit AWQ) + KV cache at 32768 ctx (GQA,
+# 8 KV heads, 128 head_dim, 64 layers, fp16 KV by default) ≈ 19.3GB +
+# ~4GB KV+overhead ≈ 23.3GB — tight but the FULL 24GB card is now
+# available (no co-resident nomic-embed/Qwen3-8B taking a share, unlike
+# the outgoing Qwen2.5-32B config). gpu_memory_utilization=0.95 (role
+# default) + enforce_eager retained as the proven-stable mitigation from
+# t_e6facb19/t_ca1af9fb (avoids CUDA graph capture VRAM spike; this host's
+# only validated way to avoid crash-loop-to-stabilize behavior on this
+# card). If 0.95 OOMs at 32768 ctx once tested live, drop to 0.90 next
+# (documented fallback, same pattern as the outgoing model).
+#
+# DeepSeek-R1 output note: reasoning traces stream in <think> tags before
+# the final answer — this is expected R1-distill behavior, not a bug.
+# Model card recommends temperature 0.5-0.7 (not 0, not vLLM's greedy
+# default) to avoid repetition/incoherence; not set here (server-side
+# default), left to be set client-side per the model card's guidance —
+# flagging for whoever wires this into Hermes profile configs next.
 vllm_models:
-  - id: "Qwen2.5-32B-Instruct-AWQ"
+  - id: "DeepSeek-R1-Distill-Qwen-32B-AWQ"
-    hf_repo: "Qwen/Qwen2.5-32B-Instruct-AWQ"
+    hf_repo: "casperhansen/deepseek-r1-distill-qwen-32b-awq"
     role: primary
     quantization: awq
     port: 8000
-    max_model_len: 8192
+    max_model_len: 32768
-    # 0.95 (role default) OOM'd during CUDA graph capture once nomic-embed
+    # VRAM correction (2026-09-01, live test): weights alone load at 18.17
-    # (role: embedding, ~814MiB actual, not the nominal 300MB) is co-resident
+    # GiB (confirmed via journalctl "Model loading took 18.17 GiB memory").
-    # on the same 24GB card (t_e6facb19, 2026-08-31): KV cache allocation
+    # fp16 KV cache at 32768 ctx needs 8.0 GiB per vLLM's own error message
-    # succeeded (14,720 tokens) but graph capture needed ~20MiB more than the
+    # ("To serve at least one request with the model's max seq len (32768),
-    # 0.95 budget left after nomic's share. Two independent, permanent
+    # 8.0 GiB KV cache is needed") — 18.17 + 8.0 = 26.17GB, DOES NOT FIT a
-    # co-residents (unlike t_ca1af9fb's shadow-window test, which had the
+    # 24GB card even at gpu_memory_utilization=1.0. Confirmed via 3 live
-    # whole 24GB free) need either a lower utilization ceiling or no graph
+    # crash-loop attempts at 0.95 (usable ~23.35GiB budget): vLLM's own
-    # capture. enforce_eager avoids the whole cudagraph capture memory spike
+    # ValueError reported "estimated maximum model length is 14528" at
-    # entirely — small throughput cost, no OOM risk, safer for a fixed
+    # whatever KV budget was actually available — nowhere close to 32768.
-    # multi-process VRAM budget than tuning utilization percentages by hand.
+    # FIX: --kv-cache-dtype fp8 halves KV cache memory (~4.0GiB instead of
-    # Even WITH enforce_eager, 0.95 left only ~847MiB genuinely free out of
+    # 8.0GiB) — 18.17 + 4.0 = ~22.2GB, fits with ~1.4GB headroom at 0.95.
-    # 24576MiB total (23,729MiB used) and both services crash-looped 6-7x
+    # fp8 KV cache is a standard vLLM feature (not experimental for this
-    # during warmup/KV-cache sizing before stabilizing — too fragile for a
+    # vLLM version), minor precision loss in attention scores, no known
-    # permanent two-process co-residency. Lowered to 0.90 for real headroom
+    # material quality impact for a distilled reasoning model at this scale.
-    # (~1.6GiB free), confirmed clean single-attempt start with no retries.
+    # VRAM correction round 3 (2026-09-01, live test): manual
-    gpu_memory_utilization: 0.90
+    # kv_cache_memory_bytes did NOT fix it either — same OOM pattern
+    # (small ~150MB alloc failing with only ~52MB actually free), meaning
+    # real GPU usage at warmup exceeds what profiling/reservation account
+    # for by roughly ~1GB (unaccounted FlashInfer warmup workspace buffers,
+    # not sized during profiling or reservation). Rather than keep guessing
+    # at a KV byte budget with ~0 margin, switched KV dtype from fp8 (8-bit)
+    # to int4_per_token_head (4-bit) — roughly HALVES KV cache footprint
+    # for the same 32768 ctx (~2.0GiB instead of ~4.0-4.3GiB), buying back
+    # ~2GiB of real headroom to absorb whatever the unaccounted warmup
+    # overhead actually is. Reverted to standard percentage-based
+    # gpu_memory_utilization (no kv_cache_memory_bytes override) so vLLM's
+    # own profiling determines the exact KV budget again, now with much
+    # more slack in play. Some precision loss vs fp8 is expected for KV
+    # cache in int4 — acceptable tradeoff for a homelab box; revisit if
+    # DeepSeek's output quality visibly degrades once running.
+    kv_cache_dtype: int4_per_token_head
+    gpu_memory_utilization: 0.95
     enforce_eager: true
     enabled: true
   - id: "Qwen3-8B-AWQ"
@@ -153,20 +196,7 @@ vllm_models:
     max_model_len: 32768
     gpu_memory_utilization: 0.15
     enforce_eager: true
-    # ATTEMPTED enabled: true (t_5508360a, 2026-08-31) per dashboard's "start
+    enabled: false   # single-model deployment — see swap note above
-    # vLLM and its 3 models" instruction -- does NOT fit. With the primary
-    # Qwen2.5-32B-Instruct-AWQ (~18.6GB weights) + nomic-embed (~0.8GB)
-    # resident, only ~1.25GiB free remained on the 23.55GiB usable budget --
-    # short of the 3.53GiB gpu_memory_utilization=0.15 floor requires. This
-    # is not the CUDA-graph-capture transient crash-loop t_e6facb19 saw and
-    # solved with enforce_eager+lower utilization -- confirmed via
-    # `journalctl`: "ValueError: Free memory on device cuda:0 (1.25/23.55
-    # GiB) on startup is less than desired GPU memory utilization" on EVERY
-    # one of 7 consecutive restart attempts, no stabilization. Reverted to
-    # false. The 24GB card fits Qwen2.5-32B-Instruct-AWQ + nomic-embed
-    # (22.6GB) with no room left for a third resident process -- 2 of the
-    # "3 models" fit, not 3. Flagged in kanban t_5508360a completion.
-    enabled: false
   - id: "nomic-embed-text-v1.5"
     hf_repo: "nomic-ai/nomic-embed-text-v1.5"
     role: embedding
@@ -175,25 +205,11 @@ vllm_models:
     max_model_len: 2048
     gpu_memory_utilization: 0.05
     trust_remote_code: true
-    enabled: true
+    enabled: false   # single-model deployment — see swap note above
 
-# --- deploy-vllm role: boot persistence ENABLED (t_5508360a, 2026-08-31) -----
+# --- deploy-vllm role: boot persistence (unchanged) -------------------------
-# Human decision (dashboard, t_5508360a comment thread): "stop and disable
+# Still permanent/boot-persistent — same policy as the outgoing Qwen2.5-32B
-# llama-swap and start vLLM and its 3 models" / "I understand this is a
+# deployment (t_5508360a), just now serving one model instead of two.
-# breaking change" -- explicit authorization to give vLLM permanent
-# residency and retire llama-swap, accepting that the 21 Hermes profiles'
-# aux-model tasks that used to have 4 llama-swap aux models to choose from
-# (Qwen3-8B, Phi-3.5-mini, Meta-Llama-3.1-8B, Qwen2.5-Coder-14B) now have
-# exactly ONE local aux model (Qwen3-8B-AWQ via vLLM :8010) -- see
-# README.md "Critical architectural finding" section for the full VRAM
-# collision history this decision resolves.
-# Qwen3-8B-AWQ flipped enabled: true + enforce_eager: true (same crash-loop
-# mitigation applied to the primary model in t_e6facb19 -- avoids CUDA graph
-# capture's VRAM spike now that 3 processes must share the budget instead of
-# 2). vllm_service_enabled/state flipped to true/started -- this is now the
-# permanent, boot-persistent serving layer. llama-swap is being stopped +
-# disabled (not removed yet -- full unit/file teardown tracked separately in
-# t_6dff1ecc once this is verified stable).
 vllm_service_enabled: true
 vllm_service_state: started
 

ansible/roles/deploy-vllm/README.md

@@ -57,6 +57,11 @@ systemd units on distinct ports, not a single multiplexed server (unlike
 llama-swap's matrix DSL). Today's phase enables only the primary model;
 flip `enabled: true` on the others as VRAM allows (see "Phased Strategy"):
 
+**⚠️ Table below reflects the ORIGINAL Qwen2.5-32B deployment. As of
+2026-09-01 (t_r1d32b_swap) the primary model is
+`DeepSeek-R1-Distill-Qwen-32B-AWQ`, single-model only (nomic-embed also
+disabled) — see the "SUPERSEDED" section further down for current state.**
+
 | id | hf_repo | role | port | quant | enabled |
 |---|---|---|---|---|---|
 | Qwen2.5-32B-Instruct-AWQ | Qwen/Qwen2.5-32B-Instruct-AWQ | primary | 8000 | awq | **true** |
@@ -309,6 +314,95 @@ aux-model migration was authorized or attempted in this task. If those 21
 profiles need a replacement aux-model path, that is separate, new,
 explicitly-scoped follow-up work, not implied by this decision.
 
+## SUPERSEDED (t_r1d32b_swap, 2026-09-01): Qwen2.5-32B-Instruct-AWQ retired, replaced with DeepSeek-R1-Distill-Qwen-32B-AWQ, single-model deployment
+
+Ryan direction: "Swap Qwen2.5-32B for DeepSeek-R1-Distill-Qwen-32B,
+max-model-len 32768. Single model only." Confirmed with Ryan that "single
+model only" includes disabling `nomic-embed-text-v1.5` (:8020) as well —
+nothing in production consumed it (Hindsight uses its own bundled 384-dim
+embedder; OpenViking pointed at the retired llama-swap endpoint). DeepSeek
+gets the entire 24GB card.
+
+**Model choice:** `casperhansen/deepseek-r1-distill-qwen-32b-awq` — same
+AutoAWQ toolchain/quant style as the outgoing Qwen2.5-32B-Instruct-AWQ,
+widely-used community quant, `Qwen2ForCausalLM` architecture (DeepSeek-R1
+reasoning distilled onto a Qwen2.5-32B base) — no new vLLM code path
+required. Native `max_position_embeddings: 131072`; capped at 32768 per
+the task's explicit requirement.
+
+**Executed:**
+1. Stopped + disabled `vllm-nomic-embed-text-v1.5.service` (single-model
+   requirement), freed its ~19GB Qwen2.5-32B model cache on disk (30GB
+   free → 48GB free) to make room for DeepSeek's ~19.3GB download.
+2. Replaced `vllm_models` in `host_vars/astro-orbiter/vars.yml`: primary
+   entry now `DeepSeek-R1-Distill-Qwen-32B-AWQ`, aux (`Qwen3-8B-AWQ`) and
+   embedding (`nomic-embed-text-v1.5`) both `enabled: false`.
+3. Staged the model via `--tags vllm-models` (idempotent `hf download`,
+   ~19GB, confirmed via `du -sh` and snapshot-dir stat).
+4. **Three rounds of live VRAM-fit debugging** before a stable config was
+   found (documented inline in host_vars comments) — worth recording here
+   since the failure mode is non-obvious and will recur for future
+   32B-class models at high context on this 24GB card:
+   - **Round 1 (fp16 KV, gpu_memory_utilization 0.90/0.95/0.98):** vLLM's
+     own pre-flight check reported 18.17GiB weights + 8.0GiB KV cache
+     needed at 32768 ctx fp16 = 26.17GB — mathematically impossible on a
+     24GB card at ANY utilization percentage. Crash-looped every attempt.
+   - **Round 2 (`--kv-cache-dtype fp8`):** halved nominal KV cache to
+     ~4.0-4.3GiB, should fit with ~1GB margin. Still OOM'd — small
+     (~50-150MB) `cudaMalloc` failures during FlashInfer kernel warmup,
+     consistently, even when vLLM's own pre-flight math said it should
+     fit. Root cause: real GPU usage during warmup kernel compilation
+     exceeds what upfront profiling/reservation accounts for by roughly
+     ~1GB (unaccounted FlashInfer/sampler warmup workspace buffers).
+     Tried both the percentage knob AND vLLM's own suggested
+     `--kv-cache-memory-bytes` exact value — same failure either way,
+     confirming the gap wasn't a rounding/estimation error in the
+     percentage math, it was a real missing ~1GB of margin.
+   - **Round 3 (`--kv-cache-dtype int4_per_token_head`, fixed): SUCCESS.**
+     Switching from 8-bit to 4-bit KV cache roughly halves the KV
+     footprint again (~2GiB instead of ~4-4.3GiB), buying back enough
+     real headroom to absorb the unaccounted warmup overhead. Clean
+     single-attempt start, `NRestarts=0`, steady-state VRAM 23.2GB/24.576GB.
+5. **Full Ansible verify phase (`--tags vllm-api-key,vllm-verify`)**
+   passed: systemd unit active, `/health` 200, `/v1/models` returns
+   `DeepSeek-R1-Distill-Qwen-32B-AWQ` with `max_model_len: 32768`, live
+   `/v1/completions` smoke test HTTP 200, clean restart + re-run of
+   `--tags vllm-systemd` confirmed idempotent (`changed=0`,
+   `NRestarts=0`, same `ActiveEnterTimestamp` — no unnecessary restart).
+6. **Manual end-to-end generation test**, not inference: a real
+   `/v1/chat/completions` call ("What is 12*8?") returned a genuine
+   DeepSeek-R1 reasoning trace in `<think>` tags followed by the correct
+   answer (96) with correct step-by-step arithmetic shown — confirms the
+   model is not just health-check-alive but actually reasoning correctly.
+
+**Role/template changes (reusable for future models on this host):**
+- Added `kv_cache_dtype` (renders `--kv-cache-dtype`) and
+  `kv_cache_memory_bytes` (renders `--kv-cache-memory-bytes`) as new
+  optional per-model fields in `vllm.service.j2` — both are `{% if
+  ... is defined %}` guarded, no effect on models that don't set them.
+
+**Final production state on astro-orbiter (verified live, 2026-09-01):**
+- `vllm.service` (DeepSeek-R1-Distill-Qwen-32B-AWQ, :8000, `max_model_len:
+  32768`, `kv_cache_dtype: int4_per_token_head`): active, enabled,
+  boot-persistent, single model on the card
+- `vllm-nomic-embed-text-v1.5.service` (:8020): inactive, disabled
+- `vllm-Qwen3-8B-AWQ.service` (:8010): inactive, disabled (unchanged from prior state)
+- `llama-swap.service`: inactive, disabled (unchanged from prior state)
+- VRAM: ~23.2GB/24.576GB steady-state, no crash-looping, `NRestarts=0`
+
+**Not done in this task (flagging, not implied by this swap):**
+- Hindsight's `HINDSIGHT_API_LLM_MODEL` / `HINDSIGHT_API_LLM_BASE_URL`
+  cluster config still references `Qwen2.5-32B-Instruct-AWQ` — that model
+  is now gone from the card. Hindsight's LLM calls to astro-orbiter will
+  fail model-not-found until that GitOps config is updated to point at
+  `DeepSeek-R1-Distill-Qwen-32B-AWQ`. Not touched here — task scope was
+  the astro-orbiter model swap itself, cluster consumer cutover is a
+  separate, explicit follow-up (same boundary respected in the prior
+  t_5508360a section: this role does not own cluster-side config).
+- DeepSeek-R1's reasoning output uses `<think>` tags and the model card
+  recommends temperature 0.5-0.7 (not greedy/0) — neither is enforced
+  server-side; any consumer wiring this model into a Hermes profile or
+  application should account for both when parsing responses.
 
 ## Validation Log (2026-08-31, t_ca1af9fb)
 

ansible/roles/deploy-vllm/templates/vllm.service.j2

@@ -46,6 +46,12 @@ ExecStart={{ vllm_venv_path }}/bin/python -m vllm.entrypoints.openai.api_server
 {% endif %}
 {% if item.quantization is defined and item.quantization != 'none' %}
     --quantization {{ item.quantization }} \
+{% endif %}
+{% if item.kv_cache_dtype is defined %}
+    --kv-cache-dtype {{ item.kv_cache_dtype }} \
+{% endif %}
+{% if item.kv_cache_memory_bytes is defined %}
+    --kv-cache-memory-bytes {{ item.kv_cache_memory_bytes }} \
 {% endif %}
     --gpu-memory-utilization {{ item.gpu_memory_utilization }} \
     --max-model-len {{ item.max_model_len }} \