# deploy-vllm

Idempotent Ansible role that deploys a vLLM OpenAI-compatible inference
server. Written for astro-orbiter (RTX 3090, 24GB VRAM, 64GB RAM, Ubuntu
24.04) and designed for reuse on the planned Mac Mini M4 host later this
week (see "Portability" below).

Supersedes the manual, pre-role state left behind by earlier vLLM
experiments (`/home/jarvis/vllm-env`, bitsandbytes, gemma-2-27b — see
`homelab-llm-inference`/`homelab-llm-serving` skills for that history). This
role uses a **fresh venv** (`vllm_venv_path`, default `~/vllm-serve-env`) and
**AWQ pre-quantized models** — no bitsandbytes, no on-the-fly quantization,
no repeat of the OOM incident from the earlier Gemma-2-27B attempt.

## Phases

| Phase | File | What it does |
|---|---|---|
| 1 | `tasks/dependencies.yml` | System Python 3.10+, dedicated venv, `pip install vllm>=0.5.0`, verifies `nvidia-smi` and `torch.cuda.is_available()` |
| 2 | `tasks/models.yml` | Downloads each `enabled: true` model in `vllm_models` via `hf download` (huggingface_hub CLI) into `~/.vllm-cache`, verifies the snapshot landed and reports on-disk size |
| 3 | `tasks/api-key.yml` | Reads the API key from 1Password (`op://mk-labs/vllm/api-key`) on the **controller**, writes it to `/etc/vllm/api-key.env` (root:root, 0600) on the target |
| 4 | `tasks/systemd.yml` | Renders and installs one systemd unit per enabled model (`vllm.service` for the `role: primary` model, `vllm-<id>.service` for others) |
| 5 | `tasks/verify.yml` | Only runs when `vllm_service_state=started`. Waits for `/health` (up to 5 min — torch.compile warmup), checks `/v1/models`, runs a live completion, scans `journalctl` for errors |

Run all phases: `ansible-playbook -i inventory.yml playbooks/day1_deploy_vllm.yml --limit astro-orbiter`
Run one phase: `--tags vllm-dependencies` / `vllm-models` / `vllm-api-key` / `vllm-systemd` / `vllm-verify`

## Deliberate staging-first default

`vllm_service_state` defaults to `stopped`. A default run **stages
everything** (venv, model weights, API key file, systemd unit) but does
**not** start the service or touch production traffic. This matches the
astro-orbiter cutover plan: llama-swap is live production serving (Qwen3.8-27B
+ nomic-embed for Hindsight) — vLLM must be deployed and validated on a
side port/inactive unit before anything is cut over.

To start and validate:

```bash
ansible-playbook -i inventory.yml playbooks/day1_deploy_vllm.yml \
  --limit astro-orbiter --extra-vars "vllm_service_state=started"
```

This starts the systemd unit(s), enables them, and runs Phase 5 verification
(health, `/v1/models`, live completion, clean journalctl).

**Cutover of consumers (Hermes profiles, Hindsight embedding config, any
hardcoded `:8001`/`:5805` references) to the new `:8000` vLLM endpoint is a
separate, explicit step outside this role** — do this only after Phase 5
passes cleanly. Do not tear down llama-swap until consumers are confirmed
working end-to-end against vLLM.

## Model roster (`vllm_models` in defaults/main.yml)

vLLM 0.5.x-0.28.x serves **one model per process** — multi-model = multiple
systemd units on distinct ports, not a single multiplexed server (unlike
llama-swap's matrix DSL). Today's phase enables only the primary model;
flip `enabled: true` on the others as VRAM allows (see "Phased Strategy"):

**⚠️ Table below reflects the ORIGINAL Qwen2.5-32B deployment. As of
2026-09-01 (t_r1d32b_swap) the primary model is
`DeepSeek-R1-Distill-Qwen-32B-AWQ`, single-model only (nomic-embed also
disabled) — see the "SUPERSEDED" section further down for current state.**

| id | hf_repo | role | port | quant | enabled |
|---|---|---|---|---|---|
| Qwen2.5-32B-Instruct-AWQ | Qwen/Qwen2.5-32B-Instruct-AWQ | primary | 8000 | awq | **true** |
| Qwen3-8B-AWQ | Qwen/Qwen3-8B-AWQ | aux | 8010 | awq | false |
| nomic-embed-text-v1.5 | nomic-ai/nomic-embed-text-v1.5 | embedding | 8020 | none | false |

**Note on the original spec's model choices:** the task body named
`Qwen/Qwen2.5-32B-Instruct` and `Qwen/Qwen3-8B-Instruct` (bf16, unquantized).
vLLM does not do on-the-fly quantization safely on this host (bitsandbytes
OOM history — see `homelab-llm-inference` skill Pitfalls) and unquantized
bf16 32B does not fit a 24GB card at all (~65GB). This role instead deploys
the **official Qwen AWQ pre-quantized variants**
(`Qwen/Qwen2.5-32B-Instruct-AWQ`, `Qwen/Qwen3-8B-AWQ`), which vLLM natively
supports (`--quantization awq`) and which fit the VRAM budget:

- Qwen2.5-32B-Instruct-AWQ: ~19.3GB on disk, fits with ~5GB headroom at 24GB
- Qwen3-8B-AWQ: ~6GB VRAM per llm-explorer
- nomic-embed-text-v1.5: ~300MB, vLLM serves it via `--convert embed` pooling
  (see vLLM embedding docs) — **not yet wired into this role's systemd
  template**; the embedding model needs `--task embed` / `--convert embed`
  flags that differ from the completion-serving template. Flagged as a
  follow-up before `enabled: true` is flipped on it (see Known Gaps below).

## Known Gaps / Follow-ups

- Quarterly API key rotation is documented (`/etc/vllm/API_KEY_ROTATION.md`
  on the target, rendered by `tasks/api-key.yml`) but not automated — no cron
  job exists to force rotation on a schedule. Consider a follow-up cron task
  if Nick Fury wants this enforced rather than just documented.
- `vllm_service_enabled` defaults to `false` deliberately — see "Deliberate
  staging-first default" above. Flip together with the cutover step, not
  before.
- **vLLM cannot replace llama-swap's full model roster on this card — see
  "Critical architectural finding" section below for the full incident.**
  Short version: vLLM's one-model-per-process design plus llama-swap's own
  VRAM needs exceed this 24GB card's capacity when both must serve real
  models simultaneously. Full llama-swap teardown (t_6dff1ecc) cannot
  proceed until a human decides the aux-model + VRAM strategy.

## Embedding-mode support (t_e6facb19, 2026-08-31)

`vllm.service.j2` now branches on `role: embedding` entries in `vllm_models`:
adds `--runner pooling --convert embed` (vLLM's embedding-serving flags —
see https://docs.vllm.ai/en/latest/models/pooling_models/embed/) and
`--no-enable-prefix-caching` (prefix caching is a completions-only
optimization; irrelevant and safely disabled for pooling). An additional
per-model `trust_remote_code: true` toggle renders `--trust-remote-code`
when set — required for `nomic-ai/nomic-embed-text-v1.5`, which ships
custom `NomicBertModel` modeling code on its HF repo.

**Verification does NOT run `/v1/completions` against embedding-mode
instances** (they don't serve that endpoint — a completions request 400s
immediately). `tasks/verify.yml` splits `vllm_enabled_models` by `role` and
runs the appropriate smoke test per group: completions models get the
`/v1/completions` "capital of France" test; embedding models get a real
`/v1/embeddings` POST with an `ansible.builtin.assert` on a non-empty
`data[0].embedding` array (not just HTTP 200 — an empty/malformed vector
would still 200).

**Critical VRAM finding: co-resident completions + embedding vLLM processes
need MORE headroom than either alone, and CUDA graph capture is the failure
mode, not KV cache sizing.** Enabling `nomic-embed-text-v1.5` alongside the
primary Qwen2.5-32B model at the role-default `gpu_memory_utilization: 0.95`
crash-looped repeatedly:
- First failure: `torch.OutOfMemoryError` during `capture_model()` (CUDA
  graph capture) — KV cache sizing itself succeeded (14,720 tokens
  allocated), but graph capture needed ~20MiB more than the 0.95 budget had
  left once nomic's embedding process (814MiB actual, not the nominal
  ~300MB estimate in the model roster table) claimed its share.
- Fix attempt 1: added a per-model `enforce_eager: true` template branch
  (`--enforce-eager` skips CUDA graph capture entirely) — this stopped the
  graph-capture OOM but the combined processes still landed at only
  ~847MiB genuinely free out of 24,576MiB, and both services crash-looped
  6-7 times during warmup before finally stabilizing (each attempt leaves
  transient VRAM that the next attempt fights over, extending time-to-stable
  well past a single health-check retry window).
- Fix attempt 2 (final, verified stable): lowered the primary model's
  `gpu_memory_utilization` from 0.95 to **0.90** (host_vars override) in
  addition to `enforce_eager: true`. Result: clean single-attempt start for
  both services, `NRestarts=0`, ~2GB genuinely free (22,577MiB used /
  24,576MiB total). Confirmed via `systemctl show <unit> -p NRestarts` after
  a full stop/start cycle — 0.95 was NOT a fluke of Restart=always masking
  the underlying fragility; 0.90 is a real, reproducible fix.
- **Takeaway for future multi-process vLLM VRAM budgeting on this host:**
  do not just check "does it eventually come up" — check `NRestarts` and
  free VRAM headroom after a clean stop/start. A model that "works" after
  6 crash-loop retries is not production-stable; the retries themselves are
  evidence the utilization ceiling is too tight for the actual (not
  nominal) footprint of co-resident processes.

## Consumer cutover status (t_e6facb19, 2026-08-31)

**Attempted, then REVERTED — Hindsight LLM cutover.** Hindsight's
`HINDSIGHT_API_LLM_BASE_URL` was pointed at vLLM `:8000`
(Qwen2.5-32B-Instruct-AWQ) and validated working in isolation: health,
`/v1/chat/completions`, and a live `hindsight_retain` + recall round-trip
all succeeded (after also fixing `HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS`,
which defaulted to 64000 — exceeding vLLM's `max_model_len=8192` — down to
4096). **Reverted anyway**, because of a severe discovery documented in the
next section: vLLM cannot stay resident on this card without starving
llama-swap, and Hindsight's LLM endpoint needs continuous availability, not
just a validation window. Restored to `http://astro-orbiter:8001/v1`
(llama-swap, Qwen3.8-27B-Q4_K_M) — the pre-task working state.

**NOT cut over — embeddings.** Hindsight was discovered to have NEVER used
astro-orbiter for embeddings — it defaults to a bundled local
`BAAI/bge-small-en-v1.5` (384-dim) embedder whenever
`HINDSIGHT_API_EMBEDDINGS_PROVIDER` is unset, which was always the case here.
Pointing it at vLLM's `nomic-embed-text-v1.5` (768-dim) crash-looped the pod:
`RuntimeError: Cannot change embedding dimension from 384 to 768:
memory_units table contains 1289 rows with embeddings.` Re-embedding all
existing memory data across ~20 agent banks is destructive and irreversible
— reverted immediately, left as a separate, explicitly-approved future task.

**NOT cut over — 21 Hermes agent profiles' aux models + OpenViking VLM.**
See "Critical architectural finding" below — this was never attempted once
the VRAM collision was discovered, would have made things categorically
worse.

## Critical architectural finding: vLLM CANNOT be continuously resident alongside llama-swap on this 24GB card (t_e6facb19, 2026-08-31)

After validating vLLM's two processes (Qwen2.5-32B-Instruct-AWQ + nomic-embed-
text-v1.5, ~22.8GB combined) work correctly in isolation, this role's
`vllm_service_enabled`/`vllm_service_state` were flipped to `true`/`started`
as host_vars overrides to make the deployment permanent (per the task's
"enable for boot" requirement) — llama-swap was then restarted alongside
vLLM to preserve its own consumers. **Result: llama-swap could no longer
load ANY of its own generative models.** Every `/v1/chat/completions`
request against `Qwen3.8-27B-Q4_K_M` or the `Qwen3-8B` aux models failed
with `{"error":"unspecific error: upstream command exited prematurely",
"src":"llama-swap"}` — llama-server's own OOM at spawn time, only ~1.8GB
free on a 24GB card once vLLM's ~22.8GB was already claimed.

**Confirmed by direct A/B test, not inference:** identical
`Qwen3.8-27B-Q4_K_M` chat completion request returned HTTP 500 with vLLM's
two processes running, then HTTP 200 with a real completion within seconds
of `systemctl stop vllm.service vllm-nomic-embed-text-v1.5.service` — same
llama-swap process, same request, only the GPU memory pressure changed.

**This is a hard architectural collision, not a tunable-parameter problem.**
llama-swap needs ~18-20GB for its own primary model (Qwen3.8-27B-Q4_K_M);
vLLM's two processes need ~22.8GB even with `enforce_eager` and a lowered
`gpu_memory_utilization`. The two together need more VRAM than a 24GB card
has once both hold real models resident — there is no `gpu_memory_utilization`
value that resolves this while both stacks serve real production models
simultaneously.

**Consequence — reverted the boot-persistence flip.** `vllm_service_enabled`
and `vllm_service_state` are back to role defaults (`false`/`stopped`) in
`host_vars/astro-orbiter/vars.yml`. vLLM stays staged (venv, model weights,
systemd units all in place) and can be started for a brief shadow-validation
window (same pattern as t_ca1af9fb's original Phase 5), but is NOT safe to
leave resident in production alongside llama-swap.

**Path forward — requires a human decision, not more role tuning:**
1. Full llama-swap teardown (t_6dff1ecc) BEFORE vLLM gets permanent
   residency — but that breaks the 21 agent profiles' aux-model tasks and
   OpenViking's VLM unless those consumers are migrated to a different
   backend first (Anthropic API, a second smaller local box, or a
   redesigned single-process serving strategy that covers all the models
   vLLM and llama-swap currently split between them).
2. Accept vLLM as a shadow-only / on-demand stack (manually started for
   specific validated windows, stopped otherwise) and do NOT attempt
   permanent Hindsight cutover — keeps llama-swap as the sole continuous
   production serving layer, matching the pre-task state.
3. A hardware change (larger GPU, or a second GPU) — out of scope for this
   task, flagging for Ryan's awareness if the aux-model consumer set is
   expected to grow.

Comment posted on t_6dff1ecc with this finding — the teardown task remains
correctly blocked; this task's completion does NOT unblock it, because full
cutover to vLLM is not achievable within this card's VRAM budget as
currently scoped.

## RESOLVED (t_5508360a, 2026-08-31/09-01): Dashboard decision applied — llama-swap retired, vLLM permanent, 2 of 3 models

Human decision (dashboard, kanban t_5508360a): **"stop and disable llama-swap
and start vLLM and its 3 models"** — explicit approval, "I understand this is
a breaking change." Chose path 1 from the three options above: retire
llama-swap, give vLLM permanent residency, accept that the 21 Hermes
profiles' aux-model consumers lose their llama-swap aux roster (Qwen3-8B,
Phi-3.5-mini, Meta-Llama-3.1-8B, Qwen2.5-Coder-14B — all 4 gone) in exchange
for vLLM's stack. Mid-run the dashboard added a course-correction: **"Don't
try to load all 3 models concurrently on first deploy. Start with
Qwen2.5-32B only"** — received after the 3-model attempt below had already
run and self-corrected to the same 2-model end state, so no further action
needed, but noted for the record.

**Executed:**
1. `sudo systemctl stop llama-swap && sudo systemctl disable llama-swap` on
   astro-orbiter — confirmed inactive+disabled, VRAM dropped to 9MiB/24576MiB
   (from 20.6GB in production use).
2. Flipped `vllm_service_enabled`/`vllm_service_state` to `true`/`started` in
   `host_vars/astro-orbiter/vars.yml` — vLLM is now the permanent,
   boot-persistent serving layer (was shadow-only/staged before this task).
3. **Attempted the literal "3 models" instruction** — flipped
   `Qwen3-8B-AWQ.enabled` to `true` too. **Does not fit.** With the 24GB
   card's usable 23.55GiB budget consumed by Qwen2.5-32B-Instruct-AWQ
   (~18.6GB weights) + nomic-embed-text-v1.5 (~0.8GB actual), only ~1.25GiB
   remained free — below the 3.53GiB floor `gpu_memory_utilization=0.15`
   requires for Qwen3-8B-AWQ even with `enforce_eager`. Confirmed via
   `journalctl`: identical `ValueError: Free memory on device cuda:0
   (1.25/23.55 GiB) on startup is less than desired GPU memory utilization`
   on all 7 consecutive systemd restart attempts — not the transient
   CUDA-graph-capture crash-loop t_e6facb19 solved with enforce_eager, a hard
   ceiling. Stopped + disabled `vllm-Qwen3-8B-AWQ.service`, reverted
   `enabled: false` in host_vars with a full writeup in the comment block.
4. Re-ran `day1_deploy_vllm.yml --extra-vars vllm_service_state=started`
   with the corrected 2-model config: **clean idempotent pass, changed=0** on
   both remaining models, Phase 5 verification passed (`/health` 200 on both
   `:8000` and `:8020`, `/v1/models` correct, live completion + live
   embeddings smoke tests both passed), `NRestarts=0` on both services.
5. **Cut over Hindsight's LLM endpoint** (the other production consumer):
   `HINDSIGHT_API_LLM_BASE_URL` llama-swap `:8001` → vLLM `:8000`,
   `HINDSIGHT_API_LLM_MODEL` → `Qwen2.5-32B-Instruct-AWQ`, added
   `HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS=4096` (vLLM's
   `max_model_len=8192` vs Hindsight's 64000 default), and switched the
   ExternalSecret's `HINDSIGHT_API_LLM_API_KEY` source from the unused
   `nous` 1Password item to `vllm`'s real `api-key` (vLLM validates its
   bearer token; llama-swap never did). Committed to
   `cluster/applications/hindsight/{values.yaml,externalsecret.yaml}`,
   pushed, ArgoCD synced, confirmed the new pod logged `Connection verified:
   openai/Qwen2.5-32B-Instruct-AWQ` on boot.
6. **Live end-to-end verification**, not inference: a real
   `POST /v1/default/banks/war-machine/memories` retain call against the
   production Hindsight endpoint returned `HTTP 200` with genuine
   fact-extraction token usage (3257 in / 245 out), and a subsequent
   `POST .../memories/recall` returned real semantically-ranked results
   including the just-retained memory.

**Final production state on astro-orbiter (verified live):**
- `vllm.service` (Qwen2.5-32B-Instruct-AWQ, :8000): active, enabled, boot-persistent
- `vllm-nomic-embed-text-v1.5.service` (:8020): active, enabled, boot-persistent
- `vllm-Qwen3-8B-AWQ.service` (:8010): inactive, disabled — does not fit, see above
- `llama-swap.service`: inactive, disabled (unit files left in place —
  full removal is t_6dff1ecc's job, tracked separately)
- VRAM: ~22.6GB/24.576GB in steady-state use, no crash-looping

**What this means for t_6dff1ecc (teardown) and the 21 aux-model profiles:**
llama-swap is now stopped+disabled — t_6dff1ecc's actual teardown steps
(remove systemd unit files, wipe caches) are now safe to execute and
unblocked from a "live production" standpoint. However, this trades away
the aux-model roster: the 21 Hermes profiles' aux-model tasks (skills_hub,
approval, mcp, title_generation, profile_describer, compression) that
used to route to llama-swap's Qwen3-8B/Phi-3.5-mini/Meta-Llama/Coder
models now have **zero local aux-model backend** — Qwen3-8B-AWQ doesn't
fit vLLM's VRAM budget either. This was accepted explicitly by the
dashboard ("I understand this is a breaking change") — no further local
aux-model migration was authorized or attempted in this task. If those 21
profiles need a replacement aux-model path, that is separate, new,
explicitly-scoped follow-up work, not implied by this decision.

## SUPERSEDED (t_r1d32b_swap, 2026-09-01): Qwen2.5-32B-Instruct-AWQ retired, replaced with DeepSeek-R1-Distill-Qwen-32B-AWQ, single-model deployment

Ryan direction: "Swap Qwen2.5-32B for DeepSeek-R1-Distill-Qwen-32B,
max-model-len 32768. Single model only." Confirmed with Ryan that "single
model only" includes disabling `nomic-embed-text-v1.5` (:8020) as well —
nothing in production consumed it (Hindsight uses its own bundled 384-dim
embedder; OpenViking pointed at the retired llama-swap endpoint). DeepSeek
gets the entire 24GB card.

**Model choice:** `casperhansen/deepseek-r1-distill-qwen-32b-awq` — same
AutoAWQ toolchain/quant style as the outgoing Qwen2.5-32B-Instruct-AWQ,
widely-used community quant, `Qwen2ForCausalLM` architecture (DeepSeek-R1
reasoning distilled onto a Qwen2.5-32B base) — no new vLLM code path
required. Native `max_position_embeddings: 131072`; capped at 32768 per
the task's explicit requirement.

**Executed:**
1. Stopped + disabled `vllm-nomic-embed-text-v1.5.service` (single-model
   requirement), freed its ~19GB Qwen2.5-32B model cache on disk (30GB
   free → 48GB free) to make room for DeepSeek's ~19.3GB download.
2. Replaced `vllm_models` in `host_vars/astro-orbiter/vars.yml`: primary
   entry now `DeepSeek-R1-Distill-Qwen-32B-AWQ`, aux (`Qwen3-8B-AWQ`) and
   embedding (`nomic-embed-text-v1.5`) both `enabled: false`.
3. Staged the model via `--tags vllm-models` (idempotent `hf download`,
   ~19GB, confirmed via `du -sh` and snapshot-dir stat).
4. **Three rounds of live VRAM-fit debugging** before a stable config was
   found (documented inline in host_vars comments) — worth recording here
   since the failure mode is non-obvious and will recur for future
   32B-class models at high context on this 24GB card:
   - **Round 1 (fp16 KV, gpu_memory_utilization 0.90/0.95/0.98):** vLLM's
     own pre-flight check reported 18.17GiB weights + 8.0GiB KV cache
     needed at 32768 ctx fp16 = 26.17GB — mathematically impossible on a
     24GB card at ANY utilization percentage. Crash-looped every attempt.
   - **Round 2 (`--kv-cache-dtype fp8`):** halved nominal KV cache to
     ~4.0-4.3GiB, should fit with ~1GB margin. Still OOM'd — small
     (~50-150MB) `cudaMalloc` failures during FlashInfer kernel warmup,
     consistently, even when vLLM's own pre-flight math said it should
     fit. Root cause: real GPU usage during warmup kernel compilation
     exceeds what upfront profiling/reservation accounts for by roughly
     ~1GB (unaccounted FlashInfer/sampler warmup workspace buffers).
     Tried both the percentage knob AND vLLM's own suggested
     `--kv-cache-memory-bytes` exact value — same failure either way,
     confirming the gap wasn't a rounding/estimation error in the
     percentage math, it was a real missing ~1GB of margin.
   - **Round 3 (`--kv-cache-dtype int4_per_token_head`, fixed): SUCCESS.**
     Switching from 8-bit to 4-bit KV cache roughly halves the KV
     footprint again (~2GiB instead of ~4-4.3GiB), buying back enough
     real headroom to absorb the unaccounted warmup overhead. Clean
     single-attempt start, `NRestarts=0`, steady-state VRAM 23.2GB/24.576GB.
5. **Full Ansible verify phase (`--tags vllm-api-key,vllm-verify`)**
   passed: systemd unit active, `/health` 200, `/v1/models` returns
   `DeepSeek-R1-Distill-Qwen-32B-AWQ` with `max_model_len: 32768`, live
   `/v1/completions` smoke test HTTP 200, clean restart + re-run of
   `--tags vllm-systemd` confirmed idempotent (`changed=0`,
   `NRestarts=0`, same `ActiveEnterTimestamp` — no unnecessary restart).
6. **Manual end-to-end generation test**, not inference: a real
   `/v1/chat/completions` call ("What is 12*8?") returned a genuine
   DeepSeek-R1 reasoning trace in `<think>` tags followed by the correct
   answer (96) with correct step-by-step arithmetic shown — confirms the
   model is not just health-check-alive but actually reasoning correctly.

**Role/template changes (reusable for future models on this host):**
- Added `kv_cache_dtype` (renders `--kv-cache-dtype`) and
  `kv_cache_memory_bytes` (renders `--kv-cache-memory-bytes`) as new
  optional per-model fields in `vllm.service.j2` — both are `{% if
  ... is defined %}` guarded, no effect on models that don't set them.

**Final production state on astro-orbiter (verified live, 2026-09-01):**
- `vllm.service` (DeepSeek-R1-Distill-Qwen-32B-AWQ, :8000, `max_model_len:
  32768`, `kv_cache_dtype: int4_per_token_head`): active, enabled,
  boot-persistent, single model on the card
- `vllm-nomic-embed-text-v1.5.service` (:8020): inactive, disabled
- `vllm-Qwen3-8B-AWQ.service` (:8010): inactive, disabled (unchanged from prior state)
- `llama-swap.service`: inactive, disabled (unchanged from prior state)
- VRAM: ~23.2GB/24.576GB steady-state, no crash-looping, `NRestarts=0`

**Not done in this task (flagging, not implied by this swap):**
- Hindsight's `HINDSIGHT_API_LLM_MODEL` / `HINDSIGHT_API_LLM_BASE_URL`
  cluster config still references `Qwen2.5-32B-Instruct-AWQ` — that model
  is now gone from the card. Hindsight's LLM calls to astro-orbiter will
  fail model-not-found until that GitOps config is updated to point at
  `DeepSeek-R1-Distill-Qwen-32B-AWQ`. Not touched here — task scope was
  the astro-orbiter model swap itself, cluster consumer cutover is a
  separate, explicit follow-up (same boundary respected in the prior
  t_5508360a section: this role does not own cluster-side config).
- DeepSeek-R1's reasoning output uses `<think>` tags and the model card
  recommends temperature 0.5-0.7 (not greedy/0) — neither is enforced
  server-side; any consumer wiring this model into a Hermes profile or
  application should account for both when parsing responses.

## Validation Log (2026-08-31, t_ca1af9fb)

Full Phase 1-5 run executed against astro-orbiter in a brief shadow-validation
window (llama-swap stopped ~5 min, per the `homelab-llm-inference` skill's
documented shadow-validation pattern — production traffic could not be
tested concurrently with vLLM's VRAM footprint on this 24GB card).

**Two real bugs found and fixed during first-start validation** (not present
in the original spec, discovered only by actually starting the service):

1. **`ninja` not on systemd's PATH.** vLLM's torch.compile path shells out to
   the bare `ninja` command. `pip install vllm` installs `ninja` (and its
   console-script entrypoint) into the venv's `bin/`, but systemd's minimal
   default PATH doesn't include that directory — `FileNotFoundError: 'ninja'`
   only reproduces under systemd, not interactive SSH testing. Fixed by
   setting `Environment="PATH=<venv>/bin:...standard dirs..."` in the unit
   template.
2. **FlashInfer sampler JIT fails to compile on RTX 3090 (SM86).**
   `flashinfer/data/csrc/sampling.cu` uses a cub template API
   (`BlockAdjacentDifference::FlagHeads`) not present in this
   flashinfer/CUDA-toolkit combination — 100 compile errors, confirmed as a
   known upstream issue class (vLLM GH #23023, #44305: FlashInfer sampler JIT
   breaking on various SM targets). Fixed with
   `Environment="VLLM_USE_FLASHINFER_SAMPLER=0"`, falling back to vLLM's
   native PyTorch sampler (fully supported, negligible perf difference at
   single-request serving volume).

Also corrected `vllm_gpu_memory_utilization` from 0.90 to 0.95 — at 0.90 the
KV cache allocation failed (`2.0 GiB KV cache needed, 1.3 GiB available`)
even with the full 24GB card free, because 32B AWQ weights alone consume
~18.4GB, leaving too little headroom at a 90% cap.

**Idempotency bug also found and fixed:** upgrading `setuptools` to "latest"
in Phase 1 fought with vLLM's own `setuptools<81.0.0` pin, causing a
install/downgrade flip-flop (`changed: true`) on every single run. Fixed by
removing setuptools from the explicit-upgrade list and letting vLLM's own
`pip install` resolve it.

**Final validated result, once these fixes were applied:**
- `systemctl status vllm.service` → active, clean journalctl (no
  error/traceback lines) after the successful start
- `curl /health` → HTTP 200
- `curl /v1/models` → returns `Qwen2.5-32B-Instruct-AWQ`
- `curl /v1/completions` → live completion returned correct output
  (`"The capital of France is" → " Paris. Correct! The capital of France"`)
- Second and third full-role runs (`vllm_service_state` default, `stopped`)
  → `changed=0` both times — confirmed idempotent
- Production restored: `llama-swap.service` active, `/health` 200,
  `/v1/embeddings` against `nomic-embed-text-v1.5` returns a valid vector —
  Hindsight retain path confirmed still working after the shadow window
- Post-restore VRAM: 486 MiB used / 24,576 MiB total (normal quiescent state)

## Testing this role (idempotency)

Second-run test (staging phases only, safe to run repeatedly):

```bash
ansible-playbook -i inventory.yml playbooks/day1_deploy_vllm.yml \
  --limit astro-orbiter --tags vllm-dependencies,vllm-models,vllm-api-key,vllm-systemd
# Run it again immediately — expect changed=0 (or only handler-driven
# restarts if vllm_service_state=started and the API key file rotated)
```

Confirmed 2026-08-31 (t_ca1af9fb): Phase 1 (dependencies) ran once with
changed=3 (venv create, pip upgrade, vllm install); a second run reported
changed=0 for those three tasks — venv `creates:` guard and pip module's
own idempotency both held.

## Portability — Mac Mini M4 (planned, end of week)

This role's host-specific assumptions live in `defaults/main.yml` (all
overridable via `host_vars/<host>/vars.yml`) plus one hard assumption baked
into `tasks/dependencies.yml`: an NVIDIA GPU (`nvidia-smi` check, CUDA
wheels). Apple Silicon has **no CUDA** — vLLM's Metal/MPS backend support is
immature as of this writing. Before reusing this role for the Mac Mini M4:

1. Fork `tasks/dependencies.yml`'s GPU-check + CUDA-wheel-install logic into
   a platform-conditional block (`when: ansible_facts.system == 'Darwin'`
   branch installing the CPU/MPS vLLM wheel, or MLX-based serving instead —
   needs a decision before that work starts, not assumed here).
2. `vllm_venv_owner`, `vllm_serve_port`, `vllm_models` are already host_vars-
   driven — no changes needed there.
3. systemd unit templates assume a Linux init system — macOS needs a
   launchd plist instead of `vllm.service.j2`.

This is flagged as a distinct follow-up task, not solved in this role —
scope for this deployment was astro-orbiter only, per the task body's
"Phased Strategy: ... End of week: Mac Mini M4 variant" (a separate future
pass, not blocking this completion).

## Files

```
roles/deploy-vllm/
├── defaults/main.yml               # all tunables — host overrides go in host_vars
├── handlers/main.yml                # reload systemd / restart vllm services
├── meta/main.yml
├── tasks/
│   ├── main.yml                     # phase orchestrator
│   ├── dependencies.yml             # Phase 1
│   ├── models.yml                   # Phase 2
│   ├── api-key.yml                  # Phase 3
│   ├── systemd.yml                  # Phase 4
│   └── verify.yml                   # Phase 5
├── templates/
│   ├── vllm.service.j2              # one instance per enabled model
│   └── vllm-workspace.sh.j2         # debugging helper deployed to the target
└── README.md                        # this file
```