homelab/ansible/roles/deploy-vllm/README.md

deploy-vllm

Idempotent Ansible role that deploys a vLLM OpenAI-compatible inference server. Written for astro-orbiter (RTX 3090, 24GB VRAM, 64GB RAM, Ubuntu 24.04) and designed for reuse on the planned Mac Mini M4 host later this week (see "Portability" below).

Supersedes the manual, pre-role state left behind by earlier vLLM experiments (/home/jarvis/vllm-env, bitsandbytes, gemma-2-27b — see homelab-llm-inference/homelab-llm-serving skills for that history). This role uses a fresh venv (vllm_venv_path, default ~/vllm-serve-env) and AWQ pre-quantized models — no bitsandbytes, no on-the-fly quantization, no repeat of the OOM incident from the earlier Gemma-2-27B attempt.

Phases

Phase	File	What it does
1	tasks/dependencies.yml	System Python 3.10+, dedicated venv, pip install vllm>=0.5.0, verifies nvidia-smi and torch.cuda.is_available()
2	tasks/models.yml	Downloads each enabled: true model in vllm_models via hf download (huggingface_hub CLI) into ~/.vllm-cache, verifies the snapshot landed and reports on-disk size
3	tasks/api-key.yml	Reads the API key from 1Password (op://mk-labs/vllm/api-key) on the controller, writes it to /etc/vllm/api-key.env (root:root, 0600) on the target
4	tasks/systemd.yml	Renders and installs one systemd unit per enabled model (vllm.service for the role: primary model, vllm-<id>.service for others)
5	tasks/verify.yml	Only runs when vllm_service_state=started. Waits for /health (up to 5 min — torch.compile warmup), checks /v1/models, runs a live completion, scans journalctl for errors

Run all phases: ansible-playbook -i inventory.yml playbooks/day1_deploy_vllm.yml --limit astro-orbiter Run one phase: --tags vllm-dependencies / vllm-models / vllm-api-key / vllm-systemd / vllm-verify

Deliberate staging-first default

vllm_service_state defaults to stopped. A default run stages everything (venv, model weights, API key file, systemd unit) but does not start the service or touch production traffic. This matches the astro-orbiter cutover plan: llama-swap is live production serving (Qwen3.8-27B

• nomic-embed for Hindsight) — vLLM must be deployed and validated on a side port/inactive unit before anything is cut over.

To start and validate:

ansible-playbook -i inventory.yml playbooks/day1_deploy_vllm.yml \
  --limit astro-orbiter --extra-vars "vllm_service_state=started"

This starts the systemd unit(s), enables them, and runs Phase 5 verification (health, /v1/models, live completion, clean journalctl).

Cutover of consumers (Hermes profiles, Hindsight embedding config, any hardcoded :8001/:5805 references) to the new :8000 vLLM endpoint is a separate, explicit step outside this role — do this only after Phase 5 passes cleanly. Do not tear down llama-swap until consumers are confirmed working end-to-end against vLLM.

Model roster (vllm_models in defaults/main.yml)

vLLM 0.5.x-0.28.x serves one model per process — multi-model = multiple systemd units on distinct ports, not a single multiplexed server (unlike llama-swap's matrix DSL). Today's phase enables only the primary model; flip enabled: true on the others as VRAM allows (see "Phased Strategy"):

id	hf_repo	role	port	quant	enabled
Qwen2.5-32B-Instruct-AWQ	Qwen/Qwen2.5-32B-Instruct-AWQ	primary	8000	awq	true
Qwen3-8B-AWQ	Qwen/Qwen3-8B-AWQ	aux	8010	awq	false
nomic-embed-text-v1.5	nomic-ai/nomic-embed-text-v1.5	embedding	8020	none	false

Note on the original spec's model choices: the task body named Qwen/Qwen2.5-32B-Instruct and Qwen/Qwen3-8B-Instruct (bf16, unquantized). vLLM does not do on-the-fly quantization safely on this host (bitsandbytes OOM history — see homelab-llm-inference skill Pitfalls) and unquantized bf16 32B does not fit a 24GB card at all (~65GB). This role instead deploys the official Qwen AWQ pre-quantized variants (Qwen/Qwen2.5-32B-Instruct-AWQ, Qwen/Qwen3-8B-AWQ), which vLLM natively supports (--quantization awq) and which fit the VRAM budget:

• Qwen2.5-32B-Instruct-AWQ: ~19.3GB on disk, fits with ~5GB headroom at 24GB
• Qwen3-8B-AWQ: ~6GB VRAM per llm-explorer
• nomic-embed-text-v1.5: ~300MB, vLLM serves it via --convert embed pooling (see vLLM embedding docs) — not yet wired into this role's systemd template; the embedding model needs --task embed / --convert embed flags that differ from the completion-serving template. Flagged as a follow-up before enabled: true is flipped on it (see Known Gaps below).

Known Gaps / Follow-ups

• The nomic-embed-text-v1.5 entry in vllm_models is present but the vllm.service.j2 template does not yet branch for embedding-mode flags (--task embed). Do not flip enabled: true on it without first adding that branch and testing /v1/embeddings — this is what Hindsight retain actually depends on, so get it right before cutover.
• Quarterly API key rotation is documented (/etc/vllm/API_KEY_ROTATION.md on the target, rendered by tasks/api-key.yml) but not automated — no cron job exists to force rotation on a schedule. Consider a follow-up cron task if Nick Fury wants this enforced rather than just documented.
• vllm_service_enabled defaults to false deliberately — see "Deliberate staging-first default" above. Flip together with the cutover step, not before.

Validation Log (2026-08-31, t_ca1af9fb)

Full Phase 1-5 run executed against astro-orbiter in a brief shadow-validation window (llama-swap stopped ~5 min, per the homelab-llm-inference skill's documented shadow-validation pattern — production traffic could not be tested concurrently with vLLM's VRAM footprint on this 24GB card).

Two real bugs found and fixed during first-start validation (not present in the original spec, discovered only by actually starting the service):

1. ninja not on systemd's PATH. vLLM's torch.compile path shells out to the bare ninja command. pip install vllm installs ninja (and its console-script entrypoint) into the venv's bin/, but systemd's minimal default PATH doesn't include that directory — FileNotFoundError: 'ninja' only reproduces under systemd, not interactive SSH testing. Fixed by setting Environment="PATH=<venv>/bin:...standard dirs..." in the unit template.
2. FlashInfer sampler JIT fails to compile on RTX 3090 (SM86). flashinfer/data/csrc/sampling.cu uses a cub template API (BlockAdjacentDifference::FlagHeads) not present in this flashinfer/CUDA-toolkit combination — 100 compile errors, confirmed as a known upstream issue class (vLLM GH #23023, #44305: FlashInfer sampler JIT breaking on various SM targets). Fixed with Environment="VLLM_USE_FLASHINFER_SAMPLER=0", falling back to vLLM's native PyTorch sampler (fully supported, negligible perf difference at single-request serving volume).

Also corrected vllm_gpu_memory_utilization from 0.90 to 0.95 — at 0.90 the KV cache allocation failed (2.0 GiB KV cache needed, 1.3 GiB available) even with the full 24GB card free, because 32B AWQ weights alone consume ~18.4GB, leaving too little headroom at a 90% cap.

Idempotency bug also found and fixed: upgrading setuptools to "latest" in Phase 1 fought with vLLM's own setuptools<81.0.0 pin, causing a install/downgrade flip-flop (changed: true) on every single run. Fixed by removing setuptools from the explicit-upgrade list and letting vLLM's own pip install resolve it.

Final validated result, once these fixes were applied:

• systemctl status vllm.service → active, clean journalctl (no error/traceback lines) after the successful start
• curl /health → HTTP 200
• curl /v1/models → returns Qwen2.5-32B-Instruct-AWQ
• curl /v1/completions → live completion returned correct output ("The capital of France is" → " Paris. Correct! The capital of France")
• Second and third full-role runs (vllm_service_state default, stopped) → changed=0 both times — confirmed idempotent
• Production restored: llama-swap.service active, /health 200, /v1/embeddings against nomic-embed-text-v1.5 returns a valid vector — Hindsight retain path confirmed still working after the shadow window
• Post-restore VRAM: 486 MiB used / 24,576 MiB total (normal quiescent state)

Testing this role (idempotency)

Second-run test (staging phases only, safe to run repeatedly):

ansible-playbook -i inventory.yml playbooks/day1_deploy_vllm.yml \
  --limit astro-orbiter --tags vllm-dependencies,vllm-models,vllm-api-key,vllm-systemd
# Run it again immediately — expect changed=0 (or only handler-driven
# restarts if vllm_service_state=started and the API key file rotated)

Confirmed 2026-08-31 (t_ca1af9fb): Phase 1 (dependencies) ran once with changed=3 (venv create, pip upgrade, vllm install); a second run reported changed=0 for those three tasks — venv creates: guard and pip module's own idempotency both held.

Portability — Mac Mini M4 (planned, end of week)

This role's host-specific assumptions live in defaults/main.yml (all overridable via host_vars/<host>/vars.yml) plus one hard assumption baked into tasks/dependencies.yml: an NVIDIA GPU (nvidia-smi check, CUDA wheels). Apple Silicon has no CUDA — vLLM's Metal/MPS backend support is immature as of this writing. Before reusing this role for the Mac Mini M4:

1. Fork tasks/dependencies.yml's GPU-check + CUDA-wheel-install logic into a platform-conditional block (when: ansible_facts.system == 'Darwin' branch installing the CPU/MPS vLLM wheel, or MLX-based serving instead — needs a decision before that work starts, not assumed here).
2. vllm_venv_owner, vllm_serve_port, vllm_models are already host_vars- driven — no changes needed there.
3. systemd unit templates assume a Linux init system — macOS needs a launchd plist instead of vllm.service.j2.

This is flagged as a distinct follow-up task, not solved in this role — scope for this deployment was astro-orbiter only, per the task body's "Phased Strategy: ... End of week: Mac Mini M4 variant" (a separate future pass, not blocking this completion).

Files

roles/deploy-vllm/
├── defaults/main.yml               # all tunables — host overrides go in host_vars
├── handlers/main.yml                # reload systemd / restart vllm services
├── meta/main.yml
├── tasks/
│   ├── main.yml                     # phase orchestrator
│   ├── dependencies.yml             # Phase 1
│   ├── models.yml                   # Phase 2
│   ├── api-key.yml                  # Phase 3
│   ├── systemd.yml                  # Phase 4
│   └── verify.yml                   # Phase 5
├── templates/
│   ├── vllm.service.j2              # one instance per enabled model
│   └── vllm-workspace.sh.j2         # debugging helper deployed to the target
└── README.md                        # this file