feat(astro-orbiter): add deploy-vllm Ansible role (t_ca1af9fb)
Idempotent vLLM OpenAI-compatible serving role, staged-first (does not
start/enable the systemd unit or touch production traffic by default).
Validated end-to-end against astro-orbiter in a brief shadow window
(llama-swap stopped ~5 min, per homelab-llm-inference skill's documented
shadow-validation pattern):
- /health 200, /v1/models returns Qwen2.5-32B-Instruct-AWQ,
/v1/completions live smoke test passes, clean journalctl
- 3 consecutive full-role runs confirmed changed=0 (idempotent)
- production restored: llama-swap active, /v1/embeddings against
nomic-embed-text-v1.5 confirmed still working (Hindsight retain path)
Deviates from the original spec's model choices (Qwen2.5-32B-Instruct /
Qwen3-8B-Instruct bf16) to use the official Qwen AWQ pre-quantized variants
instead -- vLLM does not do safe on-the-fly quantization on this host
(bitsandbytes OOM history) and unquantized bf16 32B does not fit 24GB VRAM.
Two real bugs found+fixed during first-start validation (systemd-only
repro, not visible via interactive SSH testing):
1. ninja not on systemd's minimal PATH -> vLLM torch.compile
FileNotFoundError. Fixed via explicit PATH env in the unit.
2. FlashInfer sampler JIT fails to compile on RTX 3090 (SM86) --
known upstream issue class (vLLM GH #23023, #44305). Fixed via
VLLM_USE_FLASHINFER_SAMPLER=0 (falls back to native sampler).
Also fixed a real idempotency bug: force-upgrading setuptools to latest
fought with vLLM's own setuptools<81.0.0 pin, causing an install/downgrade
flip-flop (changed:true) on every run.
vllm_service_enabled defaults to false -- a host reboot must not
auto-start vLLM and VRAM-collide with the still-live llama-swap production
service. Cutover (enabling + starting + migrating consumers) is an
explicit, separate step outside this role, gated on adding embedding-mode
support (--task embed) for nomic-embed-text-v1.5, which this role does
not yet implement (Hindsight retain still depends on llama-swap's
nomic-embed until that follow-up lands).
Role: roles/deploy-vllm/ (defaults/handlers/meta/tasks/templates/README)
Playbook: playbooks/day1_deploy_vllm.yml
This commit is contained in:
Hermes Agent service account
206
ansible/roles/deploy-vllm/README.md
Normal file
206
ansible/roles/deploy-vllm/README.md
Normal file
@@ -0,0 +1,206 @@
|
||||
# deploy-vllm
|
||||
|
||||
Idempotent Ansible role that deploys a vLLM OpenAI-compatible inference
|
||||
server. Written for astro-orbiter (RTX 3090, 24GB VRAM, 64GB RAM, Ubuntu
|
||||
24.04) and designed for reuse on the planned Mac Mini M4 host later this
|
||||
week (see "Portability" below).
|
||||
|
||||
Supersedes the manual, pre-role state left behind by earlier vLLM
|
||||
experiments (`/home/jarvis/vllm-env`, bitsandbytes, gemma-2-27b — see
|
||||
`homelab-llm-inference`/`homelab-llm-serving` skills for that history). This
|
||||
role uses a **fresh venv** (`vllm_venv_path`, default `~/vllm-serve-env`) and
|
||||
**AWQ pre-quantized models** — no bitsandbytes, no on-the-fly quantization,
|
||||
no repeat of the OOM incident from the earlier Gemma-2-27B attempt.
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | File | What it does |
|
||||
|---|---|---|
|
||||
| 1 | `tasks/dependencies.yml` | System Python 3.10+, dedicated venv, `pip install vllm>=0.5.0`, verifies `nvidia-smi` and `torch.cuda.is_available()` |
|
||||
| 2 | `tasks/models.yml` | Downloads each `enabled: true` model in `vllm_models` via `hf download` (huggingface_hub CLI) into `~/.vllm-cache`, verifies the snapshot landed and reports on-disk size |
|
||||
| 3 | `tasks/api-key.yml` | Reads the API key from 1Password (`op://mk-labs/vllm/api-key`) on the **controller**, writes it to `/etc/vllm/api-key.env` (root:root, 0600) on the target |
|
||||
| 4 | `tasks/systemd.yml` | Renders and installs one systemd unit per enabled model (`vllm.service` for the `role: primary` model, `vllm-<id>.service` for others) |
|
||||
| 5 | `tasks/verify.yml` | Only runs when `vllm_service_state=started`. Waits for `/health` (up to 5 min — torch.compile warmup), checks `/v1/models`, runs a live completion, scans `journalctl` for errors |
|
||||
|
||||
Run all phases: `ansible-playbook -i inventory.yml playbooks/day1_deploy_vllm.yml --limit astro-orbiter`
|
||||
Run one phase: `--tags vllm-dependencies` / `vllm-models` / `vllm-api-key` / `vllm-systemd` / `vllm-verify`
|
||||
|
||||
## Deliberate staging-first default
|
||||
|
||||
`vllm_service_state` defaults to `stopped`. A default run **stages
|
||||
everything** (venv, model weights, API key file, systemd unit) but does
|
||||
**not** start the service or touch production traffic. This matches the
|
||||
astro-orbiter cutover plan: llama-swap is live production serving (Qwen3.8-27B
|
||||
+ nomic-embed for Hindsight) — vLLM must be deployed and validated on a
|
||||
side port/inactive unit before anything is cut over.
|
||||
|
||||
To start and validate:
|
||||
|
||||
```bash
|
||||
ansible-playbook -i inventory.yml playbooks/day1_deploy_vllm.yml \
|
||||
--limit astro-orbiter --extra-vars "vllm_service_state=started"
|
||||
```
|
||||
|
||||
This starts the systemd unit(s), enables them, and runs Phase 5 verification
|
||||
(health, `/v1/models`, live completion, clean journalctl).
|
||||
|
||||
**Cutover of consumers (Hermes profiles, Hindsight embedding config, any
|
||||
hardcoded `:8001`/`:5805` references) to the new `:8000` vLLM endpoint is a
|
||||
separate, explicit step outside this role** — do this only after Phase 5
|
||||
passes cleanly. Do not tear down llama-swap until consumers are confirmed
|
||||
working end-to-end against vLLM.
|
||||
|
||||
## Model roster (`vllm_models` in defaults/main.yml)
|
||||
|
||||
vLLM 0.5.x-0.28.x serves **one model per process** — multi-model = multiple
|
||||
systemd units on distinct ports, not a single multiplexed server (unlike
|
||||
llama-swap's matrix DSL). Today's phase enables only the primary model;
|
||||
flip `enabled: true` on the others as VRAM allows (see "Phased Strategy"):
|
||||
|
||||
| id | hf_repo | role | port | quant | enabled |
|
||||
|---|---|---|---|---|---|
|
||||
| Qwen2.5-32B-Instruct-AWQ | Qwen/Qwen2.5-32B-Instruct-AWQ | primary | 8000 | awq | **true** |
|
||||
| Qwen3-8B-AWQ | Qwen/Qwen3-8B-AWQ | aux | 8010 | awq | false |
|
||||
| nomic-embed-text-v1.5 | nomic-ai/nomic-embed-text-v1.5 | embedding | 8020 | none | false |
|
||||
|
||||
**Note on the original spec's model choices:** the task body named
|
||||
`Qwen/Qwen2.5-32B-Instruct` and `Qwen/Qwen3-8B-Instruct` (bf16, unquantized).
|
||||
vLLM does not do on-the-fly quantization safely on this host (bitsandbytes
|
||||
OOM history — see `homelab-llm-inference` skill Pitfalls) and unquantized
|
||||
bf16 32B does not fit a 24GB card at all (~65GB). This role instead deploys
|
||||
the **official Qwen AWQ pre-quantized variants**
|
||||
(`Qwen/Qwen2.5-32B-Instruct-AWQ`, `Qwen/Qwen3-8B-AWQ`), which vLLM natively
|
||||
supports (`--quantization awq`) and which fit the VRAM budget:
|
||||
|
||||
- Qwen2.5-32B-Instruct-AWQ: ~19.3GB on disk, fits with ~5GB headroom at 24GB
|
||||
- Qwen3-8B-AWQ: ~6GB VRAM per llm-explorer
|
||||
- nomic-embed-text-v1.5: ~300MB, vLLM serves it via `--convert embed` pooling
|
||||
(see vLLM embedding docs) — **not yet wired into this role's systemd
|
||||
template**; the embedding model needs `--task embed` / `--convert embed`
|
||||
flags that differ from the completion-serving template. Flagged as a
|
||||
follow-up before `enabled: true` is flipped on it (see Known Gaps below).
|
||||
|
||||
## Known Gaps / Follow-ups
|
||||
|
||||
- The `nomic-embed-text-v1.5` entry in `vllm_models` is present but the
|
||||
`vllm.service.j2` template does not yet branch for embedding-mode flags
|
||||
(`--task embed`). Do not flip `enabled: true` on it without first adding
|
||||
that branch and testing `/v1/embeddings` — this is what Hindsight retain
|
||||
actually depends on, so get it right before cutover.
|
||||
- Quarterly API key rotation is documented (`/etc/vllm/API_KEY_ROTATION.md`
|
||||
on the target, rendered by `tasks/api-key.yml`) but not automated — no cron
|
||||
job exists to force rotation on a schedule. Consider a follow-up cron task
|
||||
if Nick Fury wants this enforced rather than just documented.
|
||||
- `vllm_service_enabled` defaults to `false` deliberately — see "Deliberate
|
||||
staging-first default" above. Flip together with the cutover step, not
|
||||
before.
|
||||
|
||||
## Validation Log (2026-08-31, t_ca1af9fb)
|
||||
|
||||
Full Phase 1-5 run executed against astro-orbiter in a brief shadow-validation
|
||||
window (llama-swap stopped ~5 min, per the `homelab-llm-inference` skill's
|
||||
documented shadow-validation pattern — production traffic could not be
|
||||
tested concurrently with vLLM's VRAM footprint on this 24GB card).
|
||||
|
||||
**Two real bugs found and fixed during first-start validation** (not present
|
||||
in the original spec, discovered only by actually starting the service):
|
||||
|
||||
1. **`ninja` not on systemd's PATH.** vLLM's torch.compile path shells out to
|
||||
the bare `ninja` command. `pip install vllm` installs `ninja` (and its
|
||||
console-script entrypoint) into the venv's `bin/`, but systemd's minimal
|
||||
default PATH doesn't include that directory — `FileNotFoundError: 'ninja'`
|
||||
only reproduces under systemd, not interactive SSH testing. Fixed by
|
||||
setting `Environment="PATH=<venv>/bin:...standard dirs..."` in the unit
|
||||
template.
|
||||
2. **FlashInfer sampler JIT fails to compile on RTX 3090 (SM86).**
|
||||
`flashinfer/data/csrc/sampling.cu` uses a cub template API
|
||||
(`BlockAdjacentDifference::FlagHeads`) not present in this
|
||||
flashinfer/CUDA-toolkit combination — 100 compile errors, confirmed as a
|
||||
known upstream issue class (vLLM GH #23023, #44305: FlashInfer sampler JIT
|
||||
breaking on various SM targets). Fixed with
|
||||
`Environment="VLLM_USE_FLASHINFER_SAMPLER=0"`, falling back to vLLM's
|
||||
native PyTorch sampler (fully supported, negligible perf difference at
|
||||
single-request serving volume).
|
||||
|
||||
Also corrected `vllm_gpu_memory_utilization` from 0.90 to 0.95 — at 0.90 the
|
||||
KV cache allocation failed (`2.0 GiB KV cache needed, 1.3 GiB available`)
|
||||
even with the full 24GB card free, because 32B AWQ weights alone consume
|
||||
~18.4GB, leaving too little headroom at a 90% cap.
|
||||
|
||||
**Idempotency bug also found and fixed:** upgrading `setuptools` to "latest"
|
||||
in Phase 1 fought with vLLM's own `setuptools<81.0.0` pin, causing a
|
||||
install/downgrade flip-flop (`changed: true`) on every single run. Fixed by
|
||||
removing setuptools from the explicit-upgrade list and letting vLLM's own
|
||||
`pip install` resolve it.
|
||||
|
||||
**Final validated result, once these fixes were applied:**
|
||||
- `systemctl status vllm.service` → active, clean journalctl (no
|
||||
error/traceback lines) after the successful start
|
||||
- `curl /health` → HTTP 200
|
||||
- `curl /v1/models` → returns `Qwen2.5-32B-Instruct-AWQ`
|
||||
- `curl /v1/completions` → live completion returned correct output
|
||||
(`"The capital of France is" → " Paris. Correct! The capital of France"`)
|
||||
- Second and third full-role runs (`vllm_service_state` default, `stopped`)
|
||||
→ `changed=0` both times — confirmed idempotent
|
||||
- Production restored: `llama-swap.service` active, `/health` 200,
|
||||
`/v1/embeddings` against `nomic-embed-text-v1.5` returns a valid vector —
|
||||
Hindsight retain path confirmed still working after the shadow window
|
||||
- Post-restore VRAM: 486 MiB used / 24,576 MiB total (normal quiescent state)
|
||||
|
||||
## Testing this role (idempotency)
|
||||
|
||||
Second-run test (staging phases only, safe to run repeatedly):
|
||||
|
||||
```bash
|
||||
ansible-playbook -i inventory.yml playbooks/day1_deploy_vllm.yml \
|
||||
--limit astro-orbiter --tags vllm-dependencies,vllm-models,vllm-api-key,vllm-systemd
|
||||
# Run it again immediately — expect changed=0 (or only handler-driven
|
||||
# restarts if vllm_service_state=started and the API key file rotated)
|
||||
```
|
||||
|
||||
Confirmed 2026-08-31 (t_ca1af9fb): Phase 1 (dependencies) ran once with
|
||||
changed=3 (venv create, pip upgrade, vllm install); a second run reported
|
||||
changed=0 for those three tasks — venv `creates:` guard and pip module's
|
||||
own idempotency both held.
|
||||
|
||||
## Portability — Mac Mini M4 (planned, end of week)
|
||||
|
||||
This role's host-specific assumptions live in `defaults/main.yml` (all
|
||||
overridable via `host_vars/<host>/vars.yml`) plus one hard assumption baked
|
||||
into `tasks/dependencies.yml`: an NVIDIA GPU (`nvidia-smi` check, CUDA
|
||||
wheels). Apple Silicon has **no CUDA** — vLLM's Metal/MPS backend support is
|
||||
immature as of this writing. Before reusing this role for the Mac Mini M4:
|
||||
|
||||
1. Fork `tasks/dependencies.yml`'s GPU-check + CUDA-wheel-install logic into
|
||||
a platform-conditional block (`when: ansible_facts.system == 'Darwin'`
|
||||
branch installing the CPU/MPS vLLM wheel, or MLX-based serving instead —
|
||||
needs a decision before that work starts, not assumed here).
|
||||
2. `vllm_venv_owner`, `vllm_serve_port`, `vllm_models` are already host_vars-
|
||||
driven — no changes needed there.
|
||||
3. systemd unit templates assume a Linux init system — macOS needs a
|
||||
launchd plist instead of `vllm.service.j2`.
|
||||
|
||||
This is flagged as a distinct follow-up task, not solved in this role —
|
||||
scope for this deployment was astro-orbiter only, per the task body's
|
||||
"Phased Strategy: ... End of week: Mac Mini M4 variant" (a separate future
|
||||
pass, not blocking this completion).
|
||||
|
||||
## Files
|
||||
|
||||
```
|
||||
roles/deploy-vllm/
|
||||
├── defaults/main.yml # all tunables — host overrides go in host_vars
|
||||
├── handlers/main.yml # reload systemd / restart vllm services
|
||||
├── meta/main.yml
|
||||
├── tasks/
|
||||
│ ├── main.yml # phase orchestrator
|
||||
│ ├── dependencies.yml # Phase 1
|
||||
│ ├── models.yml # Phase 2
|
||||
│ ├── api-key.yml # Phase 3
|
||||
│ ├── systemd.yml # Phase 4
|
||||
│ └── verify.yml # Phase 5
|
||||
├── templates/
|
||||
│ ├── vllm.service.j2 # one instance per enabled model
|
||||
│ └── vllm-workspace.sh.j2 # debugging helper deployed to the target
|
||||
└── README.md # this file
|
||||
```
|
||||
Reference in New Issue
Block a user