60220e18b6
Idempotent vLLM OpenAI-compatible serving role, staged-first (does not
start/enable the systemd unit or touch production traffic by default).
Validated end-to-end against astro-orbiter in a brief shadow window
(llama-swap stopped ~5 min, per homelab-llm-inference skill's documented
shadow-validation pattern):
- /health 200, /v1/models returns Qwen2.5-32B-Instruct-AWQ,
/v1/completions live smoke test passes, clean journalctl
- 3 consecutive full-role runs confirmed changed=0 (idempotent)
- production restored: llama-swap active, /v1/embeddings against
nomic-embed-text-v1.5 confirmed still working (Hindsight retain path)
Deviates from the original spec's model choices (Qwen2.5-32B-Instruct /
Qwen3-8B-Instruct bf16) to use the official Qwen AWQ pre-quantized variants
instead -- vLLM does not do safe on-the-fly quantization on this host
(bitsandbytes OOM history) and unquantized bf16 32B does not fit 24GB VRAM.
Two real bugs found+fixed during first-start validation (systemd-only
repro, not visible via interactive SSH testing):
1. ninja not on systemd's minimal PATH -> vLLM torch.compile
FileNotFoundError. Fixed via explicit PATH env in the unit.
2. FlashInfer sampler JIT fails to compile on RTX 3090 (SM86) --
known upstream issue class (vLLM GH #23023, #44305). Fixed via
VLLM_USE_FLASHINFER_SAMPLER=0 (falls back to native sampler).
Also fixed a real idempotency bug: force-upgrading setuptools to latest
fought with vLLM's own setuptools<81.0.0 pin, causing an install/downgrade
flip-flop (changed:true) on every run.
vllm_service_enabled defaults to false -- a host reboot must not
auto-start vLLM and VRAM-collide with the still-live llama-swap production
service. Cutover (enabling + starting + migrating consumers) is an
explicit, separate step outside this role, gated on adding embedding-mode
support (--task embed) for nomic-embed-text-v1.5, which this role does
not yet implement (Hindsight retain still depends on llama-swap's
nomic-embed until that follow-up lands).
Role: roles/deploy-vllm/ (defaults/handlers/meta/tasks/templates/README)
Playbook: playbooks/day1_deploy_vllm.yml
88 lines
4.0 KiB
YAML
88 lines
4.0 KiB
YAML
# ------------------------------------------------------------------------------
|
|
# FILE: roles/deploy-vllm/defaults/main.yml
|
|
# ROLE: deploy-vllm — vLLM OpenAI-compatible serving stack
|
|
# DESIGNED FOR REUSE: astro-orbiter (RTX 3090, 24GB) today, Mac Mini M4 later.
|
|
# Host-specific values (VRAM budget, model list, ports) belong in host_vars,
|
|
# not here. These are the safe, conservative defaults.
|
|
# ------------------------------------------------------------------------------
|
|
|
|
# --- Python / venv -----------------------------------------------------------
|
|
vllm_venv_owner: jarvis
|
|
vllm_venv_path: "/home/{{ vllm_venv_owner }}/vllm-serve-env"
|
|
vllm_python_min_version: "3.10"
|
|
vllm_version_spec: "vllm>=0.5.0"
|
|
|
|
# --- Model cache ---------------------------------------------------------
|
|
vllm_cache_dir: "/home/{{ vllm_venv_owner }}/.vllm-cache"
|
|
vllm_hf_hub_cache: "{{ vllm_cache_dir }}/huggingface"
|
|
|
|
# --- Serving ---------------------------------------------------------------
|
|
vllm_serve_host: "0.0.0.0"
|
|
vllm_serve_port: 8000
|
|
vllm_gpu_memory_utilization: 0.95
|
|
vllm_max_model_len: 8192
|
|
vllm_dtype: "auto"
|
|
|
|
# --- Models --------------------------------------------------------------
|
|
# Each entry: id (served --model / OpenAI "model" field), hf_repo, role
|
|
# (primary/aux/embedding), quantization, and per-model overrides.
|
|
# Only models with enabled: true are staged + wired into the systemd unit's
|
|
# --model roster consideration. vLLM 0.5.x serves ONE model per process, so
|
|
# multi-model = multiple systemd instances (see vllm_instances below) or a
|
|
# router in front (out of scope for this role — matches the astro-orbiter
|
|
# phased plan: Qwen2.5-32B today, add Qwen3-8B + embedding later).
|
|
vllm_models:
|
|
- id: "Qwen2.5-32B-Instruct-AWQ"
|
|
hf_repo: "Qwen/Qwen2.5-32B-Instruct-AWQ"
|
|
role: primary
|
|
quantization: awq
|
|
port: 8000
|
|
max_model_len: "{{ vllm_max_model_len }}"
|
|
gpu_memory_utilization: "{{ vllm_gpu_memory_utilization }}"
|
|
enabled: true
|
|
- id: "Qwen3-8B-AWQ"
|
|
hf_repo: "Qwen/Qwen3-8B-AWQ"
|
|
role: aux
|
|
quantization: awq
|
|
port: 8010
|
|
max_model_len: 32768
|
|
gpu_memory_utilization: 0.15
|
|
enabled: false
|
|
- id: "nomic-embed-text-v1.5"
|
|
hf_repo: "nomic-ai/nomic-embed-text-v1.5"
|
|
role: embedding
|
|
quantization: none
|
|
port: 8020
|
|
max_model_len: 2048
|
|
gpu_memory_utilization: 0.05
|
|
enabled: false
|
|
|
|
# --- systemd ---------------------------------------------------------------
|
|
vllm_service_name: vllm
|
|
vllm_service_state: stopped # deliberate: role stages everything but does NOT
|
|
# flip production traffic. Cutover is a separate,
|
|
# explicitly-approved step (see README.md).
|
|
vllm_service_enabled: false # deliberate: do NOT enable for boot by default.
|
|
# llama-swap is live production on this GPU —
|
|
# enabling vllm.service means a host reboot would
|
|
# auto-start it and immediately VRAM-collide with
|
|
# llama-swap (confirmed failure mode during Phase 5
|
|
# validation, t_ca1af9fb 2026-08-31). Flip to true
|
|
# only as part of the deliberate cutover step,
|
|
# together with tearing down llama-swap.
|
|
vllm_restart_policy: always
|
|
|
|
# --- API key -----------------------------------------------------------
|
|
# Source of truth: 1Password op://mk-labs/vllm/api-key (Nick Fury manages).
|
|
# This role does NOT generate a key by default — it expects one to already
|
|
# exist in 1Password and reads it via `op read` at deploy time (delegate_to
|
|
# localhost, where the op CLI is authenticated). Set vllm_generate_api_key
|
|
# to true only for first-ever bootstrap when no 1Password item exists yet.
|
|
vllm_generate_api_key: false
|
|
vllm_api_key_op_ref: "op://mk-labs/vllm/api-key"
|
|
vllm_api_key_env_file: "/etc/vllm/api-key.env"
|
|
|
|
# --- Verification ------------------------------------------------------
|
|
vllm_health_check_retries: 30
|
|
vllm_health_check_delay: 10
|