60220e18b6
Idempotent vLLM OpenAI-compatible serving role, staged-first (does not
start/enable the systemd unit or touch production traffic by default).
Validated end-to-end against astro-orbiter in a brief shadow window
(llama-swap stopped ~5 min, per homelab-llm-inference skill's documented
shadow-validation pattern):
- /health 200, /v1/models returns Qwen2.5-32B-Instruct-AWQ,
/v1/completions live smoke test passes, clean journalctl
- 3 consecutive full-role runs confirmed changed=0 (idempotent)
- production restored: llama-swap active, /v1/embeddings against
nomic-embed-text-v1.5 confirmed still working (Hindsight retain path)
Deviates from the original spec's model choices (Qwen2.5-32B-Instruct /
Qwen3-8B-Instruct bf16) to use the official Qwen AWQ pre-quantized variants
instead -- vLLM does not do safe on-the-fly quantization on this host
(bitsandbytes OOM history) and unquantized bf16 32B does not fit 24GB VRAM.
Two real bugs found+fixed during first-start validation (systemd-only
repro, not visible via interactive SSH testing):
1. ninja not on systemd's minimal PATH -> vLLM torch.compile
FileNotFoundError. Fixed via explicit PATH env in the unit.
2. FlashInfer sampler JIT fails to compile on RTX 3090 (SM86) --
known upstream issue class (vLLM GH #23023, #44305). Fixed via
VLLM_USE_FLASHINFER_SAMPLER=0 (falls back to native sampler).
Also fixed a real idempotency bug: force-upgrading setuptools to latest
fought with vLLM's own setuptools<81.0.0 pin, causing an install/downgrade
flip-flop (changed:true) on every run.
vllm_service_enabled defaults to false -- a host reboot must not
auto-start vLLM and VRAM-collide with the still-live llama-swap production
service. Cutover (enabling + starting + migrating consumers) is an
explicit, separate step outside this role, gated on adding embedding-mode
support (--task embed) for nomic-embed-text-v1.5, which this role does
not yet implement (Hindsight retain still depends on llama-swap's
nomic-embed until that follow-up lands).
Role: roles/deploy-vllm/ (defaults/handlers/meta/tasks/templates/README)
Playbook: playbooks/day1_deploy_vllm.yml
57 lines
2.3 KiB
YAML
57 lines
2.3 KiB
YAML
# ------------------------------------------------------------------------------
|
|
# FILE: roles/deploy-vllm/tasks/systemd.yml
|
|
# PHASE 4: vLLM systemd service(s).
|
|
#
|
|
# vLLM 0.5.x serves ONE model per process. The primary model (role: primary,
|
|
# e.g. Qwen2.5-32B-Instruct-AWQ) gets the canonical unit name vllm.service
|
|
# (matches the spec's /etc/systemd/system/vllm.service). Any additional
|
|
# enabled models (aux/embedding, added in later phases per the "Phased
|
|
# Strategy") each get their own instance unit vllm-<id>.service on a distinct
|
|
# port, generated from the same template.
|
|
#
|
|
# Idempotent: ansible.builtin.template only reports changed when content
|
|
# actually differs; the "restart vllm services" handler only fires on that
|
|
# change (or on api-key.yml rewriting the shared env file).
|
|
#
|
|
# vllm_service_state defaults to "stopped" — this role stages everything
|
|
# (venv, model, unit file, key) but does NOT flip production traffic without
|
|
# an explicit --extra-vars vllm_service_state=started, matching the deploy-
|
|
# then-validate-then-cutover sequencing approved for astro-orbiter.
|
|
# ------------------------------------------------------------------------------
|
|
|
|
- name: Render systemd unit for each enabled model
|
|
ansible.builtin.template:
|
|
src: vllm.service.j2
|
|
dest: "/etc/systemd/system/{{ 'vllm.service' if item.role == 'primary' else 'vllm-' + item.id + '.service' }}"
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
loop: "{{ vllm_enabled_models }}"
|
|
loop_control:
|
|
label: "{{ item.id }}"
|
|
become: true
|
|
notify: reload systemd
|
|
|
|
- name: Render workspace helper script (manual debugging / smoke-testing)
|
|
ansible.builtin.template:
|
|
src: vllm-workspace.sh.j2
|
|
dest: "/home/{{ vllm_venv_owner }}/vllm-workspace.sh"
|
|
owner: "{{ vllm_venv_owner }}"
|
|
group: "{{ vllm_venv_owner }}"
|
|
mode: "0750"
|
|
become: true
|
|
|
|
- name: Flush handlers so unit files are known to systemd before enabling
|
|
ansible.builtin.meta: flush_handlers
|
|
|
|
- name: Enable/disable + start/stop each vLLM systemd unit
|
|
ansible.builtin.systemd:
|
|
name: "{{ 'vllm.service' if item.role == 'primary' else 'vllm-' + item.id + '.service' }}"
|
|
enabled: "{{ vllm_service_enabled }}"
|
|
state: "{{ vllm_service_state }}"
|
|
daemon_reload: true
|
|
loop: "{{ vllm_enabled_models }}"
|
|
loop_control:
|
|
label: "{{ item.id }}"
|
|
become: true
|