60220e18b6
Idempotent vLLM OpenAI-compatible serving role, staged-first (does not
start/enable the systemd unit or touch production traffic by default).
Validated end-to-end against astro-orbiter in a brief shadow window
(llama-swap stopped ~5 min, per homelab-llm-inference skill's documented
shadow-validation pattern):
- /health 200, /v1/models returns Qwen2.5-32B-Instruct-AWQ,
/v1/completions live smoke test passes, clean journalctl
- 3 consecutive full-role runs confirmed changed=0 (idempotent)
- production restored: llama-swap active, /v1/embeddings against
nomic-embed-text-v1.5 confirmed still working (Hindsight retain path)
Deviates from the original spec's model choices (Qwen2.5-32B-Instruct /
Qwen3-8B-Instruct bf16) to use the official Qwen AWQ pre-quantized variants
instead -- vLLM does not do safe on-the-fly quantization on this host
(bitsandbytes OOM history) and unquantized bf16 32B does not fit 24GB VRAM.
Two real bugs found+fixed during first-start validation (systemd-only
repro, not visible via interactive SSH testing):
1. ninja not on systemd's minimal PATH -> vLLM torch.compile
FileNotFoundError. Fixed via explicit PATH env in the unit.
2. FlashInfer sampler JIT fails to compile on RTX 3090 (SM86) --
known upstream issue class (vLLM GH #23023, #44305). Fixed via
VLLM_USE_FLASHINFER_SAMPLER=0 (falls back to native sampler).
Also fixed a real idempotency bug: force-upgrading setuptools to latest
fought with vLLM's own setuptools<81.0.0 pin, causing an install/downgrade
flip-flop (changed:true) on every run.
vllm_service_enabled defaults to false -- a host reboot must not
auto-start vLLM and VRAM-collide with the still-live llama-swap production
service. Cutover (enabling + starting + migrating consumers) is an
explicit, separate step outside this role, gated on adding embedding-mode
support (--task embed) for nomic-embed-text-v1.5, which this role does
not yet implement (Hindsight retain still depends on llama-swap's
nomic-embed until that follow-up lands).
Role: roles/deploy-vllm/ (defaults/handlers/meta/tasks/templates/README)
Playbook: playbooks/day1_deploy_vllm.yml
88 lines
3.3 KiB
YAML
88 lines
3.3 KiB
YAML
# ------------------------------------------------------------------------------
|
|
# FILE: roles/deploy-vllm/tasks/models.yml
|
|
# PHASE 2: Model downloads via huggingface-cli into {{ vllm_hf_hub_cache }}.
|
|
#
|
|
# Idempotency: HuggingFace's on-disk cache layout is
|
|
# {cache}/models--{org}--{repo}/snapshots/{revision}/...
|
|
# We stat for an existing snapshots dir before downloading — if present with
|
|
# at least one entry, skip (huggingface-cli download is itself resumable/
|
|
# idempotent, but this avoids even the "check remote manifest" round trip on
|
|
# every run and gives a clean "already staged" line in output).
|
|
#
|
|
# Pitfall (t_3dddf37d, homelab-llm-inference skill): a config/template landing
|
|
# is NOT the same as the model being staged. Always verify via `ls`/`du` on
|
|
# the actual host, never trust a prior task's claim alone.
|
|
# ------------------------------------------------------------------------------
|
|
|
|
- name: Ensure model cache directory exists
|
|
ansible.builtin.file:
|
|
path: "{{ vllm_hf_hub_cache }}"
|
|
state: directory
|
|
owner: "{{ vllm_venv_owner }}"
|
|
group: "{{ vllm_venv_owner }}"
|
|
mode: "0755"
|
|
become: true
|
|
|
|
- name: Report models to be staged this run
|
|
ansible.builtin.debug:
|
|
msg: "{{ vllm_enabled_models | map(attribute='id') | list }}"
|
|
|
|
- name: Check for existing snapshot dir per enabled model
|
|
ansible.builtin.stat:
|
|
path: "{{ vllm_hf_hub_cache }}/models--{{ item.hf_repo | regex_replace('/', '--') }}/snapshots"
|
|
loop: "{{ vllm_enabled_models }}"
|
|
loop_control:
|
|
label: "{{ item.id }}"
|
|
register: vllm_model_snapshot_stat
|
|
|
|
- name: Download model repo(s) not yet staged
|
|
ansible.builtin.command:
|
|
cmd: >-
|
|
{{ vllm_venv_path }}/bin/hf download {{ item.item.hf_repo }}
|
|
--cache-dir {{ vllm_hf_hub_cache }}
|
|
become: true
|
|
become_user: "{{ vllm_venv_owner }}"
|
|
environment:
|
|
HF_HUB_ENABLE_HF_TRANSFER: "0"
|
|
loop: "{{ vllm_model_snapshot_stat.results }}"
|
|
loop_control:
|
|
label: "{{ item.item.id }}"
|
|
when: not (item.stat.exists | default(false)) or (item.stat.isdir | default(false) and item.stat.size == 0)
|
|
register: vllm_model_download
|
|
# Full-size model pulls (9-18GB for 32B AWQ) can take a long time on
|
|
# homelab bandwidth — allow up to 1 hour per model.
|
|
async: 3600
|
|
poll: 30
|
|
|
|
- name: Re-stat snapshot dirs to confirm download landed
|
|
ansible.builtin.stat:
|
|
path: "{{ vllm_hf_hub_cache }}/models--{{ item.hf_repo | regex_replace('/', '--') }}/snapshots"
|
|
loop: "{{ vllm_enabled_models }}"
|
|
loop_control:
|
|
label: "{{ item.id }}"
|
|
register: vllm_model_snapshot_verify
|
|
|
|
- name: Fail if any enabled model failed to stage
|
|
ansible.builtin.fail:
|
|
msg: "Model {{ item.item.id }} ({{ item.item.hf_repo }}) is not present at {{ vllm_hf_hub_cache }} after download step."
|
|
loop: "{{ vllm_model_snapshot_verify.results }}"
|
|
loop_control:
|
|
label: "{{ item.item.id }}"
|
|
when: not (item.stat.exists | default(false))
|
|
|
|
- name: Compute on-disk size of each staged model (sanity check, not a strict checksum)
|
|
ansible.builtin.command:
|
|
cmd: "du -sh {{ vllm_hf_hub_cache }}/models--{{ item.hf_repo | regex_replace('/', '--') }}"
|
|
loop: "{{ vllm_enabled_models }}"
|
|
loop_control:
|
|
label: "{{ item.id }}"
|
|
register: vllm_model_size
|
|
changed_when: false
|
|
|
|
- name: Report staged model sizes
|
|
ansible.builtin.debug:
|
|
msg: "{{ item.stdout }}"
|
|
loop: "{{ vllm_model_size.results }}"
|
|
loop_control:
|
|
label: "{{ item.item.id }}"
|