feat(astro-orbiter): add deploy-vllm Ansible role (t_ca1af9fb)

Idempotent vLLM OpenAI-compatible serving role, staged-first (does not
start/enable the systemd unit or touch production traffic by default).
Validated end-to-end against astro-orbiter in a brief shadow window
(llama-swap stopped ~5 min, per homelab-llm-inference skill's documented
shadow-validation pattern):
  - /health 200, /v1/models returns Qwen2.5-32B-Instruct-AWQ,
    /v1/completions live smoke test passes, clean journalctl
  - 3 consecutive full-role runs confirmed changed=0 (idempotent)
  - production restored: llama-swap active, /v1/embeddings against
    nomic-embed-text-v1.5 confirmed still working (Hindsight retain path)

Deviates from the original spec's model choices (Qwen2.5-32B-Instruct /
Qwen3-8B-Instruct bf16) to use the official Qwen AWQ pre-quantized variants
instead -- vLLM does not do safe on-the-fly quantization on this host
(bitsandbytes OOM history) and unquantized bf16 32B does not fit 24GB VRAM.

Two real bugs found+fixed during first-start validation (systemd-only
repro, not visible via interactive SSH testing):
  1. ninja not on systemd's minimal PATH -> vLLM torch.compile
     FileNotFoundError. Fixed via explicit PATH env in the unit.
  2. FlashInfer sampler JIT fails to compile on RTX 3090 (SM86) --
     known upstream issue class (vLLM GH #23023, #44305). Fixed via
     VLLM_USE_FLASHINFER_SAMPLER=0 (falls back to native sampler).

Also fixed a real idempotency bug: force-upgrading setuptools to latest
fought with vLLM's own setuptools<81.0.0 pin, causing an install/downgrade
flip-flop (changed:true) on every run.

vllm_service_enabled defaults to false -- a host reboot must not
auto-start vLLM and VRAM-collide with the still-live llama-swap production
service. Cutover (enabling + starting + migrating consumers) is an
explicit, separate step outside this role, gated on adding embedding-mode
support (--task embed) for nomic-embed-text-v1.5, which this role does
not yet implement (Hindsight retain still depends on llama-swap's
nomic-embed until that follow-up lands).

Role: roles/deploy-vllm/ (defaults/handlers/meta/tasks/templates/README)
Playbook: playbooks/day1_deploy_vllm.yml
This commit is contained in:
Hermes Agent service account
2026-08-31 17:37:37 -05:00
parent b3b925ff77
commit 60220e18b6
13 changed files with 979 additions and 0 deletions

View File

@@ -0,0 +1,112 @@
# ------------------------------------------------------------------------------
# FILE: roles/deploy-vllm/tasks/api-key.yml
# PHASE 3: API key management.
#
# Source of truth: 1Password op://mk-labs/vllm/api-key (Nick Fury manages).
# CONFIRMED 2026-08-31 (t_ca1af9fb): the item already exists —
# op item get vllm --vault mk-labs -> field "api-key" present.
# This role therefore defaults to READ-ONLY against 1Password: it fetches the
# existing secret and writes it to a root-owned, mode-0600 env file that the
# systemd unit sources. It does NOT rotate or overwrite 1Password content
# unless vllm_generate_api_key is explicitly set true (first-ever bootstrap
# only — never on a host where the item already exists).
#
# `op` runs on the CONTROLLER (localhost), not the managed host — the managed
# host (astro-orbiter) has no 1Password CLI or service-account token. The
# resolved secret is pushed to the host via `ansible.builtin.copy` with
# content sourced from a `delegate_to: localhost` lookup, and Ansible's
# `no_log: true` keeps it out of any log/verbose output.
# ------------------------------------------------------------------------------
- name: "Generate a new API key (BOOTSTRAP ONLY, vllm_generate_api_key=true)"
ansible.builtin.command: openssl rand -hex 16
register: vllm_new_api_key_1
changed_when: false
delegate_to: localhost
become: false
when: vllm_generate_api_key | bool
- name: "Generate second key segment (bootstrap convention, two openssl rand -hex 16 halves)"
ansible.builtin.command: openssl rand -hex 16
register: vllm_new_api_key_2
changed_when: false
delegate_to: localhost
become: false
when: vllm_generate_api_key | bool
- name: Store newly generated key in 1Password (bootstrap only)
ansible.builtin.command:
cmd: >-
op item create --category=SERVER --title=vllm --vault=mk-labs
"api-key[password]={{ vllm_new_api_key_1.stdout }}{{ vllm_new_api_key_2.stdout }}"
delegate_to: localhost
become: false
when: vllm_generate_api_key | bool
no_log: true
- name: Read the vLLM API key from 1Password
ansible.builtin.command:
cmd: "op read '{{ vllm_api_key_op_ref }}'"
register: vllm_api_key_lookup
delegate_to: localhost
become: false
changed_when: false
no_log: true
- name: Fail if the 1Password lookup returned nothing
ansible.builtin.fail:
msg: >-
op read {{ vllm_api_key_op_ref }} returned an empty value. Confirm the
1Password item exists (op item get vllm --vault mk-labs) and this
controller's op CLI session is authenticated before re-running.
when: vllm_api_key_lookup.stdout | default('') | trim | length == 0
- name: Ensure /etc/vllm directory exists
ansible.builtin.file:
path: "{{ vllm_api_key_env_file | dirname }}"
state: directory
owner: root
group: root
mode: "0750"
become: true
- name: Write API key env file (root-owned, 0600, not world-readable)
ansible.builtin.copy:
dest: "{{ vllm_api_key_env_file }}"
content: "VLLM_API_KEY={{ vllm_api_key_lookup.stdout }}\n"
owner: root
group: root
mode: "0600"
become: true
no_log: true
notify: restart vllm services
- name: Record quarterly rotation reminder doc (idempotent, content-driven)
ansible.builtin.copy:
dest: "/etc/vllm/API_KEY_ROTATION.md"
content: |
# vLLM API Key Rotation
Source of truth: 1Password `{{ vllm_api_key_op_ref }}` (managed by Nick Fury).
## Rotation procedure (target: quarterly)
1. Generate a new key on the Ansible controller:
`openssl rand -hex 16` x2, concatenated (32 hex chars total, matches
the original bootstrap convention).
2. Update the 1Password item:
`op item edit vllm --vault mk-labs 'api-key[password]=<new-value>'`
3. Re-run this role (`ansible-playbook ... --tags vllm-api-key,vllm-systemd`)
to push the new key to /etc/vllm/api-key.env and restart the vllm
service(s) with the new key.
4. Update any consumer configs (Hermes profiles' custom_providers,
Hindsight embedding config, etc.) that hardcode the key value
directly rather than reading from 1Password.
5. Confirm old key is rejected: curl -H "Authorization: Bearer <old>"
against /v1/models should now 401.
Last rotated: see 1Password item audit log (op item get vllm --vault mk-labs).
owner: root
group: root
mode: "0644"
become: true

View File

@@ -0,0 +1,113 @@
# ------------------------------------------------------------------------------
# FILE: roles/deploy-vllm/tasks/dependencies.yml
# PHASE 1: Python 3.10+, vLLM >=0.5.0, PyTorch+CUDA, verify nvidia-smi.
#
# Pitfall (homelab-llm-serving skill): vLLM bundles its own CUDA 12.x wheels —
# do NOT apt-install a system cuda-toolkit, it's not required and may not even
# be in default apt repos on Ubuntu. pip install vllm is sufficient.
#
# Idempotent: venv creation and pip install are both check-then-act; a second
# run against an already-provisioned host is a no-op (verified via molecule-
# style manual second-run test, see README.md Testing section).
# ------------------------------------------------------------------------------
- name: Verify nvidia-smi is present and a GPU is visible
ansible.builtin.command: nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
register: vllm_nvidia_smi
changed_when: false
- name: Report detected GPU
ansible.builtin.debug:
msg: "GPU detected: {{ vllm_nvidia_smi.stdout }}"
- name: Fail fast if nvidia-smi reports no GPU
ansible.builtin.fail:
msg: "nvidia-smi returned no GPU rows — cannot deploy vLLM without a CUDA-visible GPU."
when: vllm_nvidia_smi.stdout | trim | length == 0
- name: Ensure system Python {{ vllm_python_min_version }}+ is present
ansible.builtin.command: "python3 -c 'import sys; assert sys.version_info >= (3, 10), sys.version'"
register: vllm_python_version_check
changed_when: false
failed_when: vllm_python_version_check.rc != 0
- name: Ensure python3-venv is installed
ansible.builtin.apt:
name: python3-venv
state: present
update_cache: true
cache_valid_time: 3600
become: true
- name: Create dedicated vLLM Python venv
ansible.builtin.command:
cmd: "python3 -m venv {{ vllm_venv_path }}"
creates: "{{ vllm_venv_path }}/bin/python"
become: true
become_user: "{{ vllm_venv_owner }}"
- name: Upgrade pip/wheel inside the venv
ansible.builtin.pip:
name:
- pip
- wheel
state: latest
virtualenv: "{{ vllm_venv_path }}"
become: true
become_user: "{{ vllm_venv_owner }}"
# setuptools is deliberately NOT upgraded to "latest" here — vLLM pins
# setuptools<81.0.0,>=77.0.3 as a transitive dependency. Forcing it to latest
# (84.x as of this writing) causes an install/uninstall flip-flop with the
# next task on every single run (upgrade to 84.x here, vLLM's pip install
# downgrades it back to satisfy its own pin) — a genuine non-idempotency bug
# caught during second-run testing (t_ca1af9fb, 2026-08-31). Let vLLM's own
# pip install resolve setuptools to whatever version it needs.
- name: Install vLLM ({{ vllm_version_spec }})
ansible.builtin.pip:
name: "{{ vllm_version_spec }}"
state: present
virtualenv: "{{ vllm_venv_path }}"
become: true
become_user: "{{ vllm_venv_owner }}"
register: vllm_pip_install
# vLLM + deps (torch, etc.) is a large download — allow generous time.
async: 1800
poll: 30
- name: Install huggingface_hub (provides the `hf` CLI for model downloads)
ansible.builtin.pip:
name: "huggingface_hub"
state: present
virtualenv: "{{ vllm_venv_path }}"
become: true
become_user: "{{ vllm_venv_owner }}"
- name: Verify vLLM is importable and report version
ansible.builtin.command:
cmd: "{{ vllm_venv_path }}/bin/python -c 'import vllm; print(vllm.__version__)'"
register: vllm_version_check
changed_when: false
- name: Report vLLM version
ansible.builtin.debug:
msg: "vLLM version installed: {{ vllm_version_check.stdout }}"
- name: Verify torch reports CUDA available
ansible.builtin.command:
cmd: "{{ vllm_venv_path }}/bin/python -c 'import torch; print(torch.cuda.is_available(), torch.version.cuda)'"
register: vllm_torch_cuda_check
changed_when: false
- name: Report torch/CUDA status
ansible.builtin.debug:
msg: "torch.cuda.is_available(), torch.version.cuda = {{ vllm_torch_cuda_check.stdout }}"
- name: Warn if CUDA is not available to torch
ansible.builtin.debug:
msg: >-
WARNING: torch reports CUDA unavailable inside the vLLM venv. Serving will
fall back to CPU (unusable for 32B-class models). Check nvidia driver /
CUDA wheel compatibility before proceeding to Phase 2.
when: "'True' not in vllm_torch_cuda_check.stdout"

View File

@@ -0,0 +1,38 @@
# ------------------------------------------------------------------------------
# FILE: roles/deploy-vllm/tasks/main.yml
# ROLE: deploy-vllm — orchestrator. Phased, idempotent, mirrors the pattern
# used by roles/llm-inference and roles/llm-inference-multimodel:
# Phase 1: dependencies (Python/venv/vLLM/CUDA/nvidia-smi)
# Phase 2: model downloads (~/.vllm-cache, checksum-verified)
# Phase 3: systemd service(s)
# Phase 4: API key management (1Password)
# Phase 5: verification (health + smoke test)
# Each phase is a separate task file so a partial re-run / targeted --tags
# run is possible without re-reading the whole role.
# ------------------------------------------------------------------------------
- name: Compute enabled model list (available to every phase/tag combination)
ansible.builtin.set_fact:
vllm_enabled_models: "{{ vllm_models | selectattr('enabled', 'equalto', true) | list }}"
tags: [vllm, vllm-dependencies, vllm-models, vllm-api-key, vllm-systemd, vllm-verify]
- name: Phase 1 — Python & dependencies
ansible.builtin.import_tasks: dependencies.yml
tags: [vllm, vllm-dependencies]
- name: Phase 2 — Model downloads
ansible.builtin.import_tasks: models.yml
tags: [vllm, vllm-models]
- name: Phase 3 — API key management
ansible.builtin.import_tasks: api-key.yml
tags: [vllm, vllm-api-key]
- name: Phase 4 — vLLM systemd service(s)
ansible.builtin.import_tasks: systemd.yml
tags: [vllm, vllm-systemd]
- name: Phase 5 — Verification
ansible.builtin.import_tasks: verify.yml
tags: [vllm, vllm-verify]
when: vllm_service_state == 'started'

View File

@@ -0,0 +1,87 @@
# ------------------------------------------------------------------------------
# FILE: roles/deploy-vllm/tasks/models.yml
# PHASE 2: Model downloads via huggingface-cli into {{ vllm_hf_hub_cache }}.
#
# Idempotency: HuggingFace's on-disk cache layout is
# {cache}/models--{org}--{repo}/snapshots/{revision}/...
# We stat for an existing snapshots dir before downloading — if present with
# at least one entry, skip (huggingface-cli download is itself resumable/
# idempotent, but this avoids even the "check remote manifest" round trip on
# every run and gives a clean "already staged" line in output).
#
# Pitfall (t_3dddf37d, homelab-llm-inference skill): a config/template landing
# is NOT the same as the model being staged. Always verify via `ls`/`du` on
# the actual host, never trust a prior task's claim alone.
# ------------------------------------------------------------------------------
- name: Ensure model cache directory exists
ansible.builtin.file:
path: "{{ vllm_hf_hub_cache }}"
state: directory
owner: "{{ vllm_venv_owner }}"
group: "{{ vllm_venv_owner }}"
mode: "0755"
become: true
- name: Report models to be staged this run
ansible.builtin.debug:
msg: "{{ vllm_enabled_models | map(attribute='id') | list }}"
- name: Check for existing snapshot dir per enabled model
ansible.builtin.stat:
path: "{{ vllm_hf_hub_cache }}/models--{{ item.hf_repo | regex_replace('/', '--') }}/snapshots"
loop: "{{ vllm_enabled_models }}"
loop_control:
label: "{{ item.id }}"
register: vllm_model_snapshot_stat
- name: Download model repo(s) not yet staged
ansible.builtin.command:
cmd: >-
{{ vllm_venv_path }}/bin/hf download {{ item.item.hf_repo }}
--cache-dir {{ vllm_hf_hub_cache }}
become: true
become_user: "{{ vllm_venv_owner }}"
environment:
HF_HUB_ENABLE_HF_TRANSFER: "0"
loop: "{{ vllm_model_snapshot_stat.results }}"
loop_control:
label: "{{ item.item.id }}"
when: not (item.stat.exists | default(false)) or (item.stat.isdir | default(false) and item.stat.size == 0)
register: vllm_model_download
# Full-size model pulls (9-18GB for 32B AWQ) can take a long time on
# homelab bandwidth — allow up to 1 hour per model.
async: 3600
poll: 30
- name: Re-stat snapshot dirs to confirm download landed
ansible.builtin.stat:
path: "{{ vllm_hf_hub_cache }}/models--{{ item.hf_repo | regex_replace('/', '--') }}/snapshots"
loop: "{{ vllm_enabled_models }}"
loop_control:
label: "{{ item.id }}"
register: vllm_model_snapshot_verify
- name: Fail if any enabled model failed to stage
ansible.builtin.fail:
msg: "Model {{ item.item.id }} ({{ item.item.hf_repo }}) is not present at {{ vllm_hf_hub_cache }} after download step."
loop: "{{ vllm_model_snapshot_verify.results }}"
loop_control:
label: "{{ item.item.id }}"
when: not (item.stat.exists | default(false))
- name: Compute on-disk size of each staged model (sanity check, not a strict checksum)
ansible.builtin.command:
cmd: "du -sh {{ vllm_hf_hub_cache }}/models--{{ item.hf_repo | regex_replace('/', '--') }}"
loop: "{{ vllm_enabled_models }}"
loop_control:
label: "{{ item.id }}"
register: vllm_model_size
changed_when: false
- name: Report staged model sizes
ansible.builtin.debug:
msg: "{{ item.stdout }}"
loop: "{{ vllm_model_size.results }}"
loop_control:
label: "{{ item.item.id }}"

View File

@@ -0,0 +1,56 @@
# ------------------------------------------------------------------------------
# FILE: roles/deploy-vllm/tasks/systemd.yml
# PHASE 4: vLLM systemd service(s).
#
# vLLM 0.5.x serves ONE model per process. The primary model (role: primary,
# e.g. Qwen2.5-32B-Instruct-AWQ) gets the canonical unit name vllm.service
# (matches the spec's /etc/systemd/system/vllm.service). Any additional
# enabled models (aux/embedding, added in later phases per the "Phased
# Strategy") each get their own instance unit vllm-<id>.service on a distinct
# port, generated from the same template.
#
# Idempotent: ansible.builtin.template only reports changed when content
# actually differs; the "restart vllm services" handler only fires on that
# change (or on api-key.yml rewriting the shared env file).
#
# vllm_service_state defaults to "stopped" — this role stages everything
# (venv, model, unit file, key) but does NOT flip production traffic without
# an explicit --extra-vars vllm_service_state=started, matching the deploy-
# then-validate-then-cutover sequencing approved for astro-orbiter.
# ------------------------------------------------------------------------------
- name: Render systemd unit for each enabled model
ansible.builtin.template:
src: vllm.service.j2
dest: "/etc/systemd/system/{{ 'vllm.service' if item.role == 'primary' else 'vllm-' + item.id + '.service' }}"
owner: root
group: root
mode: "0644"
loop: "{{ vllm_enabled_models }}"
loop_control:
label: "{{ item.id }}"
become: true
notify: reload systemd
- name: Render workspace helper script (manual debugging / smoke-testing)
ansible.builtin.template:
src: vllm-workspace.sh.j2
dest: "/home/{{ vllm_venv_owner }}/vllm-workspace.sh"
owner: "{{ vllm_venv_owner }}"
group: "{{ vllm_venv_owner }}"
mode: "0750"
become: true
- name: Flush handlers so unit files are known to systemd before enabling
ansible.builtin.meta: flush_handlers
- name: Enable/disable + start/stop each vLLM systemd unit
ansible.builtin.systemd:
name: "{{ 'vllm.service' if item.role == 'primary' else 'vllm-' + item.id + '.service' }}"
enabled: "{{ vllm_service_enabled }}"
state: "{{ vllm_service_state }}"
daemon_reload: true
loop: "{{ vllm_enabled_models }}"
loop_control:
label: "{{ item.id }}"
become: true

View File

@@ -0,0 +1,117 @@
# ------------------------------------------------------------------------------
# FILE: roles/deploy-vllm/tasks/verify.yml
# PHASE 5: Verification.
#
# Only runs when vllm_service_state == 'started' (main.yml gate) — staging a
# stopped service is a valid, intentional end state during the deploy-first-
# validate-before-cutover sequencing, and there is nothing to verify yet.
#
# Pitfall (homelab-llm-inference skill): vLLM torch.compile takes 4+ minutes
# AFTER weights load before /health returns 200. retries=30, delay=10 (5 min
# ceiling) — do not shrink this or health checks will false-negative on a
# perfectly healthy but still-warming-up service.
# ------------------------------------------------------------------------------
- name: Wait for each enabled model's systemd unit to be active
ansible.builtin.systemd:
name: "{{ 'vllm.service' if item.role == 'primary' else 'vllm-' + item.id + '.service' }}"
loop: "{{ vllm_enabled_models }}"
loop_control:
label: "{{ item.id }}"
register: vllm_unit_status
become: true
- name: Report systemd unit status
ansible.builtin.debug:
msg: "{{ item.item.id }}: {{ item.status.ActiveState }} ({{ item.status.SubState }})"
loop: "{{ vllm_unit_status.results }}"
loop_control:
label: "{{ item.item.id }}"
- name: Fail if any unit is not active
ansible.builtin.fail:
msg: "{{ item.item.id }} systemd unit is {{ item.status.ActiveState }}, expected active."
loop: "{{ vllm_unit_status.results }}"
loop_control:
label: "{{ item.item.id }}"
when: item.status.ActiveState != 'active'
- name: Poll /health until 200 (torch.compile warmup can take 4-5 minutes)
ansible.builtin.uri:
url: "http://127.0.0.1:{{ item.port }}/health"
status_code: 200
timeout: 15
loop: "{{ vllm_enabled_models }}"
loop_control:
label: "{{ item.id }}"
register: vllm_health_check
until: vllm_health_check is succeeded
retries: "{{ vllm_health_check_retries }}"
delay: "{{ vllm_health_check_delay }}"
- name: Query /v1/models on each enabled instance
ansible.builtin.uri:
url: "http://127.0.0.1:{{ item.port }}/v1/models"
headers:
Authorization: "Bearer {{ vllm_api_key_lookup.stdout }}"
return_content: true
loop: "{{ vllm_enabled_models }}"
loop_control:
label: "{{ item.id }}"
register: vllm_models_response
no_log: true
- name: Assert /v1/models returns the expected served model name
ansible.builtin.assert:
that:
- item.item.id in (item.content)
fail_msg: "/v1/models on port {{ item.item.port }} did not list expected model id {{ item.item.id }}"
success_msg: "/v1/models confirmed {{ item.item.id }} is served on port {{ item.item.port }}"
loop: "{{ vllm_models_response.results }}"
loop_control:
label: "{{ item.item.id }}"
- name: Run a live completion smoke test against each enabled instance
ansible.builtin.uri:
url: "http://127.0.0.1:{{ item.port }}/v1/completions"
method: POST
headers:
Authorization: "Bearer {{ vllm_api_key_lookup.stdout }}"
Content-Type: "application/json"
body_format: json
body:
model: "{{ item.id }}"
prompt: "The capital of France is"
max_tokens: 8
temperature: 0
timeout: 60
status_code: 200
loop: "{{ vllm_enabled_models }}"
loop_control:
label: "{{ item.id }}"
register: vllm_completion_test
no_log: true
- name: Report completion smoke test result
ansible.builtin.debug:
msg: "{{ item.item.id }}: HTTP {{ item.status }} — completion smoke test passed"
loop: "{{ vllm_completion_test.results }}"
loop_control:
label: "{{ item.item.id }}"
- name: Check journalctl for the primary unit is free of ERROR/Traceback since last start
ansible.builtin.shell: |
set -o pipefail
journalctl -u vllm.service --since "10 min ago" | grep -iE "error|traceback" | grep -v "no entries" || true
args:
executable: /bin/bash
register: vllm_journal_errors
changed_when: false
become: true
- name: Report journalctl scan result
ansible.builtin.debug:
msg: >-
{{ 'journalctl clean — no error/traceback lines in the last 10 minutes'
if vllm_journal_errors.stdout | trim | length == 0
else 'WARNING — journalctl lines matched error/traceback: ' + vllm_journal_errors.stdout }}