60220e18b6
Idempotent vLLM OpenAI-compatible serving role, staged-first (does not
start/enable the systemd unit or touch production traffic by default).
Validated end-to-end against astro-orbiter in a brief shadow window
(llama-swap stopped ~5 min, per homelab-llm-inference skill's documented
shadow-validation pattern):
- /health 200, /v1/models returns Qwen2.5-32B-Instruct-AWQ,
/v1/completions live smoke test passes, clean journalctl
- 3 consecutive full-role runs confirmed changed=0 (idempotent)
- production restored: llama-swap active, /v1/embeddings against
nomic-embed-text-v1.5 confirmed still working (Hindsight retain path)
Deviates from the original spec's model choices (Qwen2.5-32B-Instruct /
Qwen3-8B-Instruct bf16) to use the official Qwen AWQ pre-quantized variants
instead -- vLLM does not do safe on-the-fly quantization on this host
(bitsandbytes OOM history) and unquantized bf16 32B does not fit 24GB VRAM.
Two real bugs found+fixed during first-start validation (systemd-only
repro, not visible via interactive SSH testing):
1. ninja not on systemd's minimal PATH -> vLLM torch.compile
FileNotFoundError. Fixed via explicit PATH env in the unit.
2. FlashInfer sampler JIT fails to compile on RTX 3090 (SM86) --
known upstream issue class (vLLM GH #23023, #44305). Fixed via
VLLM_USE_FLASHINFER_SAMPLER=0 (falls back to native sampler).
Also fixed a real idempotency bug: force-upgrading setuptools to latest
fought with vLLM's own setuptools<81.0.0 pin, causing an install/downgrade
flip-flop (changed:true) on every run.
vllm_service_enabled defaults to false -- a host reboot must not
auto-start vLLM and VRAM-collide with the still-live llama-swap production
service. Cutover (enabling + starting + migrating consumers) is an
explicit, separate step outside this role, gated on adding embedding-mode
support (--task embed) for nomic-embed-text-v1.5, which this role does
not yet implement (Hindsight retain still depends on llama-swap's
nomic-embed until that follow-up lands).
Role: roles/deploy-vllm/ (defaults/handlers/meta/tasks/templates/README)
Playbook: playbooks/day1_deploy_vllm.yml
55 lines
1.9 KiB
Django/Jinja
55 lines
1.9 KiB
Django/Jinja
#!/usr/bin/env bash
|
|
# ------------------------------------------------------------------------------
|
|
# FILE: vllm-workspace.sh — deployed by roles/deploy-vllm to
|
|
# /home/{{ vllm_venv_owner }}/vllm-workspace.sh
|
|
#
|
|
# Convenience wrapper for manual debugging / smoke-testing the vLLM venv
|
|
# without having to remember the venv path or model roster each time.
|
|
# Regenerated on every Ansible run — do not hand-edit, edit the template
|
|
# instead (roles/deploy-vllm/templates/vllm-workspace.sh.j2).
|
|
# ------------------------------------------------------------------------------
|
|
set -euo pipefail
|
|
|
|
VENV="{{ vllm_venv_path }}"
|
|
CACHE="{{ vllm_hf_hub_cache }}"
|
|
API_KEY_FILE="{{ vllm_api_key_env_file }}"
|
|
|
|
usage() {
|
|
cat <<EOF
|
|
Usage: $0 <command>
|
|
|
|
Commands:
|
|
activate Print the command to source the vLLM venv
|
|
version Print installed vLLM + torch/CUDA versions
|
|
models List staged model snapshots in the HF cache
|
|
curl-models curl /v1/models on each enabled instance (requires sudo to read API key)
|
|
logs <unit> Tail journalctl for a vllm systemd unit (e.g. vllm.service)
|
|
EOF
|
|
}
|
|
|
|
case "${1:-}" in
|
|
activate)
|
|
echo "source $VENV/bin/activate"
|
|
;;
|
|
version)
|
|
"$VENV/bin/python" -c 'import vllm, torch; print("vllm", vllm.__version__); print("torch", torch.__version__, "cuda", torch.version.cuda, "available", torch.cuda.is_available())'
|
|
;;
|
|
models)
|
|
find "$CACHE" -maxdepth 1 -type d -name 'models--*' -printf '%f\n' 2>/dev/null || echo "(no models staged yet)"
|
|
;;
|
|
curl-models)
|
|
{% for item in vllm_enabled_models | default([]) %}
|
|
echo "--- {{ item.id }} (:{{ item.port }}) ---"
|
|
curl -s -H "Authorization: Bearer $(sudo grep -oP '(?<=VLLM_API_KEY=).*' "$API_KEY_FILE")" \
|
|
http://127.0.0.1:{{ item.port }}/v1/models | python3 -m json.tool || true
|
|
{% endfor %}
|
|
;;
|
|
logs)
|
|
sudo journalctl -u "${2:-vllm.service}" -f
|
|
;;
|
|
*)
|
|
usage
|
|
exit 1
|
|
;;
|
|
esac
|