2cc9370f3d
- vllm.service.j2: branch on role==embedding for --runner pooling --convert embed, --no-enable-prefix-caching, per-model trust_remote_code toggle (needed for nomic-embed-text-v1.5's custom NomicBertModel code), and enforce_eager toggle (needed to avoid CUDA graph capture OOM when co-resident with another vLLM process on this 24GB card). - tasks/verify.yml: split completions vs embedding smoke tests -- embedding-mode instances don't serve /v1/completions. Assert a non-empty embedding vector, not just HTTP 200. - host_vars/astro-orbiter: enable nomic-embed-text-v1.5 (port 8020), lower primary model's gpu_memory_utilization 0.95->0.90 + add enforce_eager after finding 0.95 crash-looped 6-7x before stabilizing with co-resident nomic-embed (real fix, confirmed via NRestarts=0 after clean stop/start, not luck). - Hindsight (values.yaml + externalsecret.yaml): cut LLM + embeddings over to vLLM (:8000, :8020), wire the previously-unset HINDSIGHT_API_EMBEDDINGS_* env vars for the first time, and swap the API key secret source from the Nous fallback item to vllm/api-key (vLLM enforces real auth, llama-swap did not). - README: document the embedding-mode branch, VRAM findings, and a genuine architecture gap -- vLLM's one-model-per-process design cannot replace llama-swap's 5-model LRU roster on this 24GB card, so 21 Hermes profiles' aux-model consumers (Qwen3-8B-no_think, Phi-3.5-mini, Meta-Llama-3.1-8B, Qwen2.5-Coder-14B) and OpenViking's VLM stay on llama-swap. Full teardown (t_6dff1ecc) needs a human decision on the aux-model strategy before it can proceed.
92 lines
4.2 KiB
YAML
92 lines
4.2 KiB
YAML
# ------------------------------------------------------------------------------
|
|
# FILE: roles/deploy-vllm/defaults/main.yml
|
|
# ROLE: deploy-vllm — vLLM OpenAI-compatible serving stack
|
|
# DESIGNED FOR REUSE: astro-orbiter (RTX 3090, 24GB) today, Mac Mini M4 later.
|
|
# Host-specific values (VRAM budget, model list, ports) belong in host_vars,
|
|
# not here. These are the safe, conservative defaults.
|
|
# ------------------------------------------------------------------------------
|
|
|
|
# --- Python / venv -----------------------------------------------------------
|
|
vllm_venv_owner: jarvis
|
|
vllm_venv_path: "/home/{{ vllm_venv_owner }}/vllm-serve-env"
|
|
vllm_python_min_version: "3.10"
|
|
vllm_version_spec: "vllm>=0.5.0"
|
|
|
|
# --- Model cache ---------------------------------------------------------
|
|
vllm_cache_dir: "/home/{{ vllm_venv_owner }}/.vllm-cache"
|
|
vllm_hf_hub_cache: "{{ vllm_cache_dir }}/huggingface"
|
|
|
|
# --- Serving ---------------------------------------------------------------
|
|
vllm_serve_host: "0.0.0.0"
|
|
vllm_serve_port: 8000
|
|
vllm_gpu_memory_utilization: 0.95
|
|
vllm_max_model_len: 8192
|
|
vllm_dtype: "auto"
|
|
|
|
# --- Models --------------------------------------------------------------
|
|
# Each entry: id (served --model / OpenAI "model" field), hf_repo, role
|
|
# (primary/aux/embedding), quantization, and per-model overrides.
|
|
# Only models with enabled: true are staged + wired into the systemd unit's
|
|
# --model roster consideration. vLLM 0.5.x serves ONE model per process, so
|
|
# multi-model = multiple systemd instances (see vllm_instances below) or a
|
|
# router in front (out of scope for this role — matches the astro-orbiter
|
|
# phased plan: Qwen2.5-32B today, add Qwen3-8B + embedding later).
|
|
vllm_models:
|
|
- id: "Qwen2.5-32B-Instruct-AWQ"
|
|
hf_repo: "Qwen/Qwen2.5-32B-Instruct-AWQ"
|
|
role: primary
|
|
quantization: awq
|
|
port: 8000
|
|
max_model_len: "{{ vllm_max_model_len }}"
|
|
gpu_memory_utilization: "{{ vllm_gpu_memory_utilization }}"
|
|
enabled: true
|
|
- id: "Qwen3-8B-AWQ"
|
|
hf_repo: "Qwen/Qwen3-8B-AWQ"
|
|
role: aux
|
|
quantization: awq
|
|
port: 8010
|
|
max_model_len: 32768
|
|
gpu_memory_utilization: 0.15
|
|
enabled: false
|
|
- id: "nomic-embed-text-v1.5"
|
|
hf_repo: "nomic-ai/nomic-embed-text-v1.5"
|
|
role: embedding
|
|
quantization: none
|
|
port: 8020
|
|
max_model_len: 2048
|
|
gpu_memory_utilization: 0.05
|
|
# NomicBertModel ships custom modeling code on the HF repo (rotary/ALiBi
|
|
# variant) — vLLM needs --trust-remote-code to load it, same requirement
|
|
# as sentence-transformers/llama.cpp. Wired into vllm.service.j2 (t_e6facb19).
|
|
trust_remote_code: true
|
|
enabled: false
|
|
|
|
# --- systemd ---------------------------------------------------------------
|
|
vllm_service_name: vllm
|
|
vllm_service_state: stopped # deliberate: role stages everything but does NOT
|
|
# flip production traffic. Cutover is a separate,
|
|
# explicitly-approved step (see README.md).
|
|
vllm_service_enabled: false # deliberate: do NOT enable for boot by default.
|
|
# llama-swap is live production on this GPU —
|
|
# enabling vllm.service means a host reboot would
|
|
# auto-start it and immediately VRAM-collide with
|
|
# llama-swap (confirmed failure mode during Phase 5
|
|
# validation, t_ca1af9fb 2026-08-31). Flip to true
|
|
# only as part of the deliberate cutover step,
|
|
# together with tearing down llama-swap.
|
|
vllm_restart_policy: always
|
|
|
|
# --- API key -----------------------------------------------------------
|
|
# Source of truth: 1Password op://mk-labs/vllm/api-key (Nick Fury manages).
|
|
# This role does NOT generate a key by default — it expects one to already
|
|
# exist in 1Password and reads it via `op read` at deploy time (delegate_to
|
|
# localhost, where the op CLI is authenticated). Set vllm_generate_api_key
|
|
# to true only for first-ever bootstrap when no 1Password item exists yet.
|
|
vllm_generate_api_key: false
|
|
vllm_api_key_op_ref: "op://mk-labs/vllm/api-key"
|
|
vllm_api_key_env_file: "/etc/vllm/api-key.env"
|
|
|
|
# --- Verification ------------------------------------------------------
|
|
vllm_health_check_retries: 30
|
|
vllm_health_check_delay: 10
|