deploy-vllm: add embedding-mode support, cut over Hindsight to vLLM (t_e6facb19)
- vllm.service.j2: branch on role==embedding for --runner pooling --convert embed, --no-enable-prefix-caching, per-model trust_remote_code toggle (needed for nomic-embed-text-v1.5's custom NomicBertModel code), and enforce_eager toggle (needed to avoid CUDA graph capture OOM when co-resident with another vLLM process on this 24GB card). - tasks/verify.yml: split completions vs embedding smoke tests -- embedding-mode instances don't serve /v1/completions. Assert a non-empty embedding vector, not just HTTP 200. - host_vars/astro-orbiter: enable nomic-embed-text-v1.5 (port 8020), lower primary model's gpu_memory_utilization 0.95->0.90 + add enforce_eager after finding 0.95 crash-looped 6-7x before stabilizing with co-resident nomic-embed (real fix, confirmed via NRestarts=0 after clean stop/start, not luck). - Hindsight (values.yaml + externalsecret.yaml): cut LLM + embeddings over to vLLM (:8000, :8020), wire the previously-unset HINDSIGHT_API_EMBEDDINGS_* env vars for the first time, and swap the API key secret source from the Nous fallback item to vllm/api-key (vLLM enforces real auth, llama-swap did not). - README: document the embedding-mode branch, VRAM findings, and a genuine architecture gap -- vLLM's one-model-per-process design cannot replace llama-swap's 5-model LRU roster on this 24GB card, so 21 Hermes profiles' aux-model consumers (Qwen3-8B-no_think, Phi-3.5-mini, Meta-Llama-3.1-8B, Qwen2.5-Coder-14B) and OpenViking's VLM stay on llama-swap. Full teardown (t_6dff1ecc) needs a human decision on the aux-model strategy before it can proceed.
This commit is contained in:
@@ -109,3 +109,57 @@ llm_staged_models:
|
||||
size_bytes: 5027784224
|
||||
source_repo: "bartowski/Qwen_Qwen3-8B-GGUF"
|
||||
|
||||
# --- deploy-vllm role: vllm_models override (t_e6facb19, 2026-08-31) --------
|
||||
# Ansible's hash_behaviour is "replace" (see ansible.cfg) — a host_vars list
|
||||
# variable REPLACES the role default list wholesale, it does not deep-merge.
|
||||
# This is therefore a full copy of roles/deploy-vllm/defaults/main.yml's
|
||||
# vllm_models with ONE change: nomic-embed-text-v1.5.enabled flipped to true,
|
||||
# now that vllm.service.j2 has an embedding-mode branch (--runner pooling
|
||||
# --convert embed --trust-remote-code) tested end-to-end in a shadow window.
|
||||
# Primary (Qwen2.5-32B-Instruct-AWQ) and aux (Qwen3-8B-AWQ) entries are
|
||||
# unchanged from role defaults — reproduced here only because the whole list
|
||||
# must be redefined together. Keep this in sync with defaults/main.yml if the
|
||||
# role's model roster changes.
|
||||
vllm_models:
|
||||
- id: "Qwen2.5-32B-Instruct-AWQ"
|
||||
hf_repo: "Qwen/Qwen2.5-32B-Instruct-AWQ"
|
||||
role: primary
|
||||
quantization: awq
|
||||
port: 8000
|
||||
max_model_len: 8192
|
||||
# 0.95 (role default) OOM'd during CUDA graph capture once nomic-embed
|
||||
# (role: embedding, ~814MiB actual, not the nominal 300MB) is co-resident
|
||||
# on the same 24GB card (t_e6facb19, 2026-08-31): KV cache allocation
|
||||
# succeeded (14,720 tokens) but graph capture needed ~20MiB more than the
|
||||
# 0.95 budget left after nomic's share. Two independent, permanent
|
||||
# co-residents (unlike t_ca1af9fb's shadow-window test, which had the
|
||||
# whole 24GB free) need either a lower utilization ceiling or no graph
|
||||
# capture. enforce_eager avoids the whole cudagraph capture memory spike
|
||||
# entirely — small throughput cost, no OOM risk, safer for a fixed
|
||||
# multi-process VRAM budget than tuning utilization percentages by hand.
|
||||
# Even WITH enforce_eager, 0.95 left only ~847MiB genuinely free out of
|
||||
# 24576MiB total (23,729MiB used) and both services crash-looped 6-7x
|
||||
# during warmup/KV-cache sizing before stabilizing — too fragile for a
|
||||
# permanent two-process co-residency. Lowered to 0.90 for real headroom
|
||||
# (~1.6GiB free), confirmed clean single-attempt start with no retries.
|
||||
gpu_memory_utilization: 0.90
|
||||
enforce_eager: true
|
||||
enabled: true
|
||||
- id: "Qwen3-8B-AWQ"
|
||||
hf_repo: "Qwen/Qwen3-8B-AWQ"
|
||||
role: aux
|
||||
quantization: awq
|
||||
port: 8010
|
||||
max_model_len: 32768
|
||||
gpu_memory_utilization: 0.15
|
||||
enabled: false
|
||||
- id: "nomic-embed-text-v1.5"
|
||||
hf_repo: "nomic-ai/nomic-embed-text-v1.5"
|
||||
role: embedding
|
||||
quantization: none
|
||||
port: 8020
|
||||
max_model_len: 2048
|
||||
gpu_memory_utilization: 0.05
|
||||
trust_remote_code: true
|
||||
enabled: true
|
||||
|
||||
|
||||
Reference in New Issue
Block a user