deploy-vllm: add embedding-mode support, cut over Hindsight to vLLM (t_e6facb19)

- vllm.service.j2: branch on role==embedding for --runner pooling
  --convert embed, --no-enable-prefix-caching, per-model
  trust_remote_code toggle (needed for nomic-embed-text-v1.5's custom
  NomicBertModel code), and enforce_eager toggle (needed to avoid CUDA
  graph capture OOM when co-resident with another vLLM process on this
  24GB card).
- tasks/verify.yml: split completions vs embedding smoke tests --
  embedding-mode instances don't serve /v1/completions. Assert a
  non-empty embedding vector, not just HTTP 200.
- host_vars/astro-orbiter: enable nomic-embed-text-v1.5 (port 8020),
  lower primary model's gpu_memory_utilization 0.95->0.90 + add
  enforce_eager after finding 0.95 crash-looped 6-7x before stabilizing
  with co-resident nomic-embed (real fix, confirmed via NRestarts=0
  after clean stop/start, not luck).
- Hindsight (values.yaml + externalsecret.yaml): cut LLM + embeddings
  over to vLLM (:8000, :8020), wire the previously-unset
  HINDSIGHT_API_EMBEDDINGS_* env vars for the first time, and swap the
  API key secret source from the Nous fallback item to vllm/api-key
  (vLLM enforces real auth, llama-swap did not).
- README: document the embedding-mode branch, VRAM findings, and a
  genuine architecture gap -- vLLM's one-model-per-process design
  cannot replace llama-swap's 5-model LRU roster on this 24GB card, so
  21 Hermes profiles' aux-model consumers (Qwen3-8B-no_think,
  Phi-3.5-mini, Meta-Llama-3.1-8B, Qwen2.5-Coder-14B) and OpenViking's
  VLM stay on llama-swap. Full teardown (t_6dff1ecc) needs a human
  decision on the aux-model strategy before it can proceed.
This commit is contained in:
Hermes Agent service account
2026-08-31 18:15:22 -05:00
parent 60220e18b6
commit 2cc9370f3d
7 changed files with 275 additions and 22 deletions

View File

@@ -55,6 +55,10 @@ vllm_models:
port: 8020
max_model_len: 2048
gpu_memory_utilization: 0.05
# NomicBertModel ships custom modeling code on the HF repo (rotary/ALiBi
# variant) — vLLM needs --trust-remote-code to load it, same requirement
# as sentence-transformers/llama.cpp. Wired into vllm.service.j2 (t_e6facb19).
trust_remote_code: true
enabled: false
# --- systemd ---------------------------------------------------------------