deploy-vllm: add embedding-mode support, cut over Hindsight to vLLM (t_e6facb19)
- vllm.service.j2: branch on role==embedding for --runner pooling --convert embed, --no-enable-prefix-caching, per-model trust_remote_code toggle (needed for nomic-embed-text-v1.5's custom NomicBertModel code), and enforce_eager toggle (needed to avoid CUDA graph capture OOM when co-resident with another vLLM process on this 24GB card). - tasks/verify.yml: split completions vs embedding smoke tests -- embedding-mode instances don't serve /v1/completions. Assert a non-empty embedding vector, not just HTTP 200. - host_vars/astro-orbiter: enable nomic-embed-text-v1.5 (port 8020), lower primary model's gpu_memory_utilization 0.95->0.90 + add enforce_eager after finding 0.95 crash-looped 6-7x before stabilizing with co-resident nomic-embed (real fix, confirmed via NRestarts=0 after clean stop/start, not luck). - Hindsight (values.yaml + externalsecret.yaml): cut LLM + embeddings over to vLLM (:8000, :8020), wire the previously-unset HINDSIGHT_API_EMBEDDINGS_* env vars for the first time, and swap the API key secret source from the Nous fallback item to vllm/api-key (vLLM enforces real auth, llama-swap did not). - README: document the embedding-mode branch, VRAM findings, and a genuine architecture gap -- vLLM's one-model-per-process design cannot replace llama-swap's 5-model LRU roster on this 24GB card, so 21 Hermes profiles' aux-model consumers (Qwen3-8B-no_think, Phi-3.5-mini, Meta-Llama-3.1-8B, Qwen2.5-Coder-14B) and OpenViking's VLM stay on llama-swap. Full teardown (t_6dff1ecc) needs a human decision on the aux-model strategy before it can proceed.
This commit is contained in:
@@ -34,6 +34,16 @@ ExecStart={{ vllm_venv_path }}/bin/python -m vllm.entrypoints.openai.api_server
|
||||
--served-model-name {{ item.id }} \
|
||||
--host {{ vllm_serve_host }} \
|
||||
--port {{ item.port }} \
|
||||
{% if item.role == 'embedding' %}
|
||||
--runner pooling \
|
||||
--convert embed \
|
||||
{% endif %}
|
||||
{% if item.trust_remote_code is defined and item.trust_remote_code %}
|
||||
--trust-remote-code \
|
||||
{% endif %}
|
||||
{% if item.enforce_eager is defined and item.enforce_eager %}
|
||||
--enforce-eager \
|
||||
{% endif %}
|
||||
{% if item.quantization is defined and item.quantization != 'none' %}
|
||||
--quantization {{ item.quantization }} \
|
||||
{% endif %}
|
||||
@@ -41,7 +51,11 @@ ExecStart={{ vllm_venv_path }}/bin/python -m vllm.entrypoints.openai.api_server
|
||||
--max-model-len {{ item.max_model_len }} \
|
||||
--dtype {{ vllm_dtype }} \
|
||||
--api-key ${VLLM_API_KEY} \
|
||||
{% if item.role != 'embedding' %}
|
||||
--enable-prefix-caching
|
||||
{% else %}
|
||||
--no-enable-prefix-caching
|
||||
{% endif %}
|
||||
|
||||
Restart={{ vllm_restart_policy }}
|
||||
RestartSec=10
|
||||
|
||||
Reference in New Issue
Block a user