deploy-vllm: add embedding-mode support, cut over Hindsight to vLLM (t_e6facb19)
- vllm.service.j2: branch on role==embedding for --runner pooling --convert embed, --no-enable-prefix-caching, per-model trust_remote_code toggle (needed for nomic-embed-text-v1.5's custom NomicBertModel code), and enforce_eager toggle (needed to avoid CUDA graph capture OOM when co-resident with another vLLM process on this 24GB card). - tasks/verify.yml: split completions vs embedding smoke tests -- embedding-mode instances don't serve /v1/completions. Assert a non-empty embedding vector, not just HTTP 200. - host_vars/astro-orbiter: enable nomic-embed-text-v1.5 (port 8020), lower primary model's gpu_memory_utilization 0.95->0.90 + add enforce_eager after finding 0.95 crash-looped 6-7x before stabilizing with co-resident nomic-embed (real fix, confirmed via NRestarts=0 after clean stop/start, not luck). - Hindsight (values.yaml + externalsecret.yaml): cut LLM + embeddings over to vLLM (:8000, :8020), wire the previously-unset HINDSIGHT_API_EMBEDDINGS_* env vars for the first time, and swap the API key secret source from the Nous fallback item to vllm/api-key (vLLM enforces real auth, llama-swap did not). - README: document the embedding-mode branch, VRAM findings, and a genuine architecture gap -- vLLM's one-model-per-process design cannot replace llama-swap's 5-model LRU roster on this 24GB card, so 21 Hermes profiles' aux-model consumers (Qwen3-8B-no_think, Phi-3.5-mini, Meta-Llama-3.1-8B, Qwen2.5-Coder-14B) and OpenViking's VLM stay on llama-swap. Full teardown (t_6dff1ecc) needs a human decision on the aux-model strategy before it can proceed.
This commit is contained in:
Hermes Agent service account
@@ -55,6 +55,15 @@ spec:
|
||||
# env-var-named keys injected via envFrom
|
||||
HINDSIGHT_API_MCP_AUTH_TOKEN: "{{ .HINDSIGHT_API_MCP_AUTH_TOKEN }}"
|
||||
HINDSIGHT_API_LLM_API_KEY: "{{ .HINDSIGHT_API_LLM_API_KEY }}"
|
||||
# Cutover to vLLM (t_e6facb19, 2026-08-31): vLLM enforces its API key
|
||||
# on every request (unlike llama-swap, which accepted any/no key) —
|
||||
# confirmed empirically, a bad/missing key gets a real 401
|
||||
# {"error":"Unauthorized"}. Both the completions endpoint (:8000) and
|
||||
# the embeddings endpoint (:8020) are separate vLLM processes but
|
||||
# share the SAME key (api-key.env is written once, read by both
|
||||
# systemd units per roles/deploy-vllm/tasks/api-key.yml). Reuse
|
||||
# HINDSIGHT_API_LLM_API_KEY's value for the embeddings key too.
|
||||
HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY: "{{ .HINDSIGHT_API_LLM_API_KEY }}"
|
||||
data:
|
||||
# 1Password item "hindsight", field "postgres-password".
|
||||
# letters+digits only / URL-safe: required because the chart interpolates the
|
||||
@@ -69,11 +78,15 @@ spec:
|
||||
remoteRef:
|
||||
key: hindsight
|
||||
property: HINDSIGHT_API_MCP_AUTH_TOKEN
|
||||
# 1Password item "nous" (vault mk-labs), field "api-key" (Ryan-provisioned,
|
||||
# decision 4). Replaces the prior "local-placeholder" value — astro-orbiter
|
||||
# did not validate; now wired to the Nous free-tier inference API. Materialized
|
||||
# Secret key name stays HINDSIGHT_API_LLM_API_KEY (env-var-named, envFrom).
|
||||
# Cutover to vLLM (t_e6facb19, 2026-08-31): astro-orbiter's llama-swap
|
||||
# (which needed no real auth) is being retired for Hindsight's
|
||||
# completions+embeddings roles. vLLM enforces a real API key — read the
|
||||
# SAME key roles/deploy-vllm's api-key.yml phase writes to
|
||||
# /etc/vllm/api-key.env on astro-orbiter, sourced from 1Password
|
||||
# op://mk-labs/vllm/api-key (item "vllm", field "api-key", vault mk-labs).
|
||||
# Replaces the prior "nous" item's api-key (Nous free-tier fallback,
|
||||
# no longer the active LLM backend once this cutover lands).
|
||||
- secretKey: HINDSIGHT_API_LLM_API_KEY
|
||||
remoteRef:
|
||||
key: nous
|
||||
key: vllm
|
||||
property: api-key
|
||||
|
||||
@@ -60,12 +60,36 @@ existingSecret: hindsight-credentials
|
||||
# ----------------------------------------------------------------------------
|
||||
api:
|
||||
env:
|
||||
# Restore (2026-08-29, t_e0e6f7ca): astro-orbiter back online; move LLM back
|
||||
# to local Qwen3.8-27B-Q4_K_M on llama-swap. Nous free tier returns 400
|
||||
# 'missing tags' on Hindsight structured fact-extraction (retain broken).
|
||||
HINDSIGHT_API_LLM_BASE_URL: "http://astro-orbiter:8001/v1"
|
||||
# Cutover to vLLM (t_e6facb19, 2026-08-31): astro-orbiter's llama-swap is
|
||||
# being retired for the completions+embeddings roles Hindsight needs.
|
||||
# vLLM serves Qwen2.5-32B-Instruct-AWQ (:8000, completions) and
|
||||
# nomic-embed-text-v1.5 (:8020, --runner pooling --convert embed) as two
|
||||
# independent systemd-managed processes — validated end-to-end (health,
|
||||
# /v1/chat/completions, /v1/embeddings returning a 768-dim vector) in a
|
||||
# shadow window with llama-swap stopped. Model swap from Qwen3.8-27B to
|
||||
# Qwen2.5-32B-Instruct-AWQ: both are capable instruction-tuned models:
|
||||
# confirm structured-extraction/tagging behavior (the prior stepfun/solar
|
||||
# 400 "missing tags" failure mode) holds up after this swap — see
|
||||
# references/hindsight-backend-fallback-to-local-qwen-workflow.md if it
|
||||
# regresses and llama-swap needs to come back temporarily.
|
||||
HINDSIGHT_API_LLM_BASE_URL: "http://astro-orbiter:8000/v1"
|
||||
HINDSIGHT_API_LLM_PROVIDER: "openai"
|
||||
HINDSIGHT_API_LLM_MODEL: "Qwen3.8-27B-Q4_K_M"
|
||||
HINDSIGHT_API_LLM_MODEL: "Qwen2.5-32B-Instruct-AWQ"
|
||||
# Embeddings: llama.cpp nomic-embed-text-v1.5 (llama-swap, :8001) ->
|
||||
# vLLM nomic-embed-text-v1.5 (:8020). Same model weights, different
|
||||
# serving stack. Hindsight's embeddings provider defaults to "local"
|
||||
# (bundled sentence-transformers) unless overridden — explicit TEI/openai
|
||||
# provider config was never wired for astro-orbiter previously (verified:
|
||||
# `kubectl exec ... env | grep -i embed` showed NO HINDSIGHT_API_EMBEDDINGS_*
|
||||
# vars set, confirming Hindsight was using its bundled local embedder, NOT
|
||||
# astro-orbiter, before this change — despite the OpenViking-style stack
|
||||
# documentation implying otherwise). Wiring the openai-compatible provider
|
||||
# here for the first time to point Hindsight's actual embedding generation
|
||||
# at the vLLM-served nomic-embed-text-v1.5, matching the model OpenViking
|
||||
# already uses (consistency across mk-labs services).
|
||||
HINDSIGHT_API_EMBEDDINGS_PROVIDER: "openai"
|
||||
HINDSIGHT_API_EMBEDDINGS_OPENAI_BASE_URL: "http://astro-orbiter:8020/v1"
|
||||
HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL: "nomic-embed-text-v1.5"
|
||||
# --- t_d7f8cd65: fix 502s on the serial astro-orbiter node ---
|
||||
# astro-orbiter is a single llama-swap process (serial: 1 generate at a
|
||||
# time, ctx 64K). Hindsight's default LLM concurrency is 32, so a retain
|
||||
@@ -74,6 +98,9 @@ api:
|
||||
# is the ONLY LLM endpoint (all ops route there), so cap the whole pool to
|
||||
# 1 and pin retain to 1 as well. The upstream chart exposes these as native
|
||||
# semaphore config (HINDSIGHT_API_*_MAX_CONCURRENT); no code change needed.
|
||||
# Kept at 1 post-cutover: vLLM's single-process-per-model design is also
|
||||
# effectively serial for a single Qwen2.5-32B instance under this GPU's
|
||||
# VRAM budget (KV cache sized for low concurrency at max_model_len=8192).
|
||||
HINDSIGHT_API_LLM_MAX_CONCURRENT: "1"
|
||||
HINDSIGHT_API_RETAIN_LLM_MAX_CONCURRENT: "1"
|
||||
# Client + per-request timeout. Default is 120s; a 29K-token retain runs
|
||||
|
||||
Reference in New Issue
Block a user