REVERT: vLLM cannot be continuously resident alongside llama-swap (t_e6facb19)
Critical finding: flipping vllm_service_enabled/state=true/started and restarting llama-swap alongside it broke llama-swap's ability to load ANY of its own generative models -- every /v1/chat/completions request against Qwen3.8-27B-Q4_K_M or Qwen3-8B aux models failed with 'upstream command exited prematurely' (llama-server OOM at spawn, ~1.8GB free on this 24GB card once vLLM's ~22.8GB was claimed). Confirmed by direct A/B: same request 500s with vLLM running, 200s seconds after stopping it. This breaks 21 Hermes agent profiles' aux-model tasks (skills_hub, approval, mcp, title_generation, profile_describer, compression) plus OpenViking's VLM -- a far larger blast radius than Hindsight's single LLM endpoint. Reverted: - vllm_service_enabled/state back to role defaults (false/stopped) -- vLLM stays staged, startable for a brief validated shadow window, NOT safe to leave resident in production. - Hindsight's HINDSIGHT_API_LLM_BASE_URL back to llama-swap (astro-orbiter:8001, Qwen3.8-27B-Q4_K_M) and the API key secret source back to the Nous fallback item (pre-task state) -- the vLLM cutover, while functionally validated in isolation (health, /v1/chat/completions, and a live hindsight_retain+recall round-trip all succeeded), requires continuous vLLM availability which is now known to be unsafe on this card. Comment posted on t_6dff1ecc: teardown remains correctly blocked -- full cutover is not achievable within this card's VRAM budget as currently scoped. Needs a human decision on aux-model migration strategy (see roles/deploy-vllm README's 'Critical architectural finding' section) before any further progress.
This commit is contained in:
Hermes Agent service account
@@ -163,3 +163,26 @@ vllm_models:
|
||||
trust_remote_code: true
|
||||
enabled: true
|
||||
|
||||
# --- deploy-vllm role: boot persistence NOT enabled (t_e6facb19, 2026-08-31) ---
|
||||
# ATTEMPTED enabling vllm_service_enabled/state=started here, then reverted
|
||||
# after a production-breaking discovery: with vLLM's two processes
|
||||
# (Qwen2.5-32B + nomic-embed, ~22.8GB combined) resident, llama-swap could no
|
||||
# longer load ANY of its own generative models -- every /v1/chat/completions
|
||||
# request against Qwen3.8-27B-Q4_K_M or the Qwen3-8B aux models failed with
|
||||
# "upstream command exited prematurely" (llama-server's own OOM at spawn
|
||||
# time, silently swallowed by llama-swap's generic error). Confirmed by
|
||||
# direct A/B: same request 500s with vLLM running, 200s within seconds of
|
||||
# `systemctl stop vllm.service vllm-nomic-embed-text-v1.5.service`. This
|
||||
# breaks all 21 Hermes agent profiles' aux-model tasks (skills_hub, approval,
|
||||
# mcp, title_generation, profile_describer, compression) plus OpenViking's
|
||||
# VLM -- a severe regression, worse than the status quo. Left
|
||||
# vllm_service_enabled/state at role defaults (false/stopped) -- vLLM stays
|
||||
# staged and manually startable for a brief shadow window (same pattern as
|
||||
# t_ca1af9fb's original validation), but is NOT safe to leave resident
|
||||
# alongside llama-swap on this 24GB card. See README's "Known Gaps" section:
|
||||
# full teardown of llama-swap (t_6dff1ecc) is the ONLY path to giving vLLM
|
||||
# permanent residency without starving the other models -- this is not
|
||||
# solvable by tuning gpu_memory_utilization further, the two stacks
|
||||
# together need more VRAM than this card has once both hold real models
|
||||
# resident.
|
||||
|
||||
|
||||
@@ -89,25 +89,12 @@ supports (`--quantization awq`) and which fit the VRAM budget:
|
||||
- `vllm_service_enabled` defaults to `false` deliberately — see "Deliberate
|
||||
staging-first default" above. Flip together with the cutover step, not
|
||||
before.
|
||||
- **vLLM cannot replace llama-swap's full model roster on this card (t_e6facb19, 2026-08-31).**
|
||||
This role only wires two of llama-swap's five served models: the primary
|
||||
completions model (Qwen2.5-32B-Instruct-AWQ, replacing Qwen3.8-27B) and
|
||||
the embedding model (nomic-embed-text-v1.5). llama-swap ALSO serves
|
||||
Qwen3-8B-Q4_K_M(-no_think), Phi-3.5-mini-instruct-Q8_0,
|
||||
Meta-Llama-3.1-8B-Instruct-Q4_K_M, and Qwen2.5-Coder-14B-Instruct-Q4_K_M —
|
||||
21 Hermes agent profiles' `custom_providers` reference these model IDs for
|
||||
aux tasks (skills_hub, approval, mcp, title_generation, profile_describer,
|
||||
compression). vLLM 0.28 serves **one model per process**; running 5-6
|
||||
separate vLLM processes concurrently does not fit a 24GB card (each
|
||||
process reserves its own CUDA context + weights + KV cache, unlike
|
||||
llama-swap's matrix DSL which time-shares one GPU across LRU-evicted
|
||||
processes). **Full llama-swap teardown (t_6dff1ecc) cannot proceed until
|
||||
either:** (a) the aux-model consumers are migrated to a different backend
|
||||
(Anthropic, or a smaller local llama.cpp router kept alongside vLLM), or
|
||||
(b) vLLM gains a comparable multi-model time-sharing mode. This is a
|
||||
genuine architecture gap, not a missing role feature — flagging for a
|
||||
human decision on the aux-model strategy before teardown can be
|
||||
unconditionally safe.
|
||||
- **vLLM cannot replace llama-swap's full model roster on this card — see
|
||||
"Critical architectural finding" section below for the full incident.**
|
||||
Short version: vLLM's one-model-per-process design plus llama-swap's own
|
||||
VRAM needs exceed this 24GB card's capacity when both must serve real
|
||||
models simultaneously. Full llama-swap teardown (t_6dff1ecc) cannot
|
||||
proceed until a human decides the aux-model + VRAM strategy.
|
||||
|
||||
## Embedding-mode support (t_e6facb19, 2026-08-31)
|
||||
|
||||
@@ -162,32 +149,87 @@ crash-looped repeatedly:
|
||||
|
||||
## Consumer cutover status (t_e6facb19, 2026-08-31)
|
||||
|
||||
**Cut over (validated end-to-end):**
|
||||
- Hindsight (`cluster/applications/hindsight/values.yaml` +
|
||||
`externalsecret.yaml`): `HINDSIGHT_API_LLM_BASE_URL` → vLLM `:8000`
|
||||
(Qwen2.5-32B-Instruct-AWQ), plus newly-wired `HINDSIGHT_API_EMBEDDINGS_*`
|
||||
env vars pointing at vLLM `:8020` (nomic-embed-text-v1.5). Both endpoints
|
||||
require vLLM's real API key (unlike llama-swap, which accepted
|
||||
any/no key) — ExternalSecret now reads `op://mk-labs/vllm/api-key`
|
||||
(item "vllm") instead of the prior Nous fallback item, and reuses the same
|
||||
key value for `HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY` (both vLLM
|
||||
endpoints share one key file per `tasks/api-key.yml`).
|
||||
**Attempted, then REVERTED — Hindsight LLM cutover.** Hindsight's
|
||||
`HINDSIGHT_API_LLM_BASE_URL` was pointed at vLLM `:8000`
|
||||
(Qwen2.5-32B-Instruct-AWQ) and validated working in isolation: health,
|
||||
`/v1/chat/completions`, and a live `hindsight_retain` + recall round-trip
|
||||
all succeeded (after also fixing `HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS`,
|
||||
which defaulted to 64000 — exceeding vLLM's `max_model_len=8192` — down to
|
||||
4096). **Reverted anyway**, because of a severe discovery documented in the
|
||||
next section: vLLM cannot stay resident on this card without starving
|
||||
llama-swap, and Hindsight's LLM endpoint needs continuous availability, not
|
||||
just a validation window. Restored to `http://astro-orbiter:8001/v1`
|
||||
(llama-swap, Qwen3.8-27B-Q4_K_M) — the pre-task working state.
|
||||
|
||||
**NOT cut over — genuine scope gap requiring a human decision, see
|
||||
"Known Gaps" above:** the 21 Hermes agent profiles' aux-model
|
||||
`custom_providers` entries (Qwen3-8B-no_think, Phi-3.5-mini, Meta-Llama-3.1-8B,
|
||||
Qwen2.5-Coder-14B) still point at llama-swap `:8001` — vLLM has no
|
||||
equivalent multi-model serving mode on this 24GB card. llama-swap MUST stay
|
||||
up to serve these until that gap is resolved. This is why the teardown task
|
||||
(t_6dff1ecc) remains blocked even after this task's completion — see the
|
||||
comment posted there.
|
||||
**NOT cut over — embeddings.** Hindsight was discovered to have NEVER used
|
||||
astro-orbiter for embeddings — it defaults to a bundled local
|
||||
`BAAI/bge-small-en-v1.5` (384-dim) embedder whenever
|
||||
`HINDSIGHT_API_EMBEDDINGS_PROVIDER` is unset, which was always the case here.
|
||||
Pointing it at vLLM's `nomic-embed-text-v1.5` (768-dim) crash-looped the pod:
|
||||
`RuntimeError: Cannot change embedding dimension from 384 to 768:
|
||||
memory_units table contains 1289 rows with embeddings.` Re-embedding all
|
||||
existing memory data across ~20 agent banks is destructive and irreversible
|
||||
— reverted immediately, left as a separate, explicitly-approved future task.
|
||||
|
||||
- OpenViking (`cluster/platform/openviking/values.yaml`): still points at
|
||||
llama-swap `:8001` (`Meta-Llama-3.1-8B-Instruct-Q4_K_M` VLM + nomic-embed
|
||||
for dense embeddings). Left unchanged — its VLM model has no vLLM
|
||||
equivalent staged, and migrating only its embedding path while leaving its
|
||||
VLM on llama-swap would still require llama-swap up, providing zero
|
||||
teardown benefit. Flagged, not touched, per the same aux-model gap above.
|
||||
**NOT cut over — 21 Hermes agent profiles' aux models + OpenViking VLM.**
|
||||
See "Critical architectural finding" below — this was never attempted once
|
||||
the VRAM collision was discovered, would have made things categorically
|
||||
worse.
|
||||
|
||||
## Critical architectural finding: vLLM CANNOT be continuously resident alongside llama-swap on this 24GB card (t_e6facb19, 2026-08-31)
|
||||
|
||||
After validating vLLM's two processes (Qwen2.5-32B-Instruct-AWQ + nomic-embed-
|
||||
text-v1.5, ~22.8GB combined) work correctly in isolation, this role's
|
||||
`vllm_service_enabled`/`vllm_service_state` were flipped to `true`/`started`
|
||||
as host_vars overrides to make the deployment permanent (per the task's
|
||||
"enable for boot" requirement) — llama-swap was then restarted alongside
|
||||
vLLM to preserve its own consumers. **Result: llama-swap could no longer
|
||||
load ANY of its own generative models.** Every `/v1/chat/completions`
|
||||
request against `Qwen3.8-27B-Q4_K_M` or the `Qwen3-8B` aux models failed
|
||||
with `{"error":"unspecific error: upstream command exited prematurely",
|
||||
"src":"llama-swap"}` — llama-server's own OOM at spawn time, only ~1.8GB
|
||||
free on a 24GB card once vLLM's ~22.8GB was already claimed.
|
||||
|
||||
**Confirmed by direct A/B test, not inference:** identical
|
||||
`Qwen3.8-27B-Q4_K_M` chat completion request returned HTTP 500 with vLLM's
|
||||
two processes running, then HTTP 200 with a real completion within seconds
|
||||
of `systemctl stop vllm.service vllm-nomic-embed-text-v1.5.service` — same
|
||||
llama-swap process, same request, only the GPU memory pressure changed.
|
||||
|
||||
**This is a hard architectural collision, not a tunable-parameter problem.**
|
||||
llama-swap needs ~18-20GB for its own primary model (Qwen3.8-27B-Q4_K_M);
|
||||
vLLM's two processes need ~22.8GB even with `enforce_eager` and a lowered
|
||||
`gpu_memory_utilization`. The two together need more VRAM than a 24GB card
|
||||
has once both hold real models resident — there is no `gpu_memory_utilization`
|
||||
value that resolves this while both stacks serve real production models
|
||||
simultaneously.
|
||||
|
||||
**Consequence — reverted the boot-persistence flip.** `vllm_service_enabled`
|
||||
and `vllm_service_state` are back to role defaults (`false`/`stopped`) in
|
||||
`host_vars/astro-orbiter/vars.yml`. vLLM stays staged (venv, model weights,
|
||||
systemd units all in place) and can be started for a brief shadow-validation
|
||||
window (same pattern as t_ca1af9fb's original Phase 5), but is NOT safe to
|
||||
leave resident in production alongside llama-swap.
|
||||
|
||||
**Path forward — requires a human decision, not more role tuning:**
|
||||
1. Full llama-swap teardown (t_6dff1ecc) BEFORE vLLM gets permanent
|
||||
residency — but that breaks the 21 agent profiles' aux-model tasks and
|
||||
OpenViking's VLM unless those consumers are migrated to a different
|
||||
backend first (Anthropic API, a second smaller local box, or a
|
||||
redesigned single-process serving strategy that covers all the models
|
||||
vLLM and llama-swap currently split between them).
|
||||
2. Accept vLLM as a shadow-only / on-demand stack (manually started for
|
||||
specific validated windows, stopped otherwise) and do NOT attempt
|
||||
permanent Hindsight cutover — keeps llama-swap as the sole continuous
|
||||
production serving layer, matching the pre-task state.
|
||||
3. A hardware change (larger GPU, or a second GPU) — out of scope for this
|
||||
task, flagging for Ryan's awareness if the aux-model consumer set is
|
||||
expected to grow.
|
||||
|
||||
Comment posted on t_6dff1ecc with this finding — the teardown task remains
|
||||
correctly blocked; this task's completion does NOT unblock it, because full
|
||||
cutover to vLLM is not achievable within this card's VRAM budget as
|
||||
currently scoped.
|
||||
|
||||
|
||||
## Validation Log (2026-08-31, t_ca1af9fb)
|
||||
|
||||
@@ -75,15 +75,15 @@ spec:
|
||||
remoteRef:
|
||||
key: hindsight
|
||||
property: HINDSIGHT_API_MCP_AUTH_TOKEN
|
||||
# Cutover to vLLM (t_e6facb19, 2026-08-31): astro-orbiter's llama-swap
|
||||
# (which needed no real auth) is being retired for Hindsight's
|
||||
# completions+embeddings roles. vLLM enforces a real API key — read the
|
||||
# SAME key roles/deploy-vllm's api-key.yml phase writes to
|
||||
# /etc/vllm/api-key.env on astro-orbiter, sourced from 1Password
|
||||
# op://mk-labs/vllm/api-key (item "vllm", field "api-key", vault mk-labs).
|
||||
# Replaces the prior "nous" item's api-key (Nous free-tier fallback,
|
||||
# no longer the active LLM backend once this cutover lands).
|
||||
# 1Password item "nous" (vault mk-labs), field "api-key" (Ryan-provisioned,
|
||||
# decision 4). REVERTED to this source (t_e6facb19, 2026-08-31) after the
|
||||
# vLLM cutover attempt was reverted -- llama-swap doesn't validate the
|
||||
# API key at all, so this value is effectively unused by the live LLM
|
||||
# path, but restoring the original source keeps this file's intent
|
||||
# honest (Nous fallback item, not vLLM's real auth key) until a real
|
||||
# vLLM cutover is safe to attempt again. See values.yaml's api.env
|
||||
# comment for the full incident writeup.
|
||||
- secretKey: HINDSIGHT_API_LLM_API_KEY
|
||||
remoteRef:
|
||||
key: vllm
|
||||
key: nous
|
||||
property: api-key
|
||||
|
||||
@@ -60,29 +60,26 @@ existingSecret: hindsight-credentials
|
||||
# ----------------------------------------------------------------------------
|
||||
api:
|
||||
env:
|
||||
# Cutover to vLLM (t_e6facb19, 2026-08-31): astro-orbiter's llama-swap is
|
||||
# being retired for the completions role Hindsight needs. vLLM serves
|
||||
# Qwen2.5-32B-Instruct-AWQ (:8000, completions) — validated end-to-end
|
||||
# (health, /v1/chat/completions) in a shadow window with llama-swap
|
||||
# stopped. Model swap from Qwen3.8-27B to Qwen2.5-32B-Instruct-AWQ: both
|
||||
# are capable instruction-tuned models; confirm structured-extraction/
|
||||
# tagging behavior (the prior stepfun/solar 400 "missing tags" failure
|
||||
# mode) holds up after this swap — see
|
||||
# references/hindsight-backend-fallback-to-local-qwen-workflow.md if it
|
||||
# regresses and llama-swap needs to come back temporarily.
|
||||
HINDSIGHT_API_LLM_BASE_URL: "http://astro-orbiter:8000/v1"
|
||||
# REVERTED (t_e6facb19, 2026-08-31) -- see full incident writeup below.
|
||||
# Attempted cutover to vLLM (:8000, Qwen2.5-32B-Instruct-AWQ), validated
|
||||
# working in isolation (health, /v1/chat/completions, and a live
|
||||
# hindsight_retain + recall round-trip all succeeded), but reverted
|
||||
# after discovering vLLM cannot safely stay resident on this 24GB card
|
||||
# AT THE SAME TIME as llama-swap needs to serve its own generative
|
||||
# models: with vLLM's two processes loaded, every llama-swap
|
||||
# /v1/chat/completions request (Qwen3.8-27B, Qwen3-8B aux models) failed
|
||||
# with "upstream command exited prematurely" (an OOM at llama-server
|
||||
# spawn time). That would break the 21 Hermes agent profiles' aux-model
|
||||
# tasks and OpenViking's VLM -- a much larger blast radius than
|
||||
# Hindsight alone. Since Hindsight's LLM endpoint must be reachable
|
||||
# continuously (not just in a brief validation window), and vLLM cannot
|
||||
# be continuously resident without starving llama-swap, pointing
|
||||
# Hindsight at vLLM is unsafe until a real architectural fix lands (see
|
||||
# roles/deploy-vllm README "Known Gaps"). Restored to llama-swap
|
||||
# (astro-orbiter:8001, Qwen3.8-27B-Q4_K_M) -- the pre-task working state.
|
||||
HINDSIGHT_API_LLM_BASE_URL: "http://astro-orbiter:8001/v1"
|
||||
HINDSIGHT_API_LLM_PROVIDER: "openai"
|
||||
HINDSIGHT_API_LLM_MODEL: "Qwen2.5-32B-Instruct-AWQ"
|
||||
# vLLM's Qwen2.5-32B-Instruct-AWQ is deployed with --max-model-len 8192
|
||||
# (roles/deploy-vllm/defaults/main.yml vllm_max_model_len; llama-swap's
|
||||
# Qwen3.8-27B ran at ctx=65536, matching Hindsight's default
|
||||
# RETAIN_MAX_COMPLETION_TOKENS=64000). Discovered via a live retain 500:
|
||||
# "max_tokens=64000 cannot be greater than max_model_len=8192" — every
|
||||
# retain call was requesting a 64K-token completion budget the new
|
||||
# backend physically cannot serve. Lowered to 4096, comfortably under
|
||||
# 8192 with room for the ~3000-char (RETAIN_CHUNK_SIZE) input chunk plus
|
||||
# prompt overhead. Verified with a live hindsight_retain call post-fix.
|
||||
HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS: "4096"
|
||||
HINDSIGHT_API_LLM_MODEL: "Qwen3.8-27B-Q4_K_M"
|
||||
# DO NOT set HINDSIGHT_API_EMBEDDINGS_* here (t_e6facb19, 2026-08-31
|
||||
# attempted this, reverted after a production incident — see below).
|
||||
#
|
||||
|
||||
Reference in New Issue
Block a user