REVERT: vLLM cannot be continuously resident alongside llama-swap (t_e6facb19)

Critical finding: flipping vllm_service_enabled/state=true/started and
restarting llama-swap alongside it broke llama-swap's ability to load
ANY of its own generative models -- every /v1/chat/completions request
against Qwen3.8-27B-Q4_K_M or Qwen3-8B aux models failed with
'upstream command exited prematurely' (llama-server OOM at spawn,
~1.8GB free on this 24GB card once vLLM's ~22.8GB was claimed).
Confirmed by direct A/B: same request 500s with vLLM running, 200s
seconds after stopping it.

This breaks 21 Hermes agent profiles' aux-model tasks (skills_hub,
approval, mcp, title_generation, profile_describer, compression) plus
OpenViking's VLM -- a far larger blast radius than Hindsight's single
LLM endpoint. Reverted:
- vllm_service_enabled/state back to role defaults (false/stopped) --
  vLLM stays staged, startable for a brief validated shadow window,
  NOT safe to leave resident in production.
- Hindsight's HINDSIGHT_API_LLM_BASE_URL back to llama-swap
  (astro-orbiter:8001, Qwen3.8-27B-Q4_K_M) and the API key secret
  source back to the Nous fallback item (pre-task state) --
  the vLLM cutover, while functionally validated in isolation
  (health, /v1/chat/completions, and a live hindsight_retain+recall
  round-trip all succeeded), requires continuous vLLM availability
  which is now known to be unsafe on this card.

Comment posted on t_6dff1ecc: teardown remains correctly blocked --
full cutover is not achievable within this card's VRAM budget as
currently scoped. Needs a human decision on aux-model migration
strategy (see roles/deploy-vllm README's 'Critical architectural
finding' section) before any further progress.
This commit is contained in:
Hermes Agent service account
2026-08-31 18:36:04 -05:00
parent f3a5687adf
commit 1af645d272
4 changed files with 136 additions and 74 deletions

View File

@@ -163,3 +163,26 @@ vllm_models:
trust_remote_code: true
enabled: true
# --- deploy-vllm role: boot persistence NOT enabled (t_e6facb19, 2026-08-31) ---
# ATTEMPTED enabling vllm_service_enabled/state=started here, then reverted
# after a production-breaking discovery: with vLLM's two processes
# (Qwen2.5-32B + nomic-embed, ~22.8GB combined) resident, llama-swap could no
# longer load ANY of its own generative models -- every /v1/chat/completions
# request against Qwen3.8-27B-Q4_K_M or the Qwen3-8B aux models failed with
# "upstream command exited prematurely" (llama-server's own OOM at spawn
# time, silently swallowed by llama-swap's generic error). Confirmed by
# direct A/B: same request 500s with vLLM running, 200s within seconds of
# `systemctl stop vllm.service vllm-nomic-embed-text-v1.5.service`. This
# breaks all 21 Hermes agent profiles' aux-model tasks (skills_hub, approval,
# mcp, title_generation, profile_describer, compression) plus OpenViking's
# VLM -- a severe regression, worse than the status quo. Left
# vllm_service_enabled/state at role defaults (false/stopped) -- vLLM stays
# staged and manually startable for a brief shadow window (same pattern as
# t_ca1af9fb's original validation), but is NOT safe to leave resident
# alongside llama-swap on this 24GB card. See README's "Known Gaps" section:
# full teardown of llama-swap (t_6dff1ecc) is the ONLY path to giving vLLM
# permanent residency without starving the other models -- this is not
# solvable by tuning gpu_memory_utilization further, the two stacks
# together need more VRAM than this card has once both hold real models
# resident.