REVERT: vLLM cannot be continuously resident alongside llama-swap (t_e6facb19)
Critical finding: flipping vllm_service_enabled/state=true/started and restarting llama-swap alongside it broke llama-swap's ability to load ANY of its own generative models -- every /v1/chat/completions request against Qwen3.8-27B-Q4_K_M or Qwen3-8B aux models failed with 'upstream command exited prematurely' (llama-server OOM at spawn, ~1.8GB free on this 24GB card once vLLM's ~22.8GB was claimed). Confirmed by direct A/B: same request 500s with vLLM running, 200s seconds after stopping it. This breaks 21 Hermes agent profiles' aux-model tasks (skills_hub, approval, mcp, title_generation, profile_describer, compression) plus OpenViking's VLM -- a far larger blast radius than Hindsight's single LLM endpoint. Reverted: - vllm_service_enabled/state back to role defaults (false/stopped) -- vLLM stays staged, startable for a brief validated shadow window, NOT safe to leave resident in production. - Hindsight's HINDSIGHT_API_LLM_BASE_URL back to llama-swap (astro-orbiter:8001, Qwen3.8-27B-Q4_K_M) and the API key secret source back to the Nous fallback item (pre-task state) -- the vLLM cutover, while functionally validated in isolation (health, /v1/chat/completions, and a live hindsight_retain+recall round-trip all succeeded), requires continuous vLLM availability which is now known to be unsafe on this card. Comment posted on t_6dff1ecc: teardown remains correctly blocked -- full cutover is not achievable within this card's VRAM budget as currently scoped. Needs a human decision on aux-model migration strategy (see roles/deploy-vllm README's 'Critical architectural finding' section) before any further progress.
This commit is contained in:
@@ -163,3 +163,26 @@ vllm_models:
|
||||
trust_remote_code: true
|
||||
enabled: true
|
||||
|
||||
# --- deploy-vllm role: boot persistence NOT enabled (t_e6facb19, 2026-08-31) ---
|
||||
# ATTEMPTED enabling vllm_service_enabled/state=started here, then reverted
|
||||
# after a production-breaking discovery: with vLLM's two processes
|
||||
# (Qwen2.5-32B + nomic-embed, ~22.8GB combined) resident, llama-swap could no
|
||||
# longer load ANY of its own generative models -- every /v1/chat/completions
|
||||
# request against Qwen3.8-27B-Q4_K_M or the Qwen3-8B aux models failed with
|
||||
# "upstream command exited prematurely" (llama-server's own OOM at spawn
|
||||
# time, silently swallowed by llama-swap's generic error). Confirmed by
|
||||
# direct A/B: same request 500s with vLLM running, 200s within seconds of
|
||||
# `systemctl stop vllm.service vllm-nomic-embed-text-v1.5.service`. This
|
||||
# breaks all 21 Hermes agent profiles' aux-model tasks (skills_hub, approval,
|
||||
# mcp, title_generation, profile_describer, compression) plus OpenViking's
|
||||
# VLM -- a severe regression, worse than the status quo. Left
|
||||
# vllm_service_enabled/state at role defaults (false/stopped) -- vLLM stays
|
||||
# staged and manually startable for a brief shadow window (same pattern as
|
||||
# t_ca1af9fb's original validation), but is NOT safe to leave resident
|
||||
# alongside llama-swap on this 24GB card. See README's "Known Gaps" section:
|
||||
# full teardown of llama-swap (t_6dff1ecc) is the ONLY path to giving vLLM
|
||||
# permanent residency without starving the other models -- this is not
|
||||
# solvable by tuning gpu_memory_utilization further, the two stacks
|
||||
# together need more VRAM than this card has once both hold real models
|
||||
# resident.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user