vllm: cutover to permanent residency, retire llama-swap (t_5508360a)

Dashboard decision (t_5508360a): 'stop and disable llama-swap and start
vLLM and its 3 models' -- explicit approval of a breaking change.

Applied:
- llama-swap stopped + disabled on astro-orbiter (systemd unit removed
  from multi-user.target.wants, files left in place -- full teardown is
  t_6dff1ecc, separate task)
- vllm_service_enabled/state flipped to true/started -- vLLM is now the
  permanent, boot-persistent serving layer (was shadow-only/staged)
- Attempted enabling Qwen3-8B-AWQ (the 3rd model) -- does NOT fit.
  Qwen2.5-32B-Instruct-AWQ (~18.6GB) + nomic-embed (~0.8GB) leaves only
  ~1.25GiB free on the 23.55GiB usable budget, below the 3.53GiB floor
  Qwen3-8B-AWQ needs even at gpu_memory_utilization=0.15. Confirmed via
  journalctl: identical ValueError on 7/7 consecutive restart attempts,
  not a transient crash-loop. Reverted Qwen3-8B-AWQ to enabled: false.
  2 of the 3 requested models fit permanently, not 3.

Verified live: /health 200 on both :8000 and :8020, live completion and
live embedding both returned correct real output, NRestarts=0 on both
services after a clean idempotent re-run (changed=0).

ansible/host_vars/astro-orbiter/vars.yml

@@ -152,6 +152,20 @@ vllm_models:
     port: 8010
     max_model_len: 32768
     gpu_memory_utilization: 0.15
+    enforce_eager: true
+    # ATTEMPTED enabled: true (t_5508360a, 2026-08-31) per dashboard's "start
+    # vLLM and its 3 models" instruction -- does NOT fit. With the primary
+    # Qwen2.5-32B-Instruct-AWQ (~18.6GB weights) + nomic-embed (~0.8GB)
+    # resident, only ~1.25GiB free remained on the 23.55GiB usable budget --
+    # short of the 3.53GiB gpu_memory_utilization=0.15 floor requires. This
+    # is not the CUDA-graph-capture transient crash-loop t_e6facb19 saw and
+    # solved with enforce_eager+lower utilization -- confirmed via
+    # `journalctl`: "ValueError: Free memory on device cuda:0 (1.25/23.55
+    # GiB) on startup is less than desired GPU memory utilization" on EVERY
+    # one of 7 consecutive restart attempts, no stabilization. Reverted to
+    # false. The 24GB card fits Qwen2.5-32B-Instruct-AWQ + nomic-embed
+    # (22.6GB) with no room left for a third resident process -- 2 of the
+    # "3 models" fit, not 3. Flagged in kanban t_5508360a completion.
     enabled: false
   - id: "nomic-embed-text-v1.5"
     hf_repo: "nomic-ai/nomic-embed-text-v1.5"
@@ -163,26 +177,23 @@ vllm_models:
     trust_remote_code: true
     enabled: true
 
-# --- deploy-vllm role: boot persistence NOT enabled (t_e6facb19, 2026-08-31) ---
+# --- deploy-vllm role: boot persistence ENABLED (t_5508360a, 2026-08-31) -----
-# ATTEMPTED enabling vllm_service_enabled/state=started here, then reverted
+# Human decision (dashboard, t_5508360a comment thread): "stop and disable
-# after a production-breaking discovery: with vLLM's two processes
+# llama-swap and start vLLM and its 3 models" / "I understand this is a
-# (Qwen2.5-32B + nomic-embed, ~22.8GB combined) resident, llama-swap could no
+# breaking change" -- explicit authorization to give vLLM permanent
-# longer load ANY of its own generative models -- every /v1/chat/completions
+# residency and retire llama-swap, accepting that the 21 Hermes profiles'
-# request against Qwen3.8-27B-Q4_K_M or the Qwen3-8B aux models failed with
+# aux-model tasks that used to have 4 llama-swap aux models to choose from
-# "upstream command exited prematurely" (llama-server's own OOM at spawn
+# (Qwen3-8B, Phi-3.5-mini, Meta-Llama-3.1-8B, Qwen2.5-Coder-14B) now have
-# time, silently swallowed by llama-swap's generic error). Confirmed by
+# exactly ONE local aux model (Qwen3-8B-AWQ via vLLM :8010) -- see
-# direct A/B: same request 500s with vLLM running, 200s within seconds of
+# README.md "Critical architectural finding" section for the full VRAM
-# `systemctl stop vllm.service vllm-nomic-embed-text-v1.5.service`. This
+# collision history this decision resolves.
-# breaks all 21 Hermes agent profiles' aux-model tasks (skills_hub, approval,
+# Qwen3-8B-AWQ flipped enabled: true + enforce_eager: true (same crash-loop
-# mcp, title_generation, profile_describer, compression) plus OpenViking's
+# mitigation applied to the primary model in t_e6facb19 -- avoids CUDA graph
-# VLM -- a severe regression, worse than the status quo. Left
+# capture's VRAM spike now that 3 processes must share the budget instead of
-# vllm_service_enabled/state at role defaults (false/stopped) -- vLLM stays
+# 2). vllm_service_enabled/state flipped to true/started -- this is now the
-# staged and manually startable for a brief shadow window (same pattern as
+# permanent, boot-persistent serving layer. llama-swap is being stopped +
-# t_ca1af9fb's original validation), but is NOT safe to leave resident
+# disabled (not removed yet -- full unit/file teardown tracked separately in
-# alongside llama-swap on this 24GB card. See README's "Known Gaps" section:
+# t_6dff1ecc once this is verified stable).
-# full teardown of llama-swap (t_6dff1ecc) is the ONLY path to giving vLLM
+vllm_service_enabled: true
-# permanent residency without starving the other models -- this is not
+vllm_service_state: started
-# solvable by tuning gpu_memory_utilization further, the two stacks
-# together need more VRAM than this card has once both hold real models
-# resident.