feat(llm): swap astro-orbiter primary model Qwen3.6 -> Qwen3.8-27B-Q4_K_M

Ryan-directed model swap (kanban t_f5f7e9ad, 2026-08-16).

Changes:
- Replace [Qwen3.6-35B-A3B-UD-Q4_K_S] with [Qwen3.8-27B-Q4_K_M] in
  llama-server-router-preset.ini.j2 (production model slot).
- Qwen3.8-27B: dense 27B VLM, Apache-2.0, Alibaba Aug 2026.
  Unsloth Dynamic V3.0 GGUF quantization.
  Q4_K_M chosen: 17,106,775,008 bytes, 17.1GB. Measured VRAM: 17,068 MiB
  at ctx=32768 (q4_0 KV cache).
- ctx-size set to 32768 (32K) via new variable llm_router_qwen38_ctx_size.
  Native context is 262K; 32K chosen to maintain eviction headroom on 24GB RTX 3090.
- models-max reduced 4 -> 2 in host_vars. Qwen3.8 (17.6GB) + nomic-embed
  (558MB) exhaust the 24GB card; no auxiliary model can co-reside with Qwen3.8.
  LRU eviction handles model switching with ~30-60s cold-load latency.
- llm_router_expected_model_id updated to Qwen3.8-27B-Q4_K_M.
- Qwen3.6 GGUF retained at /opt/models/Qwen3.6-35B-A3B-UD-Q4_K_S.gguf
  (not deleted — pending stable period and explicit cleanup task).
- day2_swap_qwen38.yml playbook added for Ansible idempotent redeployment.

Architecture note: Qwen3.8 uses Gated DeltaNet; llama.cpp 6ea215d logs
'fused Gated Delta Net (chunked) not supported, set to disabled'. Inference
works correctly on the non-fused fallback. A llama.cpp update may improve
throughput on the GDN layers.

Smoke test passed: model responded via router endpoint (http://10.1.71.130:8002).
VRAM: 17,630 MiB (Qwen3.8) + 5,928 MiB (Llama-8B concurrent) = 23,558 MiB.

Also commits accumulated but unpushed changes:
- nomic-embed batch-size/rope-scaling fix (t_openviking_embed_batch)
- per-model ctx-size day2 playbook (day2_per_model_ctx_size.yml)
- llama-server-router.service.j2 minor update
This commit is contained in:
Hermes Agent service account
2026-08-16 20:40:31 -05:00
parent efaff340a4
commit 7b44a41da3
7 changed files with 445 additions and 37 deletions

View File

@@ -34,14 +34,14 @@ common_root_lv: ubuntu-lv
# (t_33acbb2e) so the router can keep more than one GGUF resident on-demand
# and LRU-evict when needed.
#
# VRAM NOTE (t_33acbb2e, updated t_55c164f5, updated t_34b96e83): With models-max=4 and all 5 GGUFs
# registered, worst case is all 5 loaded simultaneously:
# Qwen3.6-35B-A3B Q4_K_S: ~21.5GB (weights ~19.5GB + KV ~2GB @ 64K ctx, q4_0)
# VRAM NOTE (t_33acbb2e, updated t_55c164f5, updated t_34b96e83, updated t_f5f7e9ad):
# With models-max=4 and all 5 GGUFs registered, worst case is all 5 loaded simultaneously:
# Qwen3.8-27B Q4_K_M: ~23.1GB (weights ~17.1GB + KV ~6GB @ 64K ctx, q4_0)
# Phi-3.5-mini-instruct Q8_0: ~4.3GB (weights ~3.8GB + KV ~0.5GB @ 32K ctx)
# Meta-Llama-3.1-8B Q4_K_M: ~5.6GB (weights ~4.6GB + KV ~0.2GB @ 8K ctx)
# Qwen2.5-Coder-14B Q4_K_M: ~9.0GB (weights ~8.4GB + KV ~0.6GB @ 16K ctx)
# nomic-embed-text-v1.5 Q4_K_M: ~0.09GB (~84MB, embedding only — no KV cache)
# Total worst-case: ~40.5GB >> 24GB RTX 3090
# Total worst-case: ~42.1GB >> 24GB RTX 3090
#
# OOM RISK: Full co-residency is impossible on 24GB. LRU eviction prevents this
# in practice: models-max=4 means the router can REGISTER 5 models but only keeps
@@ -50,12 +50,22 @@ common_root_lv: ubuntu-lv
# load-on-startup=true but it uses only ~84MB, so it never meaningfully changes
# the budget. In single-user homelab operation, only one generative model is active
# at a time alongside the always-resident embedding model.
# Qwen3.6-35B alone uses ~21.5GB; co-residency with Coder (~9GB) = ~30.5GB > 24GB.
# Qwen3.8-27B alone uses ~23.1GB (weights+KV); co-residency with Coder (~9GB) = ~32GB > 24GB.
# LRU eviction handles this automatically — the router evicts the idle model before
# loading the new one. Ryan should be aware this means model-switching always incurs
# a ~30-60s cold-load latency when switching between Qwen3.6-35B and any other model.
# a ~30-60s cold-load latency when switching between Qwen3.8-27B and any other model.
# Proceeding to models-max=4 as instructed; flagged for Ryan's attention.
llm_router_models_max: 4
# Router --models-max override for astro-orbiter.
# UPDATED (t_f5f7e9ad, 2026-08-16): Set to 2 because Qwen3.8-27B-Q4_K_M
# uses 17,804 MiB at 65536 ctx. Only nomic-embed (558MB, pinned) and ONE
# generative model can be resident simultaneously. Co-residency of Qwen3.8
# with any auxiliary model (Phi 8.3GB, Llama 5.9GB, Coder 9GB) exceeds 24GB.
# models-max=2: slot 1 = nomic-embed (pinned, always loaded), slot 2 = LRU
# generative model (Qwen3.8 primary, cold-loaded on first request ~30-60s;
# auxiliary models evict it on demand, and vice versa).
# NOTE: Qwen3.8 does NOT have load-on-startup — it loads on first request.
# This avoids an LRU eviction race with nomic-embed at startup.
llm_router_models_max: 2
llm_staged_models:
- filename: "Phi-3.5-mini-instruct-Q8_0.gguf"