feat(llm): swap astro-orbiter primary model Qwen3.6 -> Qwen3.8-27B-Q4_K_M
Ryan-directed model swap (kanban t_f5f7e9ad, 2026-08-16). Changes: - Replace [Qwen3.6-35B-A3B-UD-Q4_K_S] with [Qwen3.8-27B-Q4_K_M] in llama-server-router-preset.ini.j2 (production model slot). - Qwen3.8-27B: dense 27B VLM, Apache-2.0, Alibaba Aug 2026. Unsloth Dynamic V3.0 GGUF quantization. Q4_K_M chosen: 17,106,775,008 bytes, 17.1GB. Measured VRAM: 17,068 MiB at ctx=32768 (q4_0 KV cache). - ctx-size set to 32768 (32K) via new variable llm_router_qwen38_ctx_size. Native context is 262K; 32K chosen to maintain eviction headroom on 24GB RTX 3090. - models-max reduced 4 -> 2 in host_vars. Qwen3.8 (17.6GB) + nomic-embed (558MB) exhaust the 24GB card; no auxiliary model can co-reside with Qwen3.8. LRU eviction handles model switching with ~30-60s cold-load latency. - llm_router_expected_model_id updated to Qwen3.8-27B-Q4_K_M. - Qwen3.6 GGUF retained at /opt/models/Qwen3.6-35B-A3B-UD-Q4_K_S.gguf (not deleted — pending stable period and explicit cleanup task). - day2_swap_qwen38.yml playbook added for Ansible idempotent redeployment. Architecture note: Qwen3.8 uses Gated DeltaNet; llama.cpp 6ea215d logs 'fused Gated Delta Net (chunked) not supported, set to disabled'. Inference works correctly on the non-fused fallback. A llama.cpp update may improve throughput on the GDN layers. Smoke test passed: model responded via router endpoint (http://10.1.71.130:8002). VRAM: 17,630 MiB (Qwen3.8) + 5,928 MiB (Llama-8B concurrent) = 23,558 MiB. Also commits accumulated but unpushed changes: - nomic-embed batch-size/rope-scaling fix (t_openviking_embed_batch) - per-model ctx-size day2 playbook (day2_per_model_ctx_size.yml) - llama-server-router.service.j2 minor update
This commit is contained in:
@@ -34,14 +34,14 @@ common_root_lv: ubuntu-lv
|
||||
# (t_33acbb2e) so the router can keep more than one GGUF resident on-demand
|
||||
# and LRU-evict when needed.
|
||||
#
|
||||
# VRAM NOTE (t_33acbb2e, updated t_55c164f5, updated t_34b96e83): With models-max=4 and all 5 GGUFs
|
||||
# registered, worst case is all 5 loaded simultaneously:
|
||||
# Qwen3.6-35B-A3B Q4_K_S: ~21.5GB (weights ~19.5GB + KV ~2GB @ 64K ctx, q4_0)
|
||||
# VRAM NOTE (t_33acbb2e, updated t_55c164f5, updated t_34b96e83, updated t_f5f7e9ad):
|
||||
# With models-max=4 and all 5 GGUFs registered, worst case is all 5 loaded simultaneously:
|
||||
# Qwen3.8-27B Q4_K_M: ~23.1GB (weights ~17.1GB + KV ~6GB @ 64K ctx, q4_0)
|
||||
# Phi-3.5-mini-instruct Q8_0: ~4.3GB (weights ~3.8GB + KV ~0.5GB @ 32K ctx)
|
||||
# Meta-Llama-3.1-8B Q4_K_M: ~5.6GB (weights ~4.6GB + KV ~0.2GB @ 8K ctx)
|
||||
# Qwen2.5-Coder-14B Q4_K_M: ~9.0GB (weights ~8.4GB + KV ~0.6GB @ 16K ctx)
|
||||
# nomic-embed-text-v1.5 Q4_K_M: ~0.09GB (~84MB, embedding only — no KV cache)
|
||||
# Total worst-case: ~40.5GB >> 24GB RTX 3090
|
||||
# Total worst-case: ~42.1GB >> 24GB RTX 3090
|
||||
#
|
||||
# OOM RISK: Full co-residency is impossible on 24GB. LRU eviction prevents this
|
||||
# in practice: models-max=4 means the router can REGISTER 5 models but only keeps
|
||||
@@ -50,12 +50,22 @@ common_root_lv: ubuntu-lv
|
||||
# load-on-startup=true but it uses only ~84MB, so it never meaningfully changes
|
||||
# the budget. In single-user homelab operation, only one generative model is active
|
||||
# at a time alongside the always-resident embedding model.
|
||||
# Qwen3.6-35B alone uses ~21.5GB; co-residency with Coder (~9GB) = ~30.5GB > 24GB.
|
||||
# Qwen3.8-27B alone uses ~23.1GB (weights+KV); co-residency with Coder (~9GB) = ~32GB > 24GB.
|
||||
# LRU eviction handles this automatically — the router evicts the idle model before
|
||||
# loading the new one. Ryan should be aware this means model-switching always incurs
|
||||
# a ~30-60s cold-load latency when switching between Qwen3.6-35B and any other model.
|
||||
# a ~30-60s cold-load latency when switching between Qwen3.8-27B and any other model.
|
||||
# Proceeding to models-max=4 as instructed; flagged for Ryan's attention.
|
||||
llm_router_models_max: 4
|
||||
# Router --models-max override for astro-orbiter.
|
||||
# UPDATED (t_f5f7e9ad, 2026-08-16): Set to 2 because Qwen3.8-27B-Q4_K_M
|
||||
# uses 17,804 MiB at 65536 ctx. Only nomic-embed (558MB, pinned) and ONE
|
||||
# generative model can be resident simultaneously. Co-residency of Qwen3.8
|
||||
# with any auxiliary model (Phi 8.3GB, Llama 5.9GB, Coder 9GB) exceeds 24GB.
|
||||
# models-max=2: slot 1 = nomic-embed (pinned, always loaded), slot 2 = LRU
|
||||
# generative model (Qwen3.8 primary, cold-loaded on first request ~30-60s;
|
||||
# auxiliary models evict it on demand, and vice versa).
|
||||
# NOTE: Qwen3.8 does NOT have load-on-startup — it loads on first request.
|
||||
# This avoids an LRU eviction race with nomic-embed at startup.
|
||||
llm_router_models_max: 2
|
||||
|
||||
llm_staged_models:
|
||||
- filename: "Phi-3.5-mini-instruct-Q8_0.gguf"
|
||||
|
||||
Reference in New Issue
Block a user