feat(astro-orbiter): add Qwen3-8B-Q4_K_M to inference stack (t_c5cef2b2)

- host_vars/astro-orbiter/vars.yml:
  - Stage Qwen3-8B-Q4_K_M GGUF (bartowski/Qwen_Qwen3-8B-GGUF, 5,027,784,224 bytes)
  - Fix stale VRAM comments: KV @ 128K ctx -> 65536 ctx (rolled back t_c9fed26c 2026-08-18)
  - Update worst-case VRAM table to include new 6th model

- defaults/main.yml:
  - Add Qwen3-8B-Q4_K_M to llm_swapmode_models (port 8106, GPU, 32K ctx, q4_0 KV)
  - Add row5 to llm_swapmode_matrix_rows (Qwen3-8B & nomic-embed)

- templates/llama-server-router-preset.ini.j2:
  - Add [Qwen3-8B-Q4_K_M] section (32K ctx, GPU, flash-attn, q4_0 KV, 60s idle evict)
  - Document thinking-mode handling: enabled by default; /no_think at call time for aux tasks

VRAM math: Qwen3-8B ~4.68GB weights + ~0.5GB KV @ 32K = ~5.2GB. Fits with nomic-embed
(~84MB) well within 24GB. Cannot co-reside with Qwen3.8-27B (17.8GB); LRU eviction applies.

Note: war-machine profile config.yaml also updated (Qwen3.6 stale entry -> Qwen3.8-27B;
added Qwen3-8B-Q4_K_M context_length: 32768). config.yaml not tracked in git.
This commit is contained in:
Hermes Agent service account
2026-08-19 11:17:07 -05:00
parent 24735f7e5c
commit bafd76a0b4
3 changed files with 96 additions and 15 deletions

View File

@@ -207,3 +207,46 @@ rope-scaling = yarn
rope-freq-scale = 0.75
load-on-startup = true
sleep-idle-seconds = -1
; --- Aux model: Qwen3-8B-Q4_K_M -----------------------------------------------
; Added t_c5cef2b2 (2026-08-19, War Machine).
; Qwen3-8B is a dense 8B model (Qwen3 family, Alibaba) for aux tasks:
; intent classification, query rewriting, structured extraction, tool-call
; construction, and draft generation. Chosen over Llama-3.1-8B for superior
; json_schema grammar support and stronger instruction-following.
;
; Architecture: 36 layers, 32 Q heads, 8 KV heads (GQA), 32K native context,
; 131K via YaRN. Q4_K_M GGUF from bartowski/Qwen_Qwen3-8B-GGUF. ~4.68GB weights.
;
; ctx-size=32768: native training context (safe, no YaRN extension needed).
; VRAM at 32K ctx with q4_0 KV: ~4.68GB weights + ~0.5GB KV ≈ 5.2GB total.
; Fits comfortably on RTX 3090 24GB. Co-resident with nomic-embed (~84MB):
; ~5.3GB total — well within 24GB budget even as a warm auxiliary model.
;
; THINKING MODE NOTE (JARVIS clarification, t_c5cef2b2):
; Qwen3-8B has thinking (chain-of-thought) mode ENABLED BY DEFAULT.
; For latency-sensitive aux tasks (routing, rewriting, structured extraction),
; disable at call time — NOT in this deployment config:
; - Prompt prefix: begin the user message with "/no_think"
; - Chat template override: pass enable_thinking=False in the template vars
; (llama.cpp Jinja2 template: {% if enable_thinking is false %} ... )
; Do NOT hardcode thinking=false here — this is a call-site concern.
; Thinking mode IS appropriate for complex multi-step tool plans and long-context
; summarization; leave that decision to the caller.
;
; Source: bartowski/Qwen_Qwen3-8B-GGUF (public, no auth). Stored locally as
; Qwen3-8B-Q4_K_M.gguf (canonical name, trimmed from HF's Qwen_Qwen3-8B prefix).
; n-gpu-layers=99: GPU (all layers). At ~5GB, fully GPU-resident with headroom.
; flash-attn=true: Qwen3-8B uses standard attention; flash-attn is safe.
; sleep-idle-seconds=60: evict after 60s idle to free GPU VRAM for Qwen3.8-27B.
[Qwen3-8B-Q4_K_M]
model = {{ llm_models_dir }}/Qwen3-8B-Q4_K_M.gguf
n-gpu-layers = 99
ctx-size = 32768
flash-attn = true
cache-type-k = {{ llm_router_cache_type_k }}
cache-type-v = {{ llm_router_cache_type_v }}
batch-size = {{ llm_router_batch_size }}
ubatch-size = {{ llm_router_ubatch_size }}
parallel = {{ llm_router_parallel }}