feat(astro-orbiter): add Qwen3-8B-Q4_K_M to inference stack (t_c5cef2b2)
- host_vars/astro-orbiter/vars.yml: - Stage Qwen3-8B-Q4_K_M GGUF (bartowski/Qwen_Qwen3-8B-GGUF, 5,027,784,224 bytes) - Fix stale VRAM comments: KV @ 128K ctx -> 65536 ctx (rolled back t_c9fed26c 2026-08-18) - Update worst-case VRAM table to include new 6th model - defaults/main.yml: - Add Qwen3-8B-Q4_K_M to llm_swapmode_models (port 8106, GPU, 32K ctx, q4_0 KV) - Add row5 to llm_swapmode_matrix_rows (Qwen3-8B & nomic-embed) - templates/llama-server-router-preset.ini.j2: - Add [Qwen3-8B-Q4_K_M] section (32K ctx, GPU, flash-attn, q4_0 KV, 60s idle evict) - Document thinking-mode handling: enabled by default; /no_think at call time for aux tasks VRAM math: Qwen3-8B ~4.68GB weights + ~0.5GB KV @ 32K = ~5.2GB. Fits with nomic-embed (~84MB) well within 24GB. Cannot co-reside with Qwen3.8-27B (17.8GB); LRU eviction applies. Note: war-machine profile config.yaml also updated (Qwen3.6 stale entry -> Qwen3.8-27B; added Qwen3-8B-Q4_K_M context_length: 32768). config.yaml not tracked in git.
This commit is contained in:
@@ -207,3 +207,46 @@ rope-scaling = yarn
|
||||
rope-freq-scale = 0.75
|
||||
load-on-startup = true
|
||||
sleep-idle-seconds = -1
|
||||
|
||||
; --- Aux model: Qwen3-8B-Q4_K_M -----------------------------------------------
|
||||
; Added t_c5cef2b2 (2026-08-19, War Machine).
|
||||
; Qwen3-8B is a dense 8B model (Qwen3 family, Alibaba) for aux tasks:
|
||||
; intent classification, query rewriting, structured extraction, tool-call
|
||||
; construction, and draft generation. Chosen over Llama-3.1-8B for superior
|
||||
; json_schema grammar support and stronger instruction-following.
|
||||
;
|
||||
; Architecture: 36 layers, 32 Q heads, 8 KV heads (GQA), 32K native context,
|
||||
; 131K via YaRN. Q4_K_M GGUF from bartowski/Qwen_Qwen3-8B-GGUF. ~4.68GB weights.
|
||||
;
|
||||
; ctx-size=32768: native training context (safe, no YaRN extension needed).
|
||||
; VRAM at 32K ctx with q4_0 KV: ~4.68GB weights + ~0.5GB KV ≈ 5.2GB total.
|
||||
; Fits comfortably on RTX 3090 24GB. Co-resident with nomic-embed (~84MB):
|
||||
; ~5.3GB total — well within 24GB budget even as a warm auxiliary model.
|
||||
;
|
||||
; THINKING MODE NOTE (JARVIS clarification, t_c5cef2b2):
|
||||
; Qwen3-8B has thinking (chain-of-thought) mode ENABLED BY DEFAULT.
|
||||
; For latency-sensitive aux tasks (routing, rewriting, structured extraction),
|
||||
; disable at call time — NOT in this deployment config:
|
||||
; - Prompt prefix: begin the user message with "/no_think"
|
||||
; - Chat template override: pass enable_thinking=False in the template vars
|
||||
; (llama.cpp Jinja2 template: {% if enable_thinking is false %} ... )
|
||||
; Do NOT hardcode thinking=false here — this is a call-site concern.
|
||||
; Thinking mode IS appropriate for complex multi-step tool plans and long-context
|
||||
; summarization; leave that decision to the caller.
|
||||
;
|
||||
; Source: bartowski/Qwen_Qwen3-8B-GGUF (public, no auth). Stored locally as
|
||||
; Qwen3-8B-Q4_K_M.gguf (canonical name, trimmed from HF's Qwen_Qwen3-8B prefix).
|
||||
; n-gpu-layers=99: GPU (all layers). At ~5GB, fully GPU-resident with headroom.
|
||||
; flash-attn=true: Qwen3-8B uses standard attention; flash-attn is safe.
|
||||
; sleep-idle-seconds=60: evict after 60s idle to free GPU VRAM for Qwen3.8-27B.
|
||||
[Qwen3-8B-Q4_K_M]
|
||||
model = {{ llm_models_dir }}/Qwen3-8B-Q4_K_M.gguf
|
||||
n-gpu-layers = 99
|
||||
ctx-size = 32768
|
||||
flash-attn = true
|
||||
cache-type-k = {{ llm_router_cache_type_k }}
|
||||
cache-type-v = {{ llm_router_cache_type_v }}
|
||||
batch-size = {{ llm_router_batch_size }}
|
||||
ubatch-size = {{ llm_router_ubatch_size }}
|
||||
parallel = {{ llm_router_parallel }}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user