feat(astro-orbiter): add Qwen3-8B-Q4_K_M to inference stack (t_c5cef2b2)
- host_vars/astro-orbiter/vars.yml: - Stage Qwen3-8B-Q4_K_M GGUF (bartowski/Qwen_Qwen3-8B-GGUF, 5,027,784,224 bytes) - Fix stale VRAM comments: KV @ 128K ctx -> 65536 ctx (rolled back t_c9fed26c 2026-08-18) - Update worst-case VRAM table to include new 6th model - defaults/main.yml: - Add Qwen3-8B-Q4_K_M to llm_swapmode_models (port 8106, GPU, 32K ctx, q4_0 KV) - Add row5 to llm_swapmode_matrix_rows (Qwen3-8B & nomic-embed) - templates/llama-server-router-preset.ini.j2: - Add [Qwen3-8B-Q4_K_M] section (32K ctx, GPU, flash-attn, q4_0 KV, 60s idle evict) - Document thinking-mode handling: enabled by default; /no_think at call time for aux tasks VRAM math: Qwen3-8B ~4.68GB weights + ~0.5GB KV @ 32K = ~5.2GB. Fits with nomic-embed (~84MB) well within 24GB. Cannot co-reside with Qwen3.8-27B (17.8GB); LRU eviction applies. Note: war-machine profile config.yaml also updated (Qwen3.6 stale entry -> Qwen3.8-27B; added Qwen3-8B-Q4_K_M context_length: 32768). config.yaml not tracked in git.
This commit is contained in:
@@ -317,6 +317,26 @@ llm_swapmode_models:
|
||||
sleep_idle_seconds: -1 # never idle (always ready for embeddings)
|
||||
load_on_startup: true
|
||||
|
||||
# Added t_c5cef2b2 (2026-08-19, War Machine): Qwen3-8B aux model.
|
||||
# Dense 8B (not MoE). ~4.68GB weights at Q4_K_M. GPU-resident (~5.2GB total
|
||||
# including KV at 32K ctx). Thinking mode on by default — callers MUST send
|
||||
# /no_think prefix for latency-sensitive aux tasks. See INI template comments.
|
||||
# VRAM budget: Qwen3.8-27B (~17,804 MiB @ 65536 ctx) + Qwen3-8B (~5,300 MiB
|
||||
# @ 32K ctx) = ~23,104 MiB — fits in 24GB (24,576 MiB) with ~1.4GB headroom.
|
||||
# llama-swap evicts Qwen3.8 when Qwen3-8B is requested (cannot co-reside).
|
||||
# LRU eviction is the operative strategy; cold load ~5s for 5GB GGUF.
|
||||
- id: Qwen3-8B-Q4_K_M
|
||||
gguf_path: "{{ llm_models_dir }}/Qwen3-8B-Q4_K_M.gguf"
|
||||
port: 8106
|
||||
n_gpu_layers: -1 # -1 = auto-detect / all layers to GPU (~5GB, fits easily)
|
||||
ctx_size: 32768
|
||||
batch_size: 4096
|
||||
ubatch_size: 4096
|
||||
parallel: 1
|
||||
cache_type: q4_0
|
||||
flash_attn: true
|
||||
sleep_idle_seconds: 60 # idle after 60s no requests
|
||||
|
||||
# llama-swap matrix routing configuration
|
||||
# Each row defines a set of models that can be co-resident and hot-swappable
|
||||
# Syntax: "model1 & model2" = both models in same row (via v250 expression DSL)
|
||||
@@ -335,3 +355,6 @@ llm_swapmode_matrix_rows:
|
||||
|
||||
- row: row4
|
||||
expr: "Phi-3.5-mini-instruct-Q8_0 & nomic-embed-text-v1.5" # Mini + embed
|
||||
|
||||
- row: row5
|
||||
expr: "Qwen3-8B-Q4_K_M & nomic-embed-text-v1.5" # Aux 8B + embed
|
||||
|
||||
Reference in New Issue
Block a user