feat(astro-orbiter): bump Qwen3.8-27B ctx-size 32768->131072 (128K) [t_441470b9]

This commit is contained in:
Hermes Agent service account
2026-08-16 22:39:12 -05:00
parent 7b44a41da3
commit a2994bf55d
4 changed files with 56 additions and 11 deletions

View File

@@ -58,10 +58,11 @@
; KV cache at ctx=65536 (q4_0 KV). Smaller than prior Qwen3.6 at ~20GB.
; Native context: 262,144 tokens. Running at 65536 (Hermes floor) for now;
; can be raised later if needed.
; VRAM footprint (measured 2026-08-16): 17,068 MiB at ctx=32768 with q4_0 KV;
; 17,804 MiB at ctx=65536. Using 32768 (32K) to give more eviction headroom
; on the 24GB RTX 3090 (nomic-embed 558MB always resident; total ~17.6GB).
; Native context is 262,144 tokens; 32K is sufficient for Hermes usage.
; VRAM footprint (empirically tested, t_4455a44c 2026-08-16):
; ctx=32768: 17,068 MiB; ctx=65536: 17,804 MiB; ctx=131072: 20,282 MiB.
; BUMPED to 131072 (128K) per Ryan approval (t_441470b9, 2026-08-16).
; nomic-embed always resident at 558 MiB -> total ~20.8GB, ~3.2GB headroom.
; Native context is 262,144 tokens; 128K is the production ceiling.
; Architecture note: Qwen3.8 uses Gated DeltaNet; llama.cpp 6ea215d logs
; "fused Gated Delta Net (chunked) not supported, set to disabled" — falls
; back to non-fused implementation. Inference works correctly but may be
@@ -73,6 +74,7 @@
; them in turn. One cold-load (~30-60s) per switch between Qwen3.8 and
; auxiliary models is expected and acceptable. In practice, once Hermes
; config.yaml references Qwen3.8 as primary, it stays resident.
; ctx-size raised to 131072 (128K) per Ryan approval (t_441470b9, 2026-08-16).
; Primary model ID: Qwen3.8-27B-Q4_K_M
[Qwen3.8-27B-Q4_K_M]
model = {{ llm_models_dir }}/Qwen3.8-27B-Q4_K_M.gguf