feat(astro-orbiter): bump Qwen3.8-27B ctx-size 32768->131072 (128K) [t_441470b9]
This commit is contained in:
@@ -58,10 +58,11 @@
|
||||
; KV cache at ctx=65536 (q4_0 KV). Smaller than prior Qwen3.6 at ~20GB.
|
||||
; Native context: 262,144 tokens. Running at 65536 (Hermes floor) for now;
|
||||
; can be raised later if needed.
|
||||
; VRAM footprint (measured 2026-08-16): 17,068 MiB at ctx=32768 with q4_0 KV;
|
||||
; 17,804 MiB at ctx=65536. Using 32768 (32K) to give more eviction headroom
|
||||
; on the 24GB RTX 3090 (nomic-embed 558MB always resident; total ~17.6GB).
|
||||
; Native context is 262,144 tokens; 32K is sufficient for Hermes usage.
|
||||
; VRAM footprint (empirically tested, t_4455a44c 2026-08-16):
|
||||
; ctx=32768: 17,068 MiB; ctx=65536: 17,804 MiB; ctx=131072: 20,282 MiB.
|
||||
; BUMPED to 131072 (128K) per Ryan approval (t_441470b9, 2026-08-16).
|
||||
; nomic-embed always resident at 558 MiB -> total ~20.8GB, ~3.2GB headroom.
|
||||
; Native context is 262,144 tokens; 128K is the production ceiling.
|
||||
; Architecture note: Qwen3.8 uses Gated DeltaNet; llama.cpp 6ea215d logs
|
||||
; "fused Gated Delta Net (chunked) not supported, set to disabled" — falls
|
||||
; back to non-fused implementation. Inference works correctly but may be
|
||||
@@ -73,6 +74,7 @@
|
||||
; them in turn. One cold-load (~30-60s) per switch between Qwen3.8 and
|
||||
; auxiliary models is expected and acceptable. In practice, once Hermes
|
||||
; config.yaml references Qwen3.8 as primary, it stays resident.
|
||||
; ctx-size raised to 131072 (128K) per Ryan approval (t_441470b9, 2026-08-16).
|
||||
; Primary model ID: Qwen3.8-27B-Q4_K_M
|
||||
[Qwen3.8-27B-Q4_K_M]
|
||||
model = {{ llm_models_dir }}/Qwen3.8-27B-Q4_K_M.gguf
|
||||
|
||||
Reference in New Issue
Block a user