Ryan-directed model swap (kanban t_f5f7e9ad, 2026-08-16).
Changes:
- Replace [Qwen3.6-35B-A3B-UD-Q4_K_S] with [Qwen3.8-27B-Q4_K_M] in
llama-server-router-preset.ini.j2 (production model slot).
- Qwen3.8-27B: dense 27B VLM, Apache-2.0, Alibaba Aug 2026.
Unsloth Dynamic V3.0 GGUF quantization.
Q4_K_M chosen: 17,106,775,008 bytes, 17.1GB. Measured VRAM: 17,068 MiB
at ctx=32768 (q4_0 KV cache).
- ctx-size set to 32768 (32K) via new variable llm_router_qwen38_ctx_size.
Native context is 262K; 32K chosen to maintain eviction headroom on 24GB RTX 3090.
- models-max reduced 4 -> 2 in host_vars. Qwen3.8 (17.6GB) + nomic-embed
(558MB) exhaust the 24GB card; no auxiliary model can co-reside with Qwen3.8.
LRU eviction handles model switching with ~30-60s cold-load latency.
- llm_router_expected_model_id updated to Qwen3.8-27B-Q4_K_M.
- Qwen3.6 GGUF retained at /opt/models/Qwen3.6-35B-A3B-UD-Q4_K_S.gguf
(not deleted — pending stable period and explicit cleanup task).
- day2_swap_qwen38.yml playbook added for Ansible idempotent redeployment.
Architecture note: Qwen3.8 uses Gated DeltaNet; llama.cpp 6ea215d logs
'fused Gated Delta Net (chunked) not supported, set to disabled'. Inference
works correctly on the non-fused fallback. A llama.cpp update may improve
throughput on the GDN layers.
Smoke test passed: model responded via router endpoint (http://10.1.71.130:8002).
VRAM: 17,630 MiB (Qwen3.8) + 5,928 MiB (Llama-8B concurrent) = 23,558 MiB.
Also commits accumulated but unpushed changes:
- nomic-embed batch-size/rope-scaling fix (t_openviking_embed_batch)
- per-model ctx-size day2 playbook (day2_per_model_ctx_size.yml)
- llama-server-router.service.j2 minor update
- host_vars/astro-orbiter/vars.yml: add Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf
to llm_staged_models (size_bytes=8988111072, bartowski GGUF public repo).
Updated VRAM note to reflect 4-model roster and LRU eviction semantics.
- defaults/main.yml: add llm_router_coder_ctx_size=16384 and
llm_router_coder_flash_attn=true variables for per-model ctx tuning.
- templates/llama-server-router-preset.ini.j2: add [Qwen2.5-Coder-14B-Instruct-Q4_K_M]
section with alias=Qwen2.5-Coder-14B-Instruct-4bit, ctx-size=16384, flash-attn=true.
- playbooks/day2_add_coder_alias.yml: new playbook that downloads the GGUF (if
absent/mismatched), deploys updated preset INI and systemd unit, restarts
llama-server-router, and verifies all 4 models in /v1/models.
VRAM: Coder ~9GB. Full 4-model co-residency impossible on 24GB — LRU eviction
handles this automatically. Qwen3.6-35B <-> Coder switches incur ~30-60s cold load.
- Add Meta-Llama-3.1-8B-Instruct-4bit alias on Meta-Llama-3.1-8B-Instruct-Q4_K_M
entry in the preset INI (per Ryan dashboard note).
- Document known issue: Phi-3.5-mini json_schema grammar sampler incompatibility
in router mode (GH #23460 variant — chat template token format mismatch).
Meta-Llama works with json_schema response_format; confirmed via live test.
- Phi-3.5-mini-instruct-8bit alias remains working for model routing;
structured output (json_schema) fails due to the model's token format.
