; ------------------------------------------------------------------------------ ; FILE: roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2 ; DESCRIPTION: llama.cpp --models-preset INI for llama-server-router. ; ; Purpose: define all router-served GGUFs as named model entries so that: ; - Each model is explicitly named and configured (no auto-discovery surprises) ; - Aliases can be added per model (impossible with --models-dir alone) ; - The Phi-3.5-mini-instruct-Q8_0 entry carries the alias ; "Phi-3.5-mini-instruct-8bit" — Ryan's Hermes auxiliary.title_generation ; already references this friendlier name; both names resolve to the same ; GGUF child process. ; ; INI format notes (llama.cpp preset.md): ; - Section header (e.g. [Phi-3.5-mini-instruct-Q8_0]) is the primary model ID ; that appears in /v1/models and that clients send in the "model" field. ; - `alias` adds an ADDITIONAL name — both the section name and the alias work. ; - The `model` key is the absolute path to the GGUF file. ; - All other keys map directly to llama-server CLI flags (underscores or hyphens). ; ; Known upstream issues (Aug 2026): ; - GH #22364: --models-preset creates an extra "default" model entry in ; /v1/models. This is cosmetic — it has no functional effect on model ; selection by name. Document and move on. ; - GH #23460: can't pass per-model samplers via --models-preset in router ; mode. Non-issue: Hermes always sends sampling params in the request body. ; ; Added 2026-08-12 (t_9adf0889): Phi alias — War Machine. ; All per-model settings carry over unchanged from the --models-dir baseline ; (ctx_size=65536, n_gpu_layers=99, cache=q4_0 for both K and V, models-max=4). ; ; UPDATED (t_ryan_per_model_ctx, per Ryan/JARVIS request): Llama-3.1-8B and ; Phi-3.5-mini now get PER-MODEL ctx-size/flash-attn matched to actual ; workload instead of the uniform 65536 used by every model previously: ; - Llama-3.1-8B-Instruct-Q4_K_M: ctx-size 8192 (tool-routing/micro-tasks) ; - Phi-3.5-mini-instruct-Q8_0: ctx-size 32768 (long web scrapes/logs) ; Both now request explicit flash-attn=true (was "auto"). Qwen3.6-35B is ; INTENTIONALLY left untouched at ctx-size 65536 / flash-attn auto — not part ; of this change. Existing aliases (Meta-Llama-3.1-8B-Instruct-4bit, ; Phi-3.5-mini-instruct-8bit) are PRESERVED unchanged to avoid breaking live ; Hermes custom_providers routing — see role README / deployment report for ; the alias-naming ambiguity flag (Ryan's pasted TOML used different alias ; strings: "llama-3.1-8b" / "phi-3.5-mini"). ; ; UPDATED (t_34b96e83, 2026-08-13, per Ryan approval): Added nomic-embed-text-v1.5 ; embedding model. Embedding models fold cleanly into the router preset via ; embedding=true. No alias needed — clients call it by section name. ; VRAM estimate ~90MB (negligible). sleep-idle-seconds=-1 keeps it always loaded ; since embedding calls are latency-sensitive and it costs near-nothing to hold. ; load-on-startup=true ensures the embedding endpoint is warm at boot without ; waiting for the first request. — War Machine. ; ------------------------------------------------------------------------------ ; --- Production model: Qwen3.8-27B-Q4_K_M ------------------------------------ ; Swapped from Qwen3.6-35B-A3B-UD-Q4_K_S by War Machine (t_f5f7e9ad, 2026-08-16). ; Ryan-directed swap. Qwen3.8-27B is a dense 27B VLM (Apache-2.0) from Alibaba, ; released Aug 2026. GGUF quantized by Unsloth Dynamic V3.0 (preview). ; Q4_K_M chosen: 17.1GB weights — fits RTX 3090 (24GB) with ~7GB headroom for ; KV cache at ctx=65536 (q4_0 KV). Smaller than prior Qwen3.6 at ~20GB. ; Native context: 262,144 tokens. Running at 65536 (Hermes floor) for now; ; can be raised later if needed. ; VRAM footprint (empirically tested, t_4455a44c 2026-08-16): ; ctx=32768: 17,068 MiB; ctx=65536: 17,804 MiB; ctx=131072: 20,282 MiB. ; BUMPED to 131072 (128K) per Ryan approval (t_441470b9, 2026-08-16). ; nomic-embed always resident at 558 MiB -> total ~20.8GB, ~3.2GB headroom. ; Native context is 262,144 tokens; 128K is the production ceiling. ; Architecture note: Qwen3.8 uses Gated DeltaNet; llama.cpp 6ea215d logs ; "fused Gated Delta Net (chunked) not supported, set to disabled" — falls ; back to non-fused implementation. Inference works correctly but may be ; slower on the GDN layers. An updated llama.cpp may improve throughput. ; load-on-startup NOT set (loads on first request, ~30-60s cold load). ; n-gpu-layers=99: GPU (all layers). Explicit here so global CLI flag removal ; (t_72646029, 2026-08-17) does not change Qwen3.8 behavior. ; Primary model ID: Qwen3.8-27B-Q4_K_M [Qwen3.8-27B-Q4_K_M] model = {{ llm_models_dir }}/Qwen3.8-27B-Q4_K_M.gguf n-gpu-layers = 99 ctx-size = {{ llm_router_qwen38_ctx_size }} cache-type-k = {{ llm_router_cache_type_k }} cache-type-v = {{ llm_router_cache_type_v }} batch-size = {{ llm_router_batch_size }} ubatch-size = {{ llm_router_ubatch_size }} parallel = {{ llm_router_parallel }} ; --- Auxiliary model: Phi-3.5-mini-instruct-Q8_0 ---------------------------- ; Primary model ID: Phi-3.5-mini-instruct-Q8_0 (unchanged from --models-dir) ; Alias: Phi-3.5-mini-instruct-8bit (NEW — Ryan's config target) ; Both names resolve to this GGUF child process. ; ~3.8GB, auxiliary.title_generation consumer in Ryan's Hermes config. ; The alias is deployed — "model not found" errors are resolved. ; ; KNOWN ISSUE (2026-08-12, t_9adf0889): ; json_schema response_format fails for Phi-3.5-mini in llama.cpp router ; mode due to chat template grammar sampler incompatibility (GH #23460). ; The grammar sampler generates root ::= "assistant|>\n" ... which fails ; to initialize. This means Hermes auxiliary.title_generation still errors ; with "HTTP 400: Failed to initialize samplers" when json_schema format ; is requested. Without response_format (plain text), Phi works fine. ; ; Resolution options: ; a) Update auxiliary.title_generation.model in ~/.hermes/config.yaml to ; Meta-Llama-3.1-8B-Instruct-Q4_K_M (which supports json_schema) — Ryan ; needs to approve this config.yaml write (protected file). ; b) Rebuild llama.cpp from a newer commit if this bug is fixed upstream. ; c) Accept title generation degradation for Phi-specific structured output. [Phi-3.5-mini-instruct-Q8_0] model = {{ llm_models_dir }}/Phi-3.5-mini-instruct-Q8_0.gguf alias = Phi-3.5-mini-instruct-8bit n-gpu-layers = 99 ctx-size = {{ llm_router_phi_ctx_size }} flash-attn = {{ llm_router_phi_flash_attn }} cache-type-k = {{ llm_router_cache_type_k }} cache-type-v = {{ llm_router_cache_type_v }} batch-size = {{ llm_router_batch_size }} ubatch-size = {{ llm_router_ubatch_size }} parallel = {{ llm_router_parallel }} ; --- Auxiliary model: Meta-Llama-3.1-8B-Instruct-Q4_K_M -------------------- ; Primary model ID: Meta-Llama-3.1-8B-Instruct-Q4_K_M (unchanged from --models-dir) ; Alias: Meta-Llama-3.1-8B-Instruct-4bit (NEW — friendlier name) ; Both names resolve to this GGUF child process. ; ~4.6GB, general-purpose small model. Works with json_schema structured output. ; n-gpu-layers=0 (CPU offload, t_72646029 2026-08-17): Llama moves to full CPU ; inference to allow concurrent residency with Qwen3.8-27B (which uses ~20.8GB ; VRAM including nomic-embed). At models-max=4, Llama and Coder run on CPU — ; llama.cpp 6ea215d still holds ~1.4-1.7GB CUDA-context VRAM per CPU model, so ; steady-state is ~24,004 MiB (below the 24,576 MiB physical limit). [Meta-Llama-3.1-8B-Instruct-Q4_K_M] model = {{ llm_models_dir }}/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf alias = Meta-Llama-3.1-8B-Instruct-4bit n-gpu-layers = {{ llm_router_llama_gpu_layers }} ctx-size = {{ llm_router_llama_ctx_size }} flash-attn = {{ llm_router_llama_flash_attn }} cache-type-k = {{ llm_router_cache_type_k }} cache-type-v = {{ llm_router_cache_type_v }} batch-size = {{ llm_router_batch_size }} ubatch-size = {{ llm_router_ubatch_size }} parallel = {{ llm_router_parallel }} ; --- Coder model: Qwen2.5-Coder-14B-Instruct-Q4_K_M ------------------------- ; Primary model ID: Qwen2.5-Coder-14B-Instruct-Q4_K_M (filename-derived) ; Alias: Qwen2.5-Coder-14B-Instruct-4bit (friendlier name) ; Both names resolve to this GGUF child process. ; ~8.4GB weights + ~0.6GB KV @ 16K ctx = ~9.0GB VRAM (GPU); ~1,390 MiB CUDA ctx (CPU). ; ctx-size=16384, flash-attn=true per task t_55c164f5 / Ryan's request. ; Source: bartowski/Qwen2.5-Coder-14B-Instruct-GGUF (public, no auth) ; Added 2026-08-13 (t_55c164f5) — War Machine. ; n-gpu-layers=0 (CPU offload, t_72646029 2026-08-17): Coder moves to full CPU ; inference to allow concurrent residency with Qwen3.8-27B (which uses ~20.8GB ; VRAM including nomic-embed). At models-max=4, Coder and Llama run on CPU — ; llama.cpp 6ea215d still holds ~1.4-1.7GB CUDA-context VRAM per CPU model, so ; steady-state is ~24,004 MiB (below the 24,576 MiB physical limit). [Qwen2.5-Coder-14B-Instruct-Q4_K_M] model = {{ llm_models_dir }}/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf alias = Qwen2.5-Coder-14B-Instruct-4bit n-gpu-layers = {{ llm_router_coder_gpu_layers }} ctx-size = {{ llm_router_coder_ctx_size }} flash-attn = {{ llm_router_coder_flash_attn }} cache-type-k = {{ llm_router_cache_type_k }} cache-type-v = {{ llm_router_cache_type_v }} batch-size = {{ llm_router_batch_size }} ubatch-size = {{ llm_router_ubatch_size }} parallel = {{ llm_router_parallel }} ; --- Embedding model: nomic-embed-text-v1.5 ---------------------------------- ; Primary model ID: nomic-embed-text-v1.5 (section name / client-visible ID) ; ~84MB GGUF — negligible VRAM, always-loaded. Embedding endpoint: /v1/embeddings. ; embedding=true: required to expose /v1/embeddings and embed the model (not chat). ; n-gpu-layers=99: GPU offload all layers (tiny model, no reason to leave on CPU). ; ctx-size=8192: per task spec (OpenViking Phase 1b, t_34b96e83). ; load-on-startup=true: warm at boot — embedding callers (peter-parker) are ; latency-sensitive; no cold-load wait on first request. ; sleep-idle-seconds=-1: never evict — ~84MB is negligible, always keep hot. ; NO flash-attn, NO KV cache params: embedding models use a different forward ; pass (bidirectional, no autoregressive KV cache). These keys are irrelevant ; for embedding inference and may be silently ignored or cause warnings; omit. ; Source: nomic-ai/nomic-embed-text-v1.5-GGUF (public, no auth needed) ; Added 2026-08-13 (t_34b96e83) — War Machine. ; ; FIXED (2026-08-14, t_openviking_embed_batch): the original section omitted ; batch-size/ubatch-size, so llama-server defaulted the PHYSICAL batch ; (ubatch-size) to 512 tokens. For embedding requests llama.cpp cannot split ; a single input across ubatches, so any OpenViking chunk over ~512 tokens ; large chunk over ~512 tokens (observed 2000-3400 tokens/chunk) failed hard with "input (N tokens) is too ; large to process. increase the physical batch size (current batch size: ; 512)". This tripped OpenViking's circuit breaker into an infinite ; fail/re-enqueue loop. Fix: set batch-size/ubatch-size to 4096 (comfortably ; over the observed max chunk size and under ctx-size=8192). ; ; FOLLOW-UP FINDING (2026-08-14, same task): after the batch-size fix landed, ; logs showed a SECOND, separate problem: llama.cpp capped the effective ; context to 2048 regardless of ctx-size=8192 ("n_ctx_seq (8192) > n_ctx_train ; (2048)" / "capping"). This is expected per the nomic-embed-text-v1.5-GGUF ; model card: the base GGUF's native RoPE training context is 2048; the ; original HF model reaches its benchmarked 8192-token context via Dynamic ; NTK-Aware RoPE scaling, which llama.cpp does not implement — so llama.cpp ; defaults to 2048 unless YaRN scaling is explicitly requested. Model card ; prescribes: --rope-scaling yarn --rope-freq-scale 0.75 alongside -c 8192. ; Added rope-scaling/rope-freq-scale below to actually reach 8192. [nomic-embed-text-v1.5] model = {{ llm_models_dir }}/nomic-embed-text-v1.5-Q4_K_M.gguf embedding = true n-gpu-layers = 99 ctx-size = {{ llm_router_nomic_ctx_size }} batch-size = {{ llm_router_nomic_batch_size }} ubatch-size = {{ llm_router_nomic_ubatch_size }} rope-scaling = yarn rope-freq-scale = 0.75 load-on-startup = true sleep-idle-seconds = -1 ; --- Auxiliary model: Qwen3-8B-Q4_K_M (thinking variant) ---------------------- ; GGUF: bartowski/Qwen_Qwen3-8B-GGUF (5,027,784,224 bytes) ; Thinking mode: ON BY DEFAULT (Qwen3 baked-in template, no override). ; - Use for complex aux tasks: long-context summarization, multi-step tool ; planning, structured extraction requiring CoT. ; - To suppress thinking at request time, send /no_think prefix in the prompt ; OR route to [Qwen3-8B-Q4_K_M-no_think] section below. ; n-gpu-layers=99: GPU-resident (~5.2GB VRAM). Cannot co-reside with ; Qwen3.8-27B-Q4_K_M (17.8GB). LRU eviction handles swapping. ; ctx-size=32768: 32K context, q4_0 KV cache. ; flash-attn=true: required for Qwen3 architecture at this context size. ; sleep-idle-seconds=60: evict after 60s idle (free VRAM for primary model). ; Added: t_c5cef2b2 (2026-08-19, War Machine) — initial Qwen3-8B deployment. [Qwen3-8B-Q4_K_M] model = {{ llm_models_dir }}/Qwen3-8B-Q4_K_M.gguf n-gpu-layers = 99 ctx-size = 32768 flash-attn = true cache-type-k = {{ llm_router_cache_type_k }} cache-type-v = {{ llm_router_cache_type_v }} batch-size = {{ llm_router_batch_size }} ubatch-size = {{ llm_router_ubatch_size }} parallel = {{ llm_router_parallel }} sleep-idle-seconds = 60 ; --- Auxiliary model: Qwen3-8B-Q4_K_M (no-think variant) ---------------------- ; Secondary section serving the SAME GGUF with enable_thinking=False via an ; overridden Jinja2 template. This gives low-latency, non-reasoning inference ; for latency-sensitive aux tasks (intent classification, query rewriting, ; structured extraction, tool-call construction, draft generation). ; ; Both sections point to the same .gguf file — llama-server spawns independent ; child processes. VRAM cost: ~5.2GB per instance (~10.4GB total). Fits on ; RTX 3090 24GB alongside nomic-embed (84MB). ; ; Thinking variant (Qwen3-8B-Q4_K_M) remains available for complex tasks that ; benefit from CoT (long-context summarization, multi-step tool planning). ; ; Cannot co-reside with Qwen3.8-27B-Q4_K_M (17.8GB); LRU eviction applies. ; Added: t_664289a0 (2026-08-19, War Machine) — dual thinking deployment. [Qwen3-8B-Q4_K_M-no_think] model = {{ llm_models_dir }}/Qwen3-8B-Q4_K_M.gguf n-gpu-layers = 99 ctx-size = 32768 flash-attn = true cache-type-k = {{ llm_router_cache_type_k }} cache-type-v = {{ llm_router_cache_type_v }} batch-size = {{ llm_router_batch_size }} ubatch-size = {{ llm_router_ubatch_size }} parallel = {{ llm_router_parallel }} chat-template-file = {{ llm_models_dir }}/templates/qwen3-no-think.jinja sleep-idle-seconds = 60