03b3ce9dee
- Remove global --n-gpu-layers from router unit ExecStart in preset mode (llama.cpp CLI arg outranked per-model INI n-gpu-layers=0; root cause from War Machine's run 1). Flag now emitted only in --models-dir mode. - All 5 preset INI sections carry explicit n-gpu-layers: Qwen3.8=99, Phi=99, nomic=99, Coder=0, Llama=0. - host_vars/astro-orbiter: llm_router_models_max 2 -> 4 so CPU-offloaded models count as loaded without LRU-evicting Qwen3.8. - defaults: llm_router_coder_gpu_layers / llm_router_llama_gpu_layers = 0. - verify.yml: fix pre-existing .meta attribute crash in router mode. - New playbook day2_cpu_offload_aux_models.yml. Deployed + verified on astro-orbiter (gates A-E PASS): concurrent residency achieved, Qwen3.8 stays GPU-resident. Measured CPU throughput Llama 9.0 / Coder 4.7 tok/s. VRAM note: llama.cpp 6ea215d allocates ~1.4-1.7GB CUDA-context per CPU model even at n-gpu-layers=0 -> ~24,004 MiB steady-state, below the 24,576 MiB physical limit. Comments corrected to match the measurement. Report: friday/inbox/ryan/2026-08-17-llm-cpu-offload-coder-llama-deployed.md
210 lines
12 KiB
Django/Jinja
210 lines
12 KiB
Django/Jinja
; ------------------------------------------------------------------------------
|
|
; FILE: roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2
|
|
; DESCRIPTION: llama.cpp --models-preset INI for llama-server-router.
|
|
;
|
|
; Purpose: define all router-served GGUFs as named model entries so that:
|
|
; - Each model is explicitly named and configured (no auto-discovery surprises)
|
|
; - Aliases can be added per model (impossible with --models-dir alone)
|
|
; - The Phi-3.5-mini-instruct-Q8_0 entry carries the alias
|
|
; "Phi-3.5-mini-instruct-8bit" — Ryan's Hermes auxiliary.title_generation
|
|
; already references this friendlier name; both names resolve to the same
|
|
; GGUF child process.
|
|
;
|
|
; INI format notes (llama.cpp preset.md):
|
|
; - Section header (e.g. [Phi-3.5-mini-instruct-Q8_0]) is the primary model ID
|
|
; that appears in /v1/models and that clients send in the "model" field.
|
|
; - `alias` adds an ADDITIONAL name — both the section name and the alias work.
|
|
; - The `model` key is the absolute path to the GGUF file.
|
|
; - All other keys map directly to llama-server CLI flags (underscores or hyphens).
|
|
;
|
|
; Known upstream issues (Aug 2026):
|
|
; - GH #22364: --models-preset creates an extra "default" model entry in
|
|
; /v1/models. This is cosmetic — it has no functional effect on model
|
|
; selection by name. Document and move on.
|
|
; - GH #23460: can't pass per-model samplers via --models-preset in router
|
|
; mode. Non-issue: Hermes always sends sampling params in the request body.
|
|
;
|
|
; Added 2026-08-12 (t_9adf0889): Phi alias — War Machine.
|
|
; All per-model settings carry over unchanged from the --models-dir baseline
|
|
; (ctx_size=65536, n_gpu_layers=99, cache=q4_0 for both K and V, models-max=4).
|
|
;
|
|
; UPDATED (t_ryan_per_model_ctx, per Ryan/JARVIS request): Llama-3.1-8B and
|
|
; Phi-3.5-mini now get PER-MODEL ctx-size/flash-attn matched to actual
|
|
; workload instead of the uniform 65536 used by every model previously:
|
|
; - Llama-3.1-8B-Instruct-Q4_K_M: ctx-size 8192 (tool-routing/micro-tasks)
|
|
; - Phi-3.5-mini-instruct-Q8_0: ctx-size 32768 (long web scrapes/logs)
|
|
; Both now request explicit flash-attn=true (was "auto"). Qwen3.6-35B is
|
|
; INTENTIONALLY left untouched at ctx-size 65536 / flash-attn auto — not part
|
|
; of this change. Existing aliases (Meta-Llama-3.1-8B-Instruct-4bit,
|
|
; Phi-3.5-mini-instruct-8bit) are PRESERVED unchanged to avoid breaking live
|
|
; Hermes custom_providers routing — see role README / deployment report for
|
|
; the alias-naming ambiguity flag (Ryan's pasted TOML used different alias
|
|
; strings: "llama-3.1-8b" / "phi-3.5-mini").
|
|
;
|
|
; UPDATED (t_34b96e83, 2026-08-13, per Ryan approval): Added nomic-embed-text-v1.5
|
|
; embedding model. Embedding models fold cleanly into the router preset via
|
|
; embedding=true. No alias needed — clients call it by section name.
|
|
; VRAM estimate ~90MB (negligible). sleep-idle-seconds=-1 keeps it always loaded
|
|
; since embedding calls are latency-sensitive and it costs near-nothing to hold.
|
|
; load-on-startup=true ensures the embedding endpoint is warm at boot without
|
|
; waiting for the first request. — War Machine.
|
|
; ------------------------------------------------------------------------------
|
|
|
|
; --- Production model: Qwen3.8-27B-Q4_K_M ------------------------------------
|
|
; Swapped from Qwen3.6-35B-A3B-UD-Q4_K_S by War Machine (t_f5f7e9ad, 2026-08-16).
|
|
; Ryan-directed swap. Qwen3.8-27B is a dense 27B VLM (Apache-2.0) from Alibaba,
|
|
; released Aug 2026. GGUF quantized by Unsloth Dynamic V3.0 (preview).
|
|
; Q4_K_M chosen: 17.1GB weights — fits RTX 3090 (24GB) with ~7GB headroom for
|
|
; KV cache at ctx=65536 (q4_0 KV). Smaller than prior Qwen3.6 at ~20GB.
|
|
; Native context: 262,144 tokens. Running at 65536 (Hermes floor) for now;
|
|
; can be raised later if needed.
|
|
; VRAM footprint (empirically tested, t_4455a44c 2026-08-16):
|
|
; ctx=32768: 17,068 MiB; ctx=65536: 17,804 MiB; ctx=131072: 20,282 MiB.
|
|
; BUMPED to 131072 (128K) per Ryan approval (t_441470b9, 2026-08-16).
|
|
; nomic-embed always resident at 558 MiB -> total ~20.8GB, ~3.2GB headroom.
|
|
; Native context is 262,144 tokens; 128K is the production ceiling.
|
|
; Architecture note: Qwen3.8 uses Gated DeltaNet; llama.cpp 6ea215d logs
|
|
; "fused Gated Delta Net (chunked) not supported, set to disabled" — falls
|
|
; back to non-fused implementation. Inference works correctly but may be
|
|
; slower on the GDN layers. An updated llama.cpp may improve throughput.
|
|
; load-on-startup NOT set (loads on first request, ~30-60s cold load).
|
|
; n-gpu-layers=99: GPU (all layers). Explicit here so global CLI flag removal
|
|
; (t_72646029, 2026-08-17) does not change Qwen3.8 behavior.
|
|
; Primary model ID: Qwen3.8-27B-Q4_K_M
|
|
[Qwen3.8-27B-Q4_K_M]
|
|
model = {{ llm_models_dir }}/Qwen3.8-27B-Q4_K_M.gguf
|
|
n-gpu-layers = 99
|
|
ctx-size = {{ llm_router_qwen38_ctx_size }}
|
|
cache-type-k = {{ llm_router_cache_type_k }}
|
|
cache-type-v = {{ llm_router_cache_type_v }}
|
|
batch-size = {{ llm_router_batch_size }}
|
|
ubatch-size = {{ llm_router_ubatch_size }}
|
|
parallel = {{ llm_router_parallel }}
|
|
|
|
; --- Auxiliary model: Phi-3.5-mini-instruct-Q8_0 ----------------------------
|
|
; Primary model ID: Phi-3.5-mini-instruct-Q8_0 (unchanged from --models-dir)
|
|
; Alias: Phi-3.5-mini-instruct-8bit (NEW — Ryan's config target)
|
|
; Both names resolve to this GGUF child process.
|
|
; ~3.8GB, auxiliary.title_generation consumer in Ryan's Hermes config.
|
|
; The alias is deployed — "model not found" errors are resolved.
|
|
;
|
|
; KNOWN ISSUE (2026-08-12, t_9adf0889):
|
|
; json_schema response_format fails for Phi-3.5-mini in llama.cpp router
|
|
; mode due to chat template grammar sampler incompatibility (GH #23460).
|
|
; The grammar sampler generates root ::= "assistant|>\n" ... which fails
|
|
; to initialize. This means Hermes auxiliary.title_generation still errors
|
|
; with "HTTP 400: Failed to initialize samplers" when json_schema format
|
|
; is requested. Without response_format (plain text), Phi works fine.
|
|
;
|
|
; Resolution options:
|
|
; a) Update auxiliary.title_generation.model in ~/.hermes/config.yaml to
|
|
; Meta-Llama-3.1-8B-Instruct-Q4_K_M (which supports json_schema) — Ryan
|
|
; needs to approve this config.yaml write (protected file).
|
|
; b) Rebuild llama.cpp from a newer commit if this bug is fixed upstream.
|
|
; c) Accept title generation degradation for Phi-specific structured output.
|
|
[Phi-3.5-mini-instruct-Q8_0]
|
|
model = {{ llm_models_dir }}/Phi-3.5-mini-instruct-Q8_0.gguf
|
|
alias = Phi-3.5-mini-instruct-8bit
|
|
n-gpu-layers = 99
|
|
ctx-size = {{ llm_router_phi_ctx_size }}
|
|
flash-attn = {{ llm_router_phi_flash_attn }}
|
|
cache-type-k = {{ llm_router_cache_type_k }}
|
|
cache-type-v = {{ llm_router_cache_type_v }}
|
|
batch-size = {{ llm_router_batch_size }}
|
|
ubatch-size = {{ llm_router_ubatch_size }}
|
|
parallel = {{ llm_router_parallel }}
|
|
|
|
; --- Auxiliary model: Meta-Llama-3.1-8B-Instruct-Q4_K_M --------------------
|
|
; Primary model ID: Meta-Llama-3.1-8B-Instruct-Q4_K_M (unchanged from --models-dir)
|
|
; Alias: Meta-Llama-3.1-8B-Instruct-4bit (NEW — friendlier name)
|
|
; Both names resolve to this GGUF child process.
|
|
; ~4.6GB, general-purpose small model. Works with json_schema structured output.
|
|
; n-gpu-layers=0 (CPU offload, t_72646029 2026-08-17): Llama moves to full CPU
|
|
; inference to allow concurrent residency with Qwen3.8-27B (which uses ~20.8GB
|
|
; VRAM including nomic-embed). At models-max=4, Llama and Coder run on CPU —
|
|
; llama.cpp 6ea215d still holds ~1.4-1.7GB CUDA-context VRAM per CPU model, so
|
|
; steady-state is ~24,004 MiB (below the 24,576 MiB physical limit).
|
|
[Meta-Llama-3.1-8B-Instruct-Q4_K_M]
|
|
model = {{ llm_models_dir }}/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
|
|
alias = Meta-Llama-3.1-8B-Instruct-4bit
|
|
n-gpu-layers = {{ llm_router_llama_gpu_layers }}
|
|
ctx-size = {{ llm_router_llama_ctx_size }}
|
|
flash-attn = {{ llm_router_llama_flash_attn }}
|
|
cache-type-k = {{ llm_router_cache_type_k }}
|
|
cache-type-v = {{ llm_router_cache_type_v }}
|
|
batch-size = {{ llm_router_batch_size }}
|
|
ubatch-size = {{ llm_router_ubatch_size }}
|
|
parallel = {{ llm_router_parallel }}
|
|
|
|
; --- Coder model: Qwen2.5-Coder-14B-Instruct-Q4_K_M -------------------------
|
|
; Primary model ID: Qwen2.5-Coder-14B-Instruct-Q4_K_M (filename-derived)
|
|
; Alias: Qwen2.5-Coder-14B-Instruct-4bit (friendlier name)
|
|
; Both names resolve to this GGUF child process.
|
|
; ~8.4GB weights + ~0.6GB KV @ 16K ctx = ~9.0GB VRAM (GPU); ~1,390 MiB CUDA ctx (CPU).
|
|
; ctx-size=16384, flash-attn=true per task t_55c164f5 / Ryan's request.
|
|
; Source: bartowski/Qwen2.5-Coder-14B-Instruct-GGUF (public, no auth)
|
|
; Added 2026-08-13 (t_55c164f5) — War Machine.
|
|
; n-gpu-layers=0 (CPU offload, t_72646029 2026-08-17): Coder moves to full CPU
|
|
; inference to allow concurrent residency with Qwen3.8-27B (which uses ~20.8GB
|
|
; VRAM including nomic-embed). At models-max=4, Coder and Llama run on CPU —
|
|
; llama.cpp 6ea215d still holds ~1.4-1.7GB CUDA-context VRAM per CPU model, so
|
|
; steady-state is ~24,004 MiB (below the 24,576 MiB physical limit).
|
|
[Qwen2.5-Coder-14B-Instruct-Q4_K_M]
|
|
model = {{ llm_models_dir }}/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf
|
|
alias = Qwen2.5-Coder-14B-Instruct-4bit
|
|
n-gpu-layers = {{ llm_router_coder_gpu_layers }}
|
|
ctx-size = {{ llm_router_coder_ctx_size }}
|
|
flash-attn = {{ llm_router_coder_flash_attn }}
|
|
cache-type-k = {{ llm_router_cache_type_k }}
|
|
cache-type-v = {{ llm_router_cache_type_v }}
|
|
batch-size = {{ llm_router_batch_size }}
|
|
ubatch-size = {{ llm_router_ubatch_size }}
|
|
parallel = {{ llm_router_parallel }}
|
|
|
|
; --- Embedding model: nomic-embed-text-v1.5 ----------------------------------
|
|
; Primary model ID: nomic-embed-text-v1.5 (section name / client-visible ID)
|
|
; ~84MB GGUF — negligible VRAM, always-loaded. Embedding endpoint: /v1/embeddings.
|
|
; embedding=true: required to expose /v1/embeddings and embed the model (not chat).
|
|
; n-gpu-layers=99: GPU offload all layers (tiny model, no reason to leave on CPU).
|
|
; ctx-size=8192: per task spec (OpenViking Phase 1b, t_34b96e83).
|
|
; load-on-startup=true: warm at boot — embedding callers (peter-parker) are
|
|
; latency-sensitive; no cold-load wait on first request.
|
|
; sleep-idle-seconds=-1: never evict — ~84MB is negligible, always keep hot.
|
|
; NO flash-attn, NO KV cache params: embedding models use a different forward
|
|
; pass (bidirectional, no autoregressive KV cache). These keys are irrelevant
|
|
; for embedding inference and may be silently ignored or cause warnings; omit.
|
|
; Source: nomic-ai/nomic-embed-text-v1.5-GGUF (public, no auth needed)
|
|
; Added 2026-08-13 (t_34b96e83) — War Machine.
|
|
;
|
|
; FIXED (2026-08-14, t_openviking_embed_batch): the original section omitted
|
|
; batch-size/ubatch-size, so llama-server defaulted the PHYSICAL batch
|
|
; (ubatch-size) to 512 tokens. For embedding requests llama.cpp cannot split
|
|
; a single input across ubatches, so any OpenViking chunk over ~512 tokens
|
|
; large chunk over ~512 tokens (observed 2000-3400 tokens/chunk) failed hard with "input (N tokens) is too
|
|
; large to process. increase the physical batch size (current batch size:
|
|
; 512)". This tripped OpenViking's circuit breaker into an infinite
|
|
; fail/re-enqueue loop. Fix: set batch-size/ubatch-size to 4096 (comfortably
|
|
; over the observed max chunk size and under ctx-size=8192).
|
|
;
|
|
; FOLLOW-UP FINDING (2026-08-14, same task): after the batch-size fix landed,
|
|
; logs showed a SECOND, separate problem: llama.cpp capped the effective
|
|
; context to 2048 regardless of ctx-size=8192 ("n_ctx_seq (8192) > n_ctx_train
|
|
; (2048)" / "capping"). This is expected per the nomic-embed-text-v1.5-GGUF
|
|
; model card: the base GGUF's native RoPE training context is 2048; the
|
|
; original HF model reaches its benchmarked 8192-token context via Dynamic
|
|
; NTK-Aware RoPE scaling, which llama.cpp does not implement — so llama.cpp
|
|
; defaults to 2048 unless YaRN scaling is explicitly requested. Model card
|
|
; prescribes: --rope-scaling yarn --rope-freq-scale 0.75 alongside -c 8192.
|
|
; Added rope-scaling/rope-freq-scale below to actually reach 8192.
|
|
[nomic-embed-text-v1.5]
|
|
model = {{ llm_models_dir }}/nomic-embed-text-v1.5-Q4_K_M.gguf
|
|
embedding = true
|
|
n-gpu-layers = 99
|
|
ctx-size = {{ llm_router_nomic_ctx_size }}
|
|
batch-size = {{ llm_router_nomic_batch_size }}
|
|
ubatch-size = {{ llm_router_nomic_ubatch_size }}
|
|
rope-scaling = yarn
|
|
rope-freq-scale = 0.75
|
|
load-on-startup = true
|
|
sleep-idle-seconds = -1
|