rblundon/homelab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

llm-router: CPU-offload Coder-14B + Llama-3.1-8B (t_72646029)

Browse Source 
- Remove global --n-gpu-layers from router unit ExecStart in preset mode
  (llama.cpp CLI arg outranked per-model INI n-gpu-layers=0; root cause from
  War Machine's run 1). Flag now emitted only in --models-dir mode.
- All 5 preset INI sections carry explicit n-gpu-layers:
  Qwen3.8=99, Phi=99, nomic=99, Coder=0, Llama=0.
- host_vars/astro-orbiter: llm_router_models_max 2 -> 4 so CPU-offloaded
  models count as loaded without LRU-evicting Qwen3.8.
- defaults: llm_router_coder_gpu_layers / llm_router_llama_gpu_layers = 0.
- verify.yml: fix pre-existing .meta attribute crash in router mode.
- New playbook day2_cpu_offload_aux_models.yml.

Deployed + verified on astro-orbiter (gates A-E PASS): concurrent residency
achieved, Qwen3.8 stays GPU-resident. Measured CPU throughput Llama 9.0 /
Coder 4.7 tok/s. VRAM note: llama.cpp 6ea215d allocates ~1.4-1.7GB CUDA-context
per CPU model even at n-gpu-layers=0 -> ~24,004 MiB steady-state, below the
24,576 MiB physical limit. Comments corrected to match the measurement.
Report: friday/inbox/ryan/2026-08-17-llm-cpu-offload-coder-llama-deployed.md
This commit is contained in:
Hermes Agent service account
2026-08-17 17:06:37 -05:00
parent a2994bf55d
commit 03b3ce9dee
6 changed files with 104 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files

31
ansible/roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2
View File

@@ -68,17 +68,12 @@
; back to non-fused implementation. Inference works correctly but may be
; slower on the GDN layers. An updated llama.cpp may improve throughput.
; load-on-startup NOT set (loads on first request, ~30-60s cold load).
; With models-max=2 in host_vars, nomic-embed occupies slot 1 (pinned),
; and the generative slot (slot 2) is Qwen3.8 on first request. Auxiliary
; models (Phi, Llama, Coder) evict Qwen3.8 when requested; Qwen3.8 evicts
; them in turn. One cold-load (~30-60s) per switch between Qwen3.8 and
; auxiliary models is expected and acceptable. In practice, once Hermes
; config.yaml references Qwen3.8 as primary, it stays resident.
; ctx-size raised to 131072 (128K) per Ryan approval (t_441470b9, 2026-08-16).
; n-gpu-layers=99: GPU (all layers). Explicit here so global CLI flag removal
; (t_72646029, 2026-08-17) does not change Qwen3.8 behavior.
; Primary model ID: Qwen3.8-27B-Q4_K_M
[Qwen3.8-27B-Q4_K_M]
model = {{ llm_models_dir }}/Qwen3.8-27B-Q4_K_M.gguf
n-gpu-layers = {{ llm_router_gpu_layers }}
n-gpu-layers = 99
ctx-size = {{ llm_router_qwen38_ctx_size }}
cache-type-k = {{ llm_router_cache_type_k }}
cache-type-v = {{ llm_router_cache_type_v }}
@@ -110,7 +105,7 @@ parallel = {{ llm_router_parallel }}
[Phi-3.5-mini-instruct-Q8_0]
model = {{ llm_models_dir }}/Phi-3.5-mini-instruct-Q8_0.gguf
alias = Phi-3.5-mini-instruct-8bit
n-gpu-layers = {{ llm_router_gpu_layers }}
n-gpu-layers = 99
ctx-size = {{ llm_router_phi_ctx_size }}
flash-attn = {{ llm_router_phi_flash_attn }}
cache-type-k = {{ llm_router_cache_type_k }}
@@ -124,10 +119,15 @@ parallel = {{ llm_router_parallel }}
; Alias: Meta-Llama-3.1-8B-Instruct-4bit (NEW — friendlier name)
; Both names resolve to this GGUF child process.
; ~4.6GB, general-purpose small model. Works with json_schema structured output.
; n-gpu-layers=0 (CPU offload, t_72646029 2026-08-17): Llama moves to full CPU
; inference to allow concurrent residency with Qwen3.8-27B (which uses ~20.8GB
; VRAM including nomic-embed). At models-max=4, Llama and Coder run on CPU —
; llama.cpp 6ea215d still holds ~1.4-1.7GB CUDA-context VRAM per CPU model, so
; steady-state is ~24,004 MiB (below the 24,576 MiB physical limit).
[Meta-Llama-3.1-8B-Instruct-Q4_K_M]
model = {{ llm_models_dir }}/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
alias = Meta-Llama-3.1-8B-Instruct-4bit
n-gpu-layers = {{ llm_router_gpu_layers }}
n-gpu-layers = {{ llm_router_llama_gpu_layers }}
ctx-size = {{ llm_router_llama_ctx_size }}
flash-attn = {{ llm_router_llama_flash_attn }}
cache-type-k = {{ llm_router_cache_type_k }}
@@ -140,14 +140,19 @@ parallel = {{ llm_router_parallel }}
; Primary model ID: Qwen2.5-Coder-14B-Instruct-Q4_K_M (filename-derived)
; Alias: Qwen2.5-Coder-14B-Instruct-4bit (friendlier name)
; Both names resolve to this GGUF child process.
; ~8.4GB weights + ~0.6GB KV @ 16K ctx = ~9.0GB VRAM.
; ~8.4GB weights + ~0.6GB KV @ 16K ctx = ~9.0GB VRAM (GPU); ~1,390 MiB CUDA ctx (CPU).
; ctx-size=16384, flash-attn=true per task t_55c164f5 / Ryan's request.
; Source: bartowski/Qwen2.5-Coder-14B-Instruct-GGUF (public, no auth)
; Added 2026-08-13 (t_55c164f5) — War Machine.
; n-gpu-layers=0 (CPU offload, t_72646029 2026-08-17): Coder moves to full CPU
; inference to allow concurrent residency with Qwen3.8-27B (which uses ~20.8GB
; VRAM including nomic-embed). At models-max=4, Coder and Llama run on CPU —
; llama.cpp 6ea215d still holds ~1.4-1.7GB CUDA-context VRAM per CPU model, so
; steady-state is ~24,004 MiB (below the 24,576 MiB physical limit).
[Qwen2.5-Coder-14B-Instruct-Q4_K_M]
model = {{ llm_models_dir }}/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf
alias = Qwen2.5-Coder-14B-Instruct-4bit
n-gpu-layers = {{ llm_router_gpu_layers }}
n-gpu-layers = {{ llm_router_coder_gpu_layers }}
ctx-size = {{ llm_router_coder_ctx_size }}
flash-attn = {{ llm_router_coder_flash_attn }}
cache-type-k = {{ llm_router_cache_type_k }}
@@ -194,7 +199,7 @@ parallel = {{ llm_router_parallel }}
[nomic-embed-text-v1.5]
model = {{ llm_models_dir }}/nomic-embed-text-v1.5-Q4_K_M.gguf
embedding = true
n-gpu-layers = {{ llm_router_gpu_layers }}
n-gpu-layers = 99
ctx-size = {{ llm_router_nomic_ctx_size }}
batch-size = {{ llm_router_nomic_batch_size }}
ubatch-size = {{ llm_router_nomic_ubatch_size }}

16
ansible/roles/llm-inference-multimodel/templates/llama-server-router.service.j2
View File

@@ -18,8 +18,8 @@ ExecStart={{ llm_binary_path }} \
--models-max {{ llm_router_models_max }} \
--host {{ llm_router_bind_address }} \
--port {{ llm_router_port }} \
--n-gpu-layers {{ llm_router_gpu_layers }} \
{% if not (llm_router_preset_enabled | default(false)) %}
--n-gpu-layers {{ llm_router_gpu_layers }} \
--ctx-size {{ llm_router_ctx_size }} \
--flash-attn {{ llm_router_flash_attn }} \
{% endif %}
@@ -30,7 +30,7 @@ ExecStart={{ llm_binary_path }} \
--parallel {{ llm_router_parallel }} \
--metrics
# ROUTER MODE NOTES (2026-08-12, t_0cca74a2 / updated t_9adf0889):
# ROUTER MODE NOTES (2026-08-12, t_0cca74a2 / updated t_9adf0889 / updated t_72646029):
# - NO -m/--model flag: this is what enables llama-server router/supervisor mode.
# Without -m, llama-server discovers all .gguf files in --models-dir, or uses
# the per-model definitions in a --models-preset INI file.
@@ -40,12 +40,16 @@ ExecStart={{ llm_binary_path }} \
# The preset INI is at {{ llm_router_preset_path | default('/opt/llama-server-router-preset.ini') }}.
# Both the section name and the alias field in the INI work as model IDs.
# GH #22364 (extra "default" entry in /v1/models) is expected in preset mode — cosmetic.
# - --n-gpu-layers is INTENTIONALLY OMITTED from preset mode (t_72646029, 2026-08-17):
# In --models-preset mode every model section in the INI sets n-gpu-layers explicitly.
# A global CLI --n-gpu-layers has HIGHEST precedence in llama.cpp (CLI > model-section > global-INI)
# and would override per-model INI values (e.g. n-gpu-layers=0 for CPU offload).
# When preset mode is disabled (--models-dir), --n-gpu-layers is emitted normally.
# - --models-max {{ llm_router_models_max }} is driven by llm_router_models_max
# (default 1 in defaults/main.yml; overridden to 4 in host_vars/astro-orbiter
# as of t_33acbb2e after VRAM budget review — see host_vars for OOM risk note).
# Default llama-server cap is 4 simultaneous — OOM on 24GB if all 3 current
# GGUFs load at once. LRU eviction mitigates in practice but review before adding
# models. See host_vars/astro-orbiter/vars.yml for full VRAM breakdown.
# as of t_72646029 after CPU-offload enabling — CPU models count against models-max
# and hold ~1.4-1.7GB CUDA-context VRAM each (llama.cpp 6ea215d allocates it even at
# n-gpu-layers=0); steady-state ~24,004 MiB, below the 24,576 MiB physical limit).
# - Clients select a model via "model": "<section-name-or-alias>" in their
# chat completion request. Hermes sends model: "<id>" on every request already.
# - Cold model load on first request: ~30-60s for Qwen3.6-35B. First response