llm-router: CPU-offload Coder-14B + Llama-3.1-8B (t_72646029)
- Remove global --n-gpu-layers from router unit ExecStart in preset mode (llama.cpp CLI arg outranked per-model INI n-gpu-layers=0; root cause from War Machine's run 1). Flag now emitted only in --models-dir mode. - All 5 preset INI sections carry explicit n-gpu-layers: Qwen3.8=99, Phi=99, nomic=99, Coder=0, Llama=0. - host_vars/astro-orbiter: llm_router_models_max 2 -> 4 so CPU-offloaded models count as loaded without LRU-evicting Qwen3.8. - defaults: llm_router_coder_gpu_layers / llm_router_llama_gpu_layers = 0. - verify.yml: fix pre-existing .meta attribute crash in router mode. - New playbook day2_cpu_offload_aux_models.yml. Deployed + verified on astro-orbiter (gates A-E PASS): concurrent residency achieved, Qwen3.8 stays GPU-resident. Measured CPU throughput Llama 9.0 / Coder 4.7 tok/s. VRAM note: llama.cpp 6ea215d allocates ~1.4-1.7GB CUDA-context per CPU model even at n-gpu-layers=0 -> ~24,004 MiB steady-state, below the 24,576 MiB physical limit. Comments corrected to match the measurement. Report: friday/inbox/ryan/2026-08-17-llm-cpu-offload-coder-llama-deployed.md
This commit is contained in:
@@ -158,6 +158,12 @@ llm_router_phi_flash_attn: "{{ llm_router_flash_attn }}"
|
||||
# Qwen2.5-Coder-14B: ctx_size=16384, flash_attn=true per task t_55c164f5
|
||||
llm_router_coder_ctx_size: 16384
|
||||
llm_router_coder_flash_attn: "true"
|
||||
# CPU offload vars (t_72646029, 2026-08-17): n-gpu-layers=0 moves Coder and Llama to
|
||||
# full CPU inference. Allows concurrent residency with Qwen3.8-27B. NOTE: llama.cpp
|
||||
# 6ea215d still allocates ~1.4-1.7GB CUDA-context VRAM per CPU model, so steady-state
|
||||
# is ~24,004 MiB (at the 24,576 MiB physical limit), not the 0-VRAM the spec assumed.
|
||||
llm_router_coder_gpu_layers: 0
|
||||
llm_router_llama_gpu_layers: 0
|
||||
llm_router_preset_path: /opt/llama-server-router-preset.ini
|
||||
# Qwen3.8-27B: ctx=131072 (128K). Bumped from 32768 -> 131072 per Ryan approval (t_441470b9, 2026-08-16).
|
||||
# Measured VRAM: 20,282 MiB at 131072 ctx (empirically tested in t_4455a44c); nomic-embed 558 MiB
|
||||
|
||||
Reference in New Issue
Block a user