llm-router: CPU-offload Coder-14B + Llama-3.1-8B (t_72646029)

- Remove global --n-gpu-layers from router unit ExecStart in preset mode
  (llama.cpp CLI arg outranked per-model INI n-gpu-layers=0; root cause from
  War Machine's run 1). Flag now emitted only in --models-dir mode.
- All 5 preset INI sections carry explicit n-gpu-layers:
  Qwen3.8=99, Phi=99, nomic=99, Coder=0, Llama=0.
- host_vars/astro-orbiter: llm_router_models_max 2 -> 4 so CPU-offloaded
  models count as loaded without LRU-evicting Qwen3.8.
- defaults: llm_router_coder_gpu_layers / llm_router_llama_gpu_layers = 0.
- verify.yml: fix pre-existing .meta attribute crash in router mode.
- New playbook day2_cpu_offload_aux_models.yml.

Deployed + verified on astro-orbiter (gates A-E PASS): concurrent residency
achieved, Qwen3.8 stays GPU-resident. Measured CPU throughput Llama 9.0 /
Coder 4.7 tok/s. VRAM note: llama.cpp 6ea215d allocates ~1.4-1.7GB CUDA-context
per CPU model even at n-gpu-layers=0 -> ~24,004 MiB steady-state, below the
24,576 MiB physical limit. Comments corrected to match the measurement.
Report: friday/inbox/ryan/2026-08-17-llm-cpu-offload-coder-llama-deployed.md
This commit is contained in:
Hermes Agent service account
2026-08-17 17:06:37 -05:00
parent a2994bf55d
commit 03b3ce9dee
6 changed files with 104 additions and 24 deletions

View File

@@ -158,6 +158,12 @@ llm_router_phi_flash_attn: "{{ llm_router_flash_attn }}"
# Qwen2.5-Coder-14B: ctx_size=16384, flash_attn=true per task t_55c164f5
llm_router_coder_ctx_size: 16384
llm_router_coder_flash_attn: "true"
# CPU offload vars (t_72646029, 2026-08-17): n-gpu-layers=0 moves Coder and Llama to
# full CPU inference. Allows concurrent residency with Qwen3.8-27B. NOTE: llama.cpp
# 6ea215d still allocates ~1.4-1.7GB CUDA-context VRAM per CPU model, so steady-state
# is ~24,004 MiB (at the 24,576 MiB physical limit), not the 0-VRAM the spec assumed.
llm_router_coder_gpu_layers: 0
llm_router_llama_gpu_layers: 0
llm_router_preset_path: /opt/llama-server-router-preset.ini
# Qwen3.8-27B: ctx=131072 (128K). Bumped from 32768 -> 131072 per Ryan approval (t_441470b9, 2026-08-16).
# Measured VRAM: 20,282 MiB at 131072 ctx (empirically tested in t_4455a44c); nomic-embed 558 MiB