03b3ce9dee
- Remove global --n-gpu-layers from router unit ExecStart in preset mode (llama.cpp CLI arg outranked per-model INI n-gpu-layers=0; root cause from War Machine's run 1). Flag now emitted only in --models-dir mode. - All 5 preset INI sections carry explicit n-gpu-layers: Qwen3.8=99, Phi=99, nomic=99, Coder=0, Llama=0. - host_vars/astro-orbiter: llm_router_models_max 2 -> 4 so CPU-offloaded models count as loaded without LRU-evicting Qwen3.8. - defaults: llm_router_coder_gpu_layers / llm_router_llama_gpu_layers = 0. - verify.yml: fix pre-existing .meta attribute crash in router mode. - New playbook day2_cpu_offload_aux_models.yml. Deployed + verified on astro-orbiter (gates A-E PASS): concurrent residency achieved, Qwen3.8 stays GPU-resident. Measured CPU throughput Llama 9.0 / Coder 4.7 tok/s. VRAM note: llama.cpp 6ea215d allocates ~1.4-1.7GB CUDA-context per CPU model even at n-gpu-layers=0 -> ~24,004 MiB steady-state, below the 24,576 MiB physical limit. Comments corrected to match the measurement. Report: friday/inbox/ryan/2026-08-17-llm-cpu-offload-coder-llama-deployed.md
60 lines
2.8 KiB
YAML
60 lines
2.8 KiB
YAML
---
|
|
# ------------------------------------------------------------------------------
|
|
# Playbook: day2_cpu_offload_aux_models.yml
|
|
# Purpose: CPU-offload Qwen2.5-Coder-14B and Meta-Llama-3.1-8B on
|
|
# astro-orbiter's production router (port 8002).
|
|
#
|
|
# What this playbook does:
|
|
# 1. Re-renders llama-server-router-preset.ini (Coder + Llama sections now
|
|
# use per-model n-gpu-layers vars = 0 -> full CPU inference).
|
|
# 2. Re-renders the router unit (--models-max now 4 via host_vars, global
|
|
# --n-gpu-layers removed per t_72646029 unit template fix) and restarts
|
|
# llama-server-router so both changes take effect.
|
|
# 3. Verifies per the role's router_preset phase.
|
|
#
|
|
# Context (2026-08-17):
|
|
# - RAM/model-swap audit, TIER 1 (Coder-14B CPU offload) + TIER 2
|
|
# (Llama-3.1-8B CPU offload) — Ryan approved 1 & 2 on 2026-08-17.
|
|
# See inbox/ryan/2026-08-17-llm-system-ram-model-swap.md.
|
|
# - Unit template fix (t_72646029): global --n-gpu-layers removed from
|
|
# ExecStart in preset mode. Each INI section now sets n-gpu-layers
|
|
# explicitly (Qwen3.8=99, Phi=99, nomic=99, Coder=0, Llama=0).
|
|
# - Concurrent residency after change: Qwen3.8-27B (20,302 MiB @ 128K ctx)
|
|
# + nomic-embed (558 MiB, pinned) + Coder (CPU, ~1,390 MiB CUDA ctx) +
|
|
# Llama (CPU, ~1,706 MiB CUDA ctx) = ~24,004 MiB. NOTE: llama.cpp 6ea215d
|
|
# allocates CUDA-context VRAM even at n-gpu-layers=0, so CPU models are not
|
|
# 0-VRAM; total sits at the 24,576 MiB physical limit (headroom ~572 MiB).
|
|
# Qwen3.8 is never evicted for a CPU aux model; Phi-3.5-mini (GPU, 8.3GB)
|
|
# still evicts as before.
|
|
# - CPU speed (8-core Ryzen 7 5800XT): ~5-10 tok/s (14B), ~10-20 tok/s (8B).
|
|
# - Semaphore SSH gap for astro-orbiter still applies (t_730f9584 /
|
|
# t_33acbb2e); running direct CLI Ansible per standing exception.
|
|
#
|
|
# Run:
|
|
# cd /home/hermes/git/homelab/ansible
|
|
# env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
|
|
# -i inventory.yml \
|
|
# playbooks/day2_cpu_offload_aux_models.yml
|
|
#
|
|
# Rollback:
|
|
# git checkout -- \
|
|
# roles/llm-inference-multimodel/templates/llama-server-router.service.j2 \
|
|
# roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2 \
|
|
# roles/llm-inference-multimodel/defaults/main.yml \
|
|
# host_vars/astro-orbiter/vars.yml
|
|
# (restores n-gpu-layers=99 global flag, models-max=2, all GPU)
|
|
# then re-run this playbook to redeploy rollback state.
|
|
# Note: playbooks/day2_cpu_offload_aux_models.yml is untracked — left on disk.
|
|
# ------------------------------------------------------------------------------
|
|
- name: CPU-offload Coder-14B and Llama-3.1-8B on astro-orbiter
|
|
hosts: astro-orbiter
|
|
become: true
|
|
vars:
|
|
llm_router_preset_enabled: true
|
|
llm_router_enabled: true
|
|
llm_router_port: 8002
|
|
|
|
roles:
|
|
- role: llm-inference-multimodel
|
|
tags: [always]
|