7aea88724f
- host_vars/astro-orbiter/vars.yml: add Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf to llm_staged_models (size_bytes=8988111072, bartowski GGUF public repo). Updated VRAM note to reflect 4-model roster and LRU eviction semantics. - defaults/main.yml: add llm_router_coder_ctx_size=16384 and llm_router_coder_flash_attn=true variables for per-model ctx tuning. - templates/llama-server-router-preset.ini.j2: add [Qwen2.5-Coder-14B-Instruct-Q4_K_M] section with alias=Qwen2.5-Coder-14B-Instruct-4bit, ctx-size=16384, flash-attn=true. - playbooks/day2_add_coder_alias.yml: new playbook that downloads the GGUF (if absent/mismatched), deploys updated preset INI and systemd unit, restarts llama-server-router, and verifies all 4 models in /v1/models. VRAM: Coder ~9GB. Full 4-model co-residency impossible on 24GB — LRU eviction handles this automatically. Qwen3.6-35B <-> Coder switches incur ~30-60s cold load.
72 lines
4.0 KiB
YAML
72 lines
4.0 KiB
YAML
---
|
|
# ------------------------------------------------------------------------------
|
|
# FILE: ansible/host_vars/astro_orbiter/vars.yml
|
|
# HOST: astro-orbiter (10.1.71.130)
|
|
# ROLE: llama.cpp LLM inference host — Ryzen 7 5800XT / RTX 3090 (ATX rebuild,
|
|
# 2026-08-04). Superseded the prior AMD RX 5700 / Ollama config below;
|
|
# drive was transplanted into new hardware, not reinstalled.
|
|
# ------------------------------------------------------------------------------
|
|
|
|
ansible_host: 10.1.71.130
|
|
ansible_user: jarvis
|
|
ansible_ssh_private_key_file: ~/.ssh/id_jarvis
|
|
ansible_become: true
|
|
|
|
# LVM root expansion — xlarge template uses sda3 partition, standard VG/LV names
|
|
common_expand_root_lvm: true
|
|
common_root_pv: /dev/sda3
|
|
common_root_vg: ubuntu-vg
|
|
common_root_lv: ubuntu-lv
|
|
|
|
# --- Staged GGUF models for the llama.cpp router (:8002) ---------------------
|
|
# Data-driven list consumed by roles/llm-inference-multimodel tasks/models.yml
|
|
# (loop -> tasks/stage_model.yml). Each entry is idempotently staged into
|
|
# /opt/models: stat + EXACT-size check vs HF manifest; skip (no download, no
|
|
# restart) when present + size matches. Source repos are public bartowski GGUFs
|
|
# on HuggingFace (no auth). A router restart is notified ONLY when a new GGUF
|
|
# is actually downloaded.
|
|
# Added 2026-08-12 (War Machine): codify Phi-3.5-mini-instruct-Q8_0 and
|
|
# Meta-Llama-3.1-8B-Instruct-Q4_K_M as router models alongside the production
|
|
# Qwen3.6-35B-A3B-UD-Q4_K_S. The live files were already present/correct on
|
|
# astro-orbiter; this pass codifies them. Future adds = append to this list.
|
|
# Router --models-max override for astro-orbiter.
|
|
# Default in defaults/main.yml is 1 (conservative). Bumped to 4 on 2026-08-12
|
|
# (t_33acbb2e) so the router can keep more than one GGUF resident on-demand
|
|
# and LRU-evict when needed.
|
|
#
|
|
# VRAM NOTE (t_33acbb2e, updated t_55c164f5): With models-max=4 and all 4 GGUFs
|
|
# registered, worst case is all 4 loaded simultaneously:
|
|
# Qwen3.6-35B-A3B Q4_K_S: ~21.5GB (weights ~19.5GB + KV ~2GB @ 64K ctx, q4_0)
|
|
# Phi-3.5-mini-instruct Q8_0: ~4.3GB (weights ~3.8GB + KV ~0.5GB @ 32K ctx)
|
|
# Meta-Llama-3.1-8B Q4_K_M: ~5.6GB (weights ~4.6GB + KV ~0.2GB @ 8K ctx)
|
|
# Qwen2.5-Coder-14B Q4_K_M: ~9.0GB (weights ~8.4GB + KV ~0.6GB @ 16K ctx)
|
|
# Total worst-case: ~40.4GB >> 24GB RTX 3090
|
|
#
|
|
# OOM RISK: Full co-residency is impossible on 24GB. LRU eviction prevents this
|
|
# in practice: models-max=4 means the router can REGISTER 4 models but only keeps
|
|
# up to 4 LOADED simultaneously — the router will evict the LRU model when a new
|
|
# one is needed. In single-user homelab operation, only one model is active at a
|
|
# time. The realistic maximum co-residency is 2 models (whichever was last used).
|
|
# Qwen3.6-35B alone uses ~21.5GB; co-residency with Coder (~9GB) = ~30.5GB > 24GB.
|
|
# So effectively: Qwen3.6-35B + any second model will OOM IF both are held concurrently.
|
|
# LRU eviction handles this automatically — the router evicts the idle model before
|
|
# loading the new one. Ryan should be aware this means model-switching always incurs
|
|
# a ~30-60s cold-load latency when switching between Qwen3.6-35B and any other model.
|
|
# Proceeding to models-max=4 as instructed; flagged for Ryan's attention.
|
|
llm_router_models_max: 4
|
|
|
|
llm_staged_models:
|
|
- filename: "Phi-3.5-mini-instruct-Q8_0.gguf"
|
|
url: "https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF/resolve/main/Phi-3.5-mini-instruct-Q8_0.gguf"
|
|
size_bytes: 4061222688
|
|
source_repo: "bartowski/Phi-3.5-mini-instruct-GGUF"
|
|
- filename: "Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
|
|
url: "https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
|
|
size_bytes: 4920739232
|
|
source_repo: "bartowski/Meta-Llama-3.1-8B-Instruct-GGUF"
|
|
- filename: "Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
|
|
url: "https://huggingface.co/bartowski/Qwen2.5-Coder-14B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
|
|
size_bytes: 8988111072
|
|
source_repo: "bartowski/Qwen2.5-Coder-14B-Instruct-GGUF"
|
|
|