782cbe33d1
Previous ctx-size=8192/parallel=4 gave 2048 tokens/slot, too small for context compression inputs (observed live rejection at 3826 tokens). Measured VRAM on astro-orbiter (RTX 3090 24GB): weights ~17GB resident, ~294KiB/token pool-wide for KV cache+buffers at prior sizing. New: ctx-size=16384, parallel=2 -> 8192 tokens/slot (matches model's native n_ctx_train max). Projected VRAM ~21.8GB, ~2.7GB headroom. Applied directly via ansible-playbook (Semaphore currently broken -- fix tracked separately).
56 lines
2.5 KiB
YAML
56 lines
2.5 KiB
YAML
---
|
|
# ------------------------------------------------------------------------------
|
|
# FILE: roles/llm-inference/defaults/main.yml
|
|
# DESCRIPTION: Overridable defaults for the llm-inference role.
|
|
# Deploy target: astro-orbiter (Dell OptiPlex 7050 SFF, RTX 3090
|
|
# via OCuLink, Ubuntu 24.04.4 LTS).
|
|
# ------------------------------------------------------------------------------
|
|
|
|
# NVIDIA driver
|
|
llm_nvidia_driver_package: nvidia-driver-595-open
|
|
|
|
# Python venv
|
|
llm_venv_path: /home/jarvis/vllm-env
|
|
llm_venv_owner: jarvis
|
|
|
|
# HuggingFace
|
|
llm_hf_cache_dir: /home/jarvis/.cache/huggingface
|
|
llm_hf_model: google/gemma-2-27b-it
|
|
|
|
# vLLM serve (deprecated — replaced by llama-server)
|
|
# llama-server serve
|
|
llm_serve_port: 8000
|
|
llm_serve_host: "0.0.0.0"
|
|
# NOTE (2026-08-05): --ctx-size is llama.cpp's TOTAL KV cache pool, divided
|
|
# evenly across --parallel slots (per-slot context = ctx-size / parallel).
|
|
# Previous 8192/4=2048 tokens-per-slot was too small for aux task offload
|
|
# (context compression) and caused live rejections: "request (3826 tokens)
|
|
# exceeds the available context size (2048 tokens)".
|
|
# Sized against measured VRAM on astro-orbiter (RTX 3090, 24576MiB total):
|
|
# - Weights (Q4_K_M, 27B) ~16998MiB resident.
|
|
# - At ctx-size=8192/parallel=4, total llama-server VRAM = 19404MiB
|
|
# (nvidia-smi), i.e. ~2406MiB for KV cache + compute buffers at 8192
|
|
# total context tokens -> ~294KiB/token (pool-wide, incl. buffers).
|
|
# - Model n_ctx_train=8192 is the native max; per-slot context beyond
|
|
# this degrades coherence, so per-slot should cap at 8192.
|
|
# - New sizing: ctx-size=16384, parallel=2 -> 8192 tokens/slot (native
|
|
# max, covers compression's multi-thousand-token inputs with margin).
|
|
# Projected VRAM: 16998 + (~294KiB/token * 16384) ≈ 21.8GB used,
|
|
# leaving ~2.7GB headroom on the 24GB card.
|
|
# - parallel=2 (not 4) trades some concurrency for correct per-slot
|
|
# context; 2 concurrent aux-task requests is enough headroom before
|
|
# the known "3+ simultaneous compressions" GPU bottleneck kicks in.
|
|
llm_max_model_len: 16384
|
|
llm_gpu_layers: 99 # offload all layers to GPU
|
|
llm_parallel_slots: 2 # concurrent request slots -> 8192 tokens/slot (ctx-size / parallel)
|
|
llm_gguf_dir: /home/jarvis/models
|
|
llm_gguf_path: /home/jarvis/models/gemma-2-27b-it-Q4_K_M.gguf
|
|
|
|
# Legacy vLLM vars (kept for role documentation, not used by llama-server)
|
|
llm_quantization: "bitsandbytes"
|
|
llm_gpu_memory_utilization: "0.92"
|
|
|
|
# Monitoring
|
|
llm_gpu_exporter_version: "1.13.1"
|
|
llm_gpu_exporter_port: 9835
|