Phase 2 revised: consolidate astro-orbiter to single Qwen2.5-14B-1M model (port 8002)
- Retire llama-server-aux (Phi-4, 8000) and llama-server-toolcall (Mistral-Small-24B, 8001): stopped, disabled, unit files removed from host and Ansible role - Promote llama-server-qwen (Qwen2.5-14B-Instruct-1M, port 8002) to sole production model, serving both friday and war-machine Hermes profiles - Verified live: n_ctx=65536/n_ctx_train=1010000, and tool_calls response via /v1/chat/completions probe (no hallucination) - Deleted superseded GGUF weights (phi-4, mistral-small, orphaned base-Qwen, gemma-2-27b) from astro-orbiter, ~45GB reclaimed - Updated friday and war-machine Hermes profile configs (model + compression + skills_hub aux) to point at 10.1.71.130:8002 - Ryan explicitly accepted single-model tradeoffs for both profiles
This commit is contained in:
Hermes Agent service account
@@ -26,43 +26,21 @@ llm_bind_address: "10.1.71.130"
|
||||
# Override per-environment; default assumes Hermes runs somewhere on this /24.
|
||||
llm_allowed_source_cidr: "10.1.70.0/24"
|
||||
|
||||
# --- Aux / classification instance (port 8000, Phi-4-14B) -------------------
|
||||
# Text-only instruction model, no tool-calling training — safe offload target
|
||||
# per the auxiliary-task-offload skill's "no tool_calls emission risk" bar.
|
||||
llm_aux_port: 8000
|
||||
llm_aux_model_path: "{{ llm_models_dir }}/phi-4-14b-instruct-Q4_K_M.gguf"
|
||||
llm_aux_model_url: "https://huggingface.co/bartowski/phi-4-GGUF/resolve/main/phi-4-Q4_K_M.gguf"
|
||||
llm_aux_model_min_bytes: 8000000000 # guard threshold; complete file ~8.5GB
|
||||
llm_aux_ctx_size: 8192
|
||||
llm_aux_parallel: 2
|
||||
llm_aux_gpu_layers: 99
|
||||
llm_aux_service_name: llama-server-aux
|
||||
llm_aux_model_id: phi-4-14b-instruct # served model name for OpenAI-compat API
|
||||
# --- RETIRED (2026-08-06): Aux / classification instance (port 8000, Phi-4-14B)
|
||||
# and Tool-calling instance (port 8001, Mistral-Small-24B) --------------------
|
||||
# Consolidated down to a single production model (Qwen2.5-14B-Instruct-1M,
|
||||
# port 8002) serving BOTH the friday and war-machine Hermes profiles. Ryan
|
||||
# explicitly accepted the tradeoffs (single model for chat + tool-calling +
|
||||
# aux duties) over keeping the aux/toolcall split running.
|
||||
# Both llama-server-aux and llama-server-toolcall services were stopped,
|
||||
# disabled, and had their unit files removed from astro-orbiter; their GGUF
|
||||
# weights (phi-4-14b-instruct-Q4_K_M.gguf, mistral-small-24b-instruct-2501-
|
||||
# Q3_K_M.gguf) were deleted from /opt/models (~45GB reclaimed). The
|
||||
# templates/tasks that deployed them have been removed from this role — see
|
||||
# git log for the prior variable definitions and unit templates if a future
|
||||
# rollback needs them restored.
|
||||
|
||||
# --- Tool-calling instance (port 8001, Mistral-Small-24B) --------------------
|
||||
# Native function-calling support; deployed at Q3_K_M per plan §1 Option B
|
||||
# to fit VRAM budget. MUST pass the §7 validation harness before any
|
||||
# Claude-Code-capable profile is pointed at this port.
|
||||
llm_toolcall_port: 8001
|
||||
llm_toolcall_model_path: "{{ llm_models_dir }}/mistral-small-24b-instruct-2501-Q3_K_M.gguf"
|
||||
llm_toolcall_model_url: "https://huggingface.co/bartowski/Mistral-Small-24B-Instruct-2501-GGUF/resolve/main/Mistral-Small-24B-Instruct-2501-Q3_K_M.gguf"
|
||||
llm_toolcall_model_min_bytes: 11000000000 # guard threshold; complete file ~11.7GB
|
||||
llm_toolcall_ctx_size: 4096
|
||||
llm_toolcall_parallel: 1
|
||||
# 99 (full offload) OOMs on a 24GB GPU when co-resident with llama-server
|
||||
# (phi-4, port 8000, ~10.4GB). 20 layers validated stable alongside it with
|
||||
# headroom to spare (see docs/validation-log.md).
|
||||
llm_toolcall_gpu_layers: 20
|
||||
# Chat template shipped with Mistral-Small-24B-Instruct-2501 has no tool-call
|
||||
# support (no [AVAILABLE_TOOLS]/[TOOL_CALLS] handling) — see
|
||||
# docs/validation-log.md for the investigation. We use Mistral-Nemo-Instruct-2407's
|
||||
# tool-calling-capable template instead; Mistral-Small-24B shares the same
|
||||
# Mistral tokenizer/instruct format family so the template is compatible.
|
||||
llm_toolcall_chat_template_file: "{{ llm_models_dir }}/templates/mistral-small-tool-use.jinja"
|
||||
llm_toolcall_service_name: llama-server-toolcall
|
||||
llm_toolcall_model_id: mistral-small-24b-instruct-2501
|
||||
|
||||
# --- Shadow instance (port 8002) ---------------------------------------------
|
||||
# --- Production instance (port 8002, Qwen2.5-14B-Instruct-1M) ----------------
|
||||
# History (2026-08-06): Qwen2.5-14B-Instruct (base) was deployed to this slot
|
||||
# and DISQUALIFIED — live /v1/models meta reported n_ctx_train=32768, well
|
||||
# under the 64K Hermes floor (the model card's "128K" figure conflated
|
||||
@@ -72,16 +50,17 @@ llm_toolcall_model_id: mistral-small-24b-instruct-2501
|
||||
# tool-calling validation harness badly (8/10 hallucination-stress prompts
|
||||
# triggered spurious tool_calls even at temp=0.1 with the correct official
|
||||
# chat template) — purged from disk and Ansible entirely, see git log.
|
||||
# Current candidate: Qwen2.5-14B-Instruct-1M (bartowski GGUF) — distinct
|
||||
# Current model: Qwen2.5-14B-Instruct-1M (bartowski GGUF) — distinct
|
||||
# checkpoint with genuine additional long-context pretraining, NOT the same
|
||||
# weights as the disqualified base model above. MUST re-verify n_ctx_train
|
||||
# live from /v1/models before trusting any claim — the base-Qwen and
|
||||
# Llama-3.1 lessons both apply here.
|
||||
# GATE: VRAM co-residency with Phi-4(8000)+Mistral(8001) not guaranteed —
|
||||
# stop llama-server-toolcall for the shadow-test window if needed (proven
|
||||
# pattern from prior attempts). llm_qwen_service_enabled defaults to false so
|
||||
# a normal playbook run will template the unit but NOT start it.
|
||||
llm_qwen_service_enabled: false
|
||||
# weights as the disqualified base model above. Live-verified 2026-08-06:
|
||||
# /v1/models reports n_ctx=65536, n_ctx_train=1010000 (well over the 64K
|
||||
# floor). Tool-calling verified live via a /v1/chat/completions probe with a
|
||||
# tools= payload — returned a well-formed tool_calls response (finish_reason
|
||||
# "tool_calls", valid JSON arguments), no hallucinated calls observed.
|
||||
# PROMOTED TO PRODUCTION (2026-08-06): llm_qwen_service_enabled now defaults
|
||||
# to true — this is the sole model serving both Hermes profiles. Ports
|
||||
# 8000/8001 are permanently freed; no co-residency VRAM gate applies anymore.
|
||||
llm_qwen_service_enabled: true
|
||||
llm_qwen_port: 8002
|
||||
llm_qwen_model_path: "{{ llm_models_dir }}/Qwen2.5-14B-Instruct-1M-Q5_K_M.gguf"
|
||||
llm_qwen_model_url: "https://huggingface.co/bartowski/Qwen2.5-14B-Instruct-1M-GGUF/resolve/main/Qwen2.5-14B-Instruct-1M-Q5_K_M.gguf"
|
||||
|
||||
Reference in New Issue
Block a user