081156ecab
Adds idempotent, data-driven GGUF staging for the two new router models on
astro-orbiter alongside the production Qwen3.6-35B-A3B-UD-Q4_K_S. Both files
were already staged live (byte-exact); this commit codifies them in Ansible so
future re-runs and any new model adds are version-controlled and audit-friendly.
Changes:
- roles/llm-inference-multimodel/tasks/stage_model.yml (NEW)
Idempotent per-model task: stat -> exact byte-size guard -> conditional
get_url -> ownership/mode ensure -> notify router restart handler only on
actual download. Loops from models.yml; nothing hardcoded.
- roles/llm-inference-multimodel/tasks/models.yml
Appends the stage_model.yml loop (tagged: models) after the existing Qwen3.6
download tasks. Data driven from host_vars/astro-orbiter/vars.yml.
- roles/llm-inference-multimodel/defaults/main.yml
Adds llm_staged_models: [] default (empty = safe no-op for hosts with no
staged model list defined).
- roles/llm-inference-multimodel/handlers/main.yml
Adds 'restart llama-server-router on new GGUF' handler. Only fires when
stage_model.yml performs an actual download or corrects ownership/mode.
Normal idempotent re-runs (files already correct) do NOT fire this handler.
- host_vars/astro-orbiter/vars.yml
Adds llm_staged_models list with the two new models:
* Phi-3.5-mini-instruct-Q8_0.gguf (4,061,222,688 bytes,
bartowski/Phi-3.5-mini-instruct-GGUF)
* Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf (4,920,739,232 bytes,
bartowski/Meta-Llama-3.1-8B-Instruct-GGUF)
- playbooks/day1_deploy_llm_inference_multimodel.yml
Updates header comment: removes stale 'Semaphore broken' note, documents
the correct execution channel (Semaphore template
llm_inference_multimodel_stage_models, --tags models).
- group_vars/all/semaphore.yml
Adds llm_inference_multimodel_stage_models template entry (config-as-code).
Template is scoped to --tags models explicitly. Phase 4 (verify) is
EXCLUDED: verify.yml starts llama-server-qwen on :8002, which would collide
with the production llama-server-router.service already running on :8002.
Semaphore template created via API: project 1 / template id 19.
Execution: triggered immediately after this commit via Semaphore REST API.
77 lines
3.8 KiB
YAML
77 lines
3.8 KiB
YAML
---
|
|
# ------------------------------------------------------------------------------
|
|
# FILE: roles/llm-inference-multimodel/tasks/models.yml
|
|
# DESCRIPTION: Phase 1 — ensure the production Qwen GGUF is present on disk.
|
|
# Idempotent: reuses the stat + size-threshold guard pattern.
|
|
#
|
|
# HISTORY (2026-08-06): This file previously downloaded Phi-4-14B
|
|
# (aux, port 8000) and Mistral-Small-24B (tool-calling, port 8001).
|
|
# Both were retired on 2026-08-06 when the deployment was
|
|
# consolidated to a single model (Qwen2.5-14B-Instruct-1M, port
|
|
# 8002). The download tasks and VRAM co-residency logic were
|
|
# removed from this file; see git log if a rollback needs them.
|
|
#
|
|
# HISTORY (2026-08-07): Qwen2.5-14B-Instruct-1M was superseded by
|
|
# Qwen3.6-35B-A3B-UD-Q4_K_S (see task t_2ffc0f63). The model
|
|
# was downloaded out-of-band (direct wget per t_2ffc0f63 runbook)
|
|
# rather than via this role's get_url pattern. The path and
|
|
# variables below are updated to reflect the current production
|
|
# model; the download task is a no-op if the file is already
|
|
# present (which it is on astro-orbiter as of 2026-08-07+).
|
|
#
|
|
# 2026-08-12 (t_0cca74a2): Cleaned up stale Phi-4/Mistral tasks
|
|
# that referenced undefined variables after the Aug 2026
|
|
# consolidation. models.yml now only manages the Qwen3.6-35B
|
|
# model that is the sole production model.
|
|
# ------------------------------------------------------------------------------
|
|
|
|
- name: Create models directory
|
|
ansible.builtin.file:
|
|
path: "{{ llm_models_dir }}"
|
|
state: directory
|
|
owner: "{{ llm_service_user }}"
|
|
group: "{{ llm_service_user }}"
|
|
mode: "0755"
|
|
become: true
|
|
|
|
# --- Production model: Qwen3.6-35B-A3B-UD-Q4_K_S (port 8002 / router :8003) -
|
|
|
|
- name: Check if Qwen3.6-35B GGUF is present on disk
|
|
ansible.builtin.stat:
|
|
path: "{{ llm_qwen_model_path }}"
|
|
register: llm_qwen_model_stat
|
|
|
|
- name: Report Qwen model presence (model was downloaded out-of-band via t_2ffc0f63)
|
|
ansible.builtin.debug:
|
|
msg: >-
|
|
Qwen model at {{ llm_qwen_model_path }}:
|
|
exists={{ llm_qwen_model_stat.stat.exists | default(false) }},
|
|
size={{ (llm_qwen_model_stat.stat.size | default(0) | int / 1073741824) | round(2) }}GB
|
|
when: llm_qwen_model_stat.stat.exists | default(false)
|
|
|
|
- name: WARN — Qwen model GGUF not found at expected path
|
|
ansible.builtin.debug:
|
|
msg: >-
|
|
WARNING: Qwen model NOT found at {{ llm_qwen_model_path }}.
|
|
This model was originally downloaded via task t_2ffc0f63 (direct wget,
|
|
not via this role's get_url). If the file is missing, re-download it
|
|
manually or add a get_url task here with the correct HuggingFace URL.
|
|
Expected URL (bartowski UD-Q4_K_S):
|
|
https://huggingface.co/bartowski/Qwen3.6-35B-A3B-UD-Q4_K_S-GGUF/resolve/main/Qwen3.6-35B-A3B-UD-Q4_K_S.gguf
|
|
when: not (llm_qwen_model_stat.stat.exists | default(false))
|
|
|
|
# --- Staged GGUF models (data-driven, idempotent) ----------------------------
|
|
# Ensure every entry in llm_staged_models is present in llm_models_dir with the
|
|
# EXACT expected byte size. When present AND size matches, this is a pure
|
|
# no-op: no download, no service touch. When a genuine new/mismatched GGUF is
|
|
# detected, it is downloaded + ownership/mode corrected and the router restart
|
|
# handler is notified so the llama.cpp router re-discovers the models_dir.
|
|
# Driven entirely by inventory vars (host_vars) — nothing hardcoded here, so
|
|
# adding a future model = append to llm_staged_models in host_vars.
|
|
- name: Stage data-driven GGUF models into {{ llm_models_dir }}
|
|
ansible.builtin.include_tasks: stage_model.yml
|
|
loop: "{{ llm_staged_models | default([]) }}"
|
|
loop_control:
|
|
loop_var: staged_model
|
|
tags: [models]
|