feat(llm-inference-multimodel): codify Phi-3.5-mini + Llama-3.1-8B GGUF staging (t_730f9584)
Adds idempotent, data-driven GGUF staging for the two new router models on
astro-orbiter alongside the production Qwen3.6-35B-A3B-UD-Q4_K_S. Both files
were already staged live (byte-exact); this commit codifies them in Ansible so
future re-runs and any new model adds are version-controlled and audit-friendly.
Changes:
- roles/llm-inference-multimodel/tasks/stage_model.yml (NEW)
Idempotent per-model task: stat -> exact byte-size guard -> conditional
get_url -> ownership/mode ensure -> notify router restart handler only on
actual download. Loops from models.yml; nothing hardcoded.
- roles/llm-inference-multimodel/tasks/models.yml
Appends the stage_model.yml loop (tagged: models) after the existing Qwen3.6
download tasks. Data driven from host_vars/astro-orbiter/vars.yml.
- roles/llm-inference-multimodel/defaults/main.yml
Adds llm_staged_models: [] default (empty = safe no-op for hosts with no
staged model list defined).
- roles/llm-inference-multimodel/handlers/main.yml
Adds 'restart llama-server-router on new GGUF' handler. Only fires when
stage_model.yml performs an actual download or corrects ownership/mode.
Normal idempotent re-runs (files already correct) do NOT fire this handler.
- host_vars/astro-orbiter/vars.yml
Adds llm_staged_models list with the two new models:
* Phi-3.5-mini-instruct-Q8_0.gguf (4,061,222,688 bytes,
bartowski/Phi-3.5-mini-instruct-GGUF)
* Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf (4,920,739,232 bytes,
bartowski/Meta-Llama-3.1-8B-Instruct-GGUF)
- playbooks/day1_deploy_llm_inference_multimodel.yml
Updates header comment: removes stale 'Semaphore broken' note, documents
the correct execution channel (Semaphore template
llm_inference_multimodel_stage_models, --tags models).
- group_vars/all/semaphore.yml
Adds llm_inference_multimodel_stage_models template entry (config-as-code).
Template is scoped to --tags models explicitly. Phase 4 (verify) is
EXCLUDED: verify.yml starts llama-server-qwen on :8002, which would collide
with the production llama-server-router.service already running on :8002.
Semaphore template created via API: project 1 / template id 19.
Execution: triggered immediately after this commit via Semaphore REST API.
This commit is contained in:
@@ -59,3 +59,18 @@
|
||||
Expected URL (bartowski UD-Q4_K_S):
|
||||
https://huggingface.co/bartowski/Qwen3.6-35B-A3B-UD-Q4_K_S-GGUF/resolve/main/Qwen3.6-35B-A3B-UD-Q4_K_S.gguf
|
||||
when: not (llm_qwen_model_stat.stat.exists | default(false))
|
||||
|
||||
# --- Staged GGUF models (data-driven, idempotent) ----------------------------
|
||||
# Ensure every entry in llm_staged_models is present in llm_models_dir with the
|
||||
# EXACT expected byte size. When present AND size matches, this is a pure
|
||||
# no-op: no download, no service touch. When a genuine new/mismatched GGUF is
|
||||
# detected, it is downloaded + ownership/mode corrected and the router restart
|
||||
# handler is notified so the llama.cpp router re-discovers the models_dir.
|
||||
# Driven entirely by inventory vars (host_vars) — nothing hardcoded here, so
|
||||
# adding a future model = append to llm_staged_models in host_vars.
|
||||
- name: Stage data-driven GGUF models into {{ llm_models_dir }}
|
||||
ansible.builtin.include_tasks: stage_model.yml
|
||||
loop: "{{ llm_staged_models | default([]) }}"
|
||||
loop_control:
|
||||
loop_var: staged_model
|
||||
tags: [models]
|
||||
|
||||
67
ansible/roles/llm-inference-multimodel/tasks/stage_model.yml
Normal file
67
ansible/roles/llm-inference-multimodel/tasks/stage_model.yml
Normal file
@@ -0,0 +1,67 @@
|
||||
---
|
||||
# ------------------------------------------------------------------------------
|
||||
# FILE: roles/llm-inference-multimodel/tasks/stage_model.yml
|
||||
# DESCRIPTION: Idempotent staging of a SINGLE GGUF listed in llm_staged_models.
|
||||
# Looped from tasks/models.yml (one include per entry).
|
||||
#
|
||||
# GUARD (idempotency):
|
||||
# - stat the target path in llm_models_dir
|
||||
# - get_url ONLY when the file is MISSING or its on-disk size
|
||||
# != the exact HF manifest size (staged_model.size_bytes)
|
||||
# - when present AND size matches -> pure no-op (no download,
|
||||
# no service touch)
|
||||
#
|
||||
# SIDE EFFECT ON CHANGE:
|
||||
# When a genuine new/mismatched GGUF is downloaded (or its
|
||||
# ownership/mode corrected), this task notifies the
|
||||
# "restart llama-server-router on new GGUF" handler so the
|
||||
# llama.cpp router re-discovers the models_dir. A normal
|
||||
# re-run that finds the files already correct will NOT fire
|
||||
# the handler — the live router is left untouched.
|
||||
#
|
||||
# Driven entirely by inventory vars (host_vars), nothing
|
||||
# hardcoded here, so adding a future model = append to the list.
|
||||
# ------------------------------------------------------------------------------
|
||||
|
||||
- name: "Check if {{ staged_model.filename }} is present on disk"
|
||||
ansible.builtin.stat:
|
||||
path: "{{ llm_models_dir }}/{{ staged_model.filename }}"
|
||||
register: _staged_stat
|
||||
|
||||
- name: "Report {{ staged_model.filename }} presence (source: {{ staged_model.source_repo | default('n/a') }})"
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
{{ staged_model.filename }}:
|
||||
exists={{ _staged_stat.stat.exists | default(false) }},
|
||||
size={{ (_staged_stat.stat.size | default(0) | int) }},
|
||||
expected={{ staged_model.size_bytes | int }},
|
||||
match={{ (_staged_stat.stat.exists | default(false)) and
|
||||
((_staged_stat.stat.size | default(0) | int) == (staged_model.size_bytes | int)) }}
|
||||
when: _staged_stat.stat.exists | default(false)
|
||||
|
||||
- name: "Download {{ staged_model.filename }} (missing or size mismatch)"
|
||||
ansible.builtin.get_url:
|
||||
url: "{{ staged_model.url }}"
|
||||
dest: "{{ llm_models_dir }}/{{ staged_model.filename }}"
|
||||
owner: "{{ llm_service_user }}"
|
||||
group: "{{ llm_service_user }}"
|
||||
mode: "0664"
|
||||
timeout: 600
|
||||
become: true
|
||||
when: >-
|
||||
not (_staged_stat.stat.exists | default(false))
|
||||
or (_staged_stat.stat.size | default(0) | int != (staged_model.size_bytes | int))
|
||||
register: _staged_download
|
||||
notify: restart llama-server-router on new GGUF
|
||||
|
||||
- name: "Ensure ownership/mode on {{ staged_model.filename }}"
|
||||
ansible.builtin.file:
|
||||
path: "{{ llm_models_dir }}/{{ staged_model.filename }}"
|
||||
owner: "{{ llm_service_user }}"
|
||||
group: "{{ llm_service_user }}"
|
||||
mode: "0664"
|
||||
become: true
|
||||
when: >-
|
||||
(_staged_stat.stat.exists | default(false))
|
||||
or (_staged_download is changed | default(false))
|
||||
notify: restart llama-server-router on new GGUF
|
||||
Reference in New Issue
Block a user