7b44a41da3
Ryan-directed model swap (kanban t_f5f7e9ad, 2026-08-16). Changes: - Replace [Qwen3.6-35B-A3B-UD-Q4_K_S] with [Qwen3.8-27B-Q4_K_M] in llama-server-router-preset.ini.j2 (production model slot). - Qwen3.8-27B: dense 27B VLM, Apache-2.0, Alibaba Aug 2026. Unsloth Dynamic V3.0 GGUF quantization. Q4_K_M chosen: 17,106,775,008 bytes, 17.1GB. Measured VRAM: 17,068 MiB at ctx=32768 (q4_0 KV cache). - ctx-size set to 32768 (32K) via new variable llm_router_qwen38_ctx_size. Native context is 262K; 32K chosen to maintain eviction headroom on 24GB RTX 3090. - models-max reduced 4 -> 2 in host_vars. Qwen3.8 (17.6GB) + nomic-embed (558MB) exhaust the 24GB card; no auxiliary model can co-reside with Qwen3.8. LRU eviction handles model switching with ~30-60s cold-load latency. - llm_router_expected_model_id updated to Qwen3.8-27B-Q4_K_M. - Qwen3.6 GGUF retained at /opt/models/Qwen3.6-35B-A3B-UD-Q4_K_S.gguf (not deleted — pending stable period and explicit cleanup task). - day2_swap_qwen38.yml playbook added for Ansible idempotent redeployment. Architecture note: Qwen3.8 uses Gated DeltaNet; llama.cpp 6ea215d logs 'fused Gated Delta Net (chunked) not supported, set to disabled'. Inference works correctly on the non-fused fallback. A llama.cpp update may improve throughput on the GDN layers. Smoke test passed: model responded via router endpoint (http://10.1.71.130:8002). VRAM: 17,630 MiB (Qwen3.8) + 5,928 MiB (Llama-8B concurrent) = 23,558 MiB. Also commits accumulated but unpushed changes: - nomic-embed batch-size/rope-scaling fix (t_openviking_embed_batch) - per-model ctx-size day2 playbook (day2_per_model_ctx_size.yml) - llama-server-router.service.j2 minor update
37 lines
1.4 KiB
YAML
37 lines
1.4 KiB
YAML
---
|
|
# ------------------------------------------------------------------------------
|
|
# Playbook: day2_swap_qwen38.yml
|
|
# Purpose: Swap the primary production model on astro-orbiter router from
|
|
# Qwen3.6-35B-A3B-UD-Q4_K_S to Qwen3.8-27B-Q4_K_M.
|
|
# This is a GitOps-encoded record of the swap performed 2026-08-16
|
|
# per Ryan's direction (kanban task t_f5f7e9ad).
|
|
#
|
|
# What this playbook does:
|
|
# 1. Renders the updated llama-server-router-preset.ini.j2 to
|
|
# /opt/llama-server-router-preset.ini on astro-orbiter.
|
|
# 2. Reloads the llama-server-router service (SIGHUP / restart as needed).
|
|
# 3. Verifies the new model ID appears in /v1/models.
|
|
#
|
|
# Prerequisites:
|
|
# - Qwen3.8-27B-Q4_K_M.gguf must be present in /opt/models on astro-orbiter.
|
|
# (Downloaded out-of-band via wget during the swap task.)
|
|
# - roles/llm-inference-multimodel/defaults/main.yml updated to reference
|
|
# Qwen3.8-27B-Q4_K_M (done in this same commit).
|
|
#
|
|
# Run:
|
|
# env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
|
|
# -i inventory.yml \
|
|
# playbooks/day2_swap_qwen38.yml
|
|
#
|
|
# Task reference: t_f5f7e9ad — War Machine, 2026-08-16
|
|
# ------------------------------------------------------------------------------
|
|
- name: Swap primary model to Qwen3.8-27B-Q4_K_M on astro-orbiter
|
|
hosts: astro-orbiter
|
|
become: true
|
|
vars:
|
|
llm_router_preset_enabled: true
|
|
|
|
roles:
|
|
- role: llm-inference-multimodel
|
|
tags: [preset, systemd, verify]
|