5cf4468754
Part A: qwen3-no-think.jinja.j2 Ansible template
- New template: templates/qwen3-no-think.jinja.j2
Standard Qwen3 chat template with enable_thinking unconditionally false
(hardcoded empty <think></think> prefix at add_generation_prompt step).
Deployed to /opt/models/templates/qwen3-no-think.jinja on astro-orbiter.
- tasks/models.yml: deploy templates dir + qwen3-no-think.jinja via
ansible.builtin.template task (tags: models, chat_templates).
Part B: INI preset template — two Qwen3-8B sections
- templates/llama-server-router-preset.ini.j2:
[Qwen3-8B-Q4_K_M] — thinking variant, same GGUF, baked-in template.
[Qwen3-8B-Q4_K_M-no_think] — no-think variant, same GGUF,
chat-template-file = /opt/models/templates/qwen3-no-think.jinja.
Both sections: n-gpu-layers=99, ctx-size=32768, flash-attn=true,
q4_0 KV cache, sleep-idle-seconds=60.
Part C: defaults/main.yml — llama-swap models + matrix
- llm_swapmode_models: added Qwen3-8B-Q4_K_M (port 8106, GPU) and
Qwen3-8B-Q4_K_M-no_think (port 8107, GPU, chat_template_file set).
- llm_swapmode_matrix_rows: row5 (Qwen3-8B thinking + embed),
row6 (Qwen3-8B no_think + embed). Neither co-resident with Qwen3.8-27B.
- templates/llama-swap-config.yaml.j2: added chat_template_file support
(--chat-template-file flag conditional on model.chat_template_file).
- tasks/swapmode.yml: GATE 2 assert updated 5 -> 7 models.
Part D: war-machine config.yaml (not tracked in git)
- custom_providers.astro-orbiter.models: added Qwen3-8B-Q4_K_M-no_think
(context_length: 32768).
- Reassigned 5 latency-sensitive aux tasks from Meta-Llama-3.1-8B to
Qwen3-8B-Q4_K_M-no_think: skills_hub, approval, mcp, title_generation,
profile_describer. Rationale: GPU-resident, lower latency, json_schema OK.
- web_extract and compression remain on Phi-3.5-mini (long scrapes).
VRAM: both Qwen3-8B variants co-reside with nomic-embed only (~10.4GB +
84MB). Cannot co-reside with Qwen3.8-27B (17.8GB); LRU eviction applies.
61 lines
2.8 KiB
Django/Jinja
61 lines
2.8 KiB
Django/Jinja
{#
|
|
FILE: roles/llm-inference-multimodel/templates/llama-swap-config.yaml.j2
|
|
DESCRIPTION: llama-swap v250 configuration template.
|
|
Generates /etc/llama-swap/config.yaml with all models, routing matrix,
|
|
and per-model settings (ctx_size, n_gpu_layers, cmd args).
|
|
|
|
v250 SYNTAX NOTES:
|
|
- Uses routing.router DSL with expression-based matrix (not old list-of-arrays)
|
|
- Each model has its own cmd field with full per-model args
|
|
- Matrix rows use "model1 & model2" syntax for co-resident sets
|
|
- sleep_idle_seconds: -1 = never idle; 0+ = idle after N seconds
|
|
- load_on_startup: true = start this model on service startup
|
|
|
|
Reference: /etc/llama-swap/config.yaml on astro-orbiter (Phase 1 artifact)
|
|
#}
|
|
# llama-swap configuration for astro-orbiter
|
|
# Generated by Ansible roles/llm-inference-multimodel on {{ ansible_date_time.iso8601 }}
|
|
# See: https://github.com/mostlygeek/llama-swap (v250 release notes for syntax)
|
|
|
|
# ============================================================================
|
|
# LISTEN — Address and port for the llama-swap proxy
|
|
# ============================================================================
|
|
listen: "{{ llm_swapmode_bind_address }}:{{ llm_swapmode_port }}"
|
|
|
|
# ============================================================================
|
|
# MODELS — All model definitions (cmd, port, ctx_size, etc.)
|
|
# ============================================================================
|
|
models:
|
|
{% for model in llm_swapmode_models %}
|
|
{{ model.id }}:
|
|
cmd: >
|
|
llama-server
|
|
--port ${PORT}
|
|
--model {{ model.gguf_path }}
|
|
--n-gpu-layers {{ model.n_gpu_layers }}
|
|
--ctx-size {{ model.ctx_size }}
|
|
--batch-size {{ model.batch_size }}
|
|
--ubatch-size {{ model.ubatch_size }}
|
|
--parallel {{ model.parallel }}
|
|
{% if model.cache_type is defined %}--cache-type-k {{ model.cache_type }} --cache-type-v {{ model.cache_type }}{% endif %}
|
|
{% if model.flash_attn is defined %}--flash-attn {{ model.flash_attn }}{% endif %}
|
|
{% if model.chat_template_file is defined %}--chat-template-file {{ model.chat_template_file }}{% endif %}
|
|
{% if model.sleep_idle_seconds is defined %}--sleep-idle-seconds {{ model.sleep_idle_seconds }}{% endif %}
|
|
{% if model.load_on_startup is defined and model.load_on_startup %}--load-on-startup{% endif %}
|
|
--host 127.0.0.1
|
|
port: {{ model.port }}
|
|
{% endfor %}
|
|
|
|
# ============================================================================
|
|
# ROUTING — Matrix-based hot-swap policy (v250 expression DSL)
|
|
# ============================================================================
|
|
routing:
|
|
router:
|
|
use: matrix
|
|
settings:
|
|
matrix:
|
|
sets:
|
|
{% for row in llm_swapmode_matrix_rows %}
|
|
{{ row.row }}: "{{ row.expr }}"
|
|
{% endfor %}
|