Add Qwen3-8B no-think variant — dual thinking deployment (t_664289a0)
Part A: qwen3-no-think.jinja.j2 Ansible template
- New template: templates/qwen3-no-think.jinja.j2
Standard Qwen3 chat template with enable_thinking unconditionally false
(hardcoded empty <think></think> prefix at add_generation_prompt step).
Deployed to /opt/models/templates/qwen3-no-think.jinja on astro-orbiter.
- tasks/models.yml: deploy templates dir + qwen3-no-think.jinja via
ansible.builtin.template task (tags: models, chat_templates).
Part B: INI preset template — two Qwen3-8B sections
- templates/llama-server-router-preset.ini.j2:
[Qwen3-8B-Q4_K_M] — thinking variant, same GGUF, baked-in template.
[Qwen3-8B-Q4_K_M-no_think] — no-think variant, same GGUF,
chat-template-file = /opt/models/templates/qwen3-no-think.jinja.
Both sections: n-gpu-layers=99, ctx-size=32768, flash-attn=true,
q4_0 KV cache, sleep-idle-seconds=60.
Part C: defaults/main.yml — llama-swap models + matrix
- llm_swapmode_models: added Qwen3-8B-Q4_K_M (port 8106, GPU) and
Qwen3-8B-Q4_K_M-no_think (port 8107, GPU, chat_template_file set).
- llm_swapmode_matrix_rows: row5 (Qwen3-8B thinking + embed),
row6 (Qwen3-8B no_think + embed). Neither co-resident with Qwen3.8-27B.
- templates/llama-swap-config.yaml.j2: added chat_template_file support
(--chat-template-file flag conditional on model.chat_template_file).
- tasks/swapmode.yml: GATE 2 assert updated 5 -> 7 models.
Part D: war-machine config.yaml (not tracked in git)
- custom_providers.astro-orbiter.models: added Qwen3-8B-Q4_K_M-no_think
(context_length: 32768).
- Reassigned 5 latency-sensitive aux tasks from Meta-Llama-3.1-8B to
Qwen3-8B-Q4_K_M-no_think: skills_hub, approval, mcp, title_generation,
profile_describer. Rationale: GPU-resident, lower latency, json_schema OK.
- web_extract and compression remain on Phi-3.5-mini (long scrapes).
VRAM: both Qwen3-8B variants co-reside with nomic-embed only (~10.4GB +
84MB). Cannot co-reside with Qwen3.8-27B (17.8GB); LRU eviction applies.
This commit is contained in:
@@ -74,3 +74,30 @@
|
||||
loop_control:
|
||||
loop_var: staged_model
|
||||
tags: [models]
|
||||
|
||||
# --- Chat template overrides ---------------------------------------------------
|
||||
# Deploy per-model chat template files used by llama-server via chat-template-file.
|
||||
# These are static files dropped into {{ llm_models_dir }}/templates/.
|
||||
# t_664289a0: qwen3-no-think.jinja — Qwen3 template with enable_thinking=false
|
||||
# hardcoded. Used by [Qwen3-8B-Q4_K_M-no_think] in the router preset INI and
|
||||
# the llama-swap config. The companion [Qwen3-8B-Q4_K_M] section uses the GGUF's
|
||||
# baked-in template (thinking ON by default).
|
||||
- name: Ensure chat template directory exists at {{ llm_models_dir }}/templates
|
||||
ansible.builtin.file:
|
||||
path: "{{ llm_models_dir }}/templates"
|
||||
state: directory
|
||||
owner: "{{ llm_service_user }}"
|
||||
group: "{{ llm_service_user }}"
|
||||
mode: "0755"
|
||||
become: true
|
||||
tags: [models, chat_templates]
|
||||
|
||||
- name: Deploy qwen3-no-think.jinja (thinking=false hard-switch for Qwen3-8B no_think variant)
|
||||
ansible.builtin.template:
|
||||
src: qwen3-no-think.jinja.j2
|
||||
dest: "{{ llm_models_dir }}/templates/qwen3-no-think.jinja"
|
||||
owner: "{{ llm_service_user }}"
|
||||
group: "{{ llm_service_user }}"
|
||||
mode: "0644"
|
||||
become: true
|
||||
tags: [models, chat_templates]
|
||||
|
||||
Reference in New Issue
Block a user