rblundon/homelab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

llm-inference-multimodel: enable Qwen2.5-14B shadow instance (port 8002) for shadow-test window

Browse Source 
Mistral (llama-server-toolcall, 8001) stopped temporarily on astro-orbiter to
free ~6.2GB VRAM headroom for this test window per Ryan/JARVIS approval.
Not a permanent decommission of Mistral.
This commit is contained in:
Hermes Agent service account
2026-08-06 09:10:06 -05:00
parent d10255297c
commit 75cb93f25c
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
ansible/roles/llm-inference-multimodel/defaults/main.yml
View File

@@ -71,7 +71,7 @@ llm_toolcall_model_id: mistral-small-24b-instruct-2501
# (either free VRAM by stopping llama-server-toolcall for the shadow window,
# or reduce ctx-size). llm_qwen_service_enabled defaults to false so a normal
# playbook run will template the unit but NOT start it.
llm_qwen_service_enabled: false
llm_qwen_service_enabled: true
llm_qwen_port: 8002
llm_qwen_model_path: "{{ llm_models_dir }}/Qwen2.5-14B-Instruct-Q5_K_M.gguf"
llm_qwen_model_url: "https://huggingface.co/bartowski/Qwen2.5-14B-Instruct-GGUF/resolve/main/Qwen2.5-14B-Instruct-Q5_K_M.gguf"