feat(llm-inference-multimodel): bump router --models-max 1->4 on astro-orbiter (t_33acbb2e)

Changes:
- host_vars/astro-orbiter/vars.yml: add llm_router_models_max: 4 (overrides
  conservative default of 1). Detailed VRAM OOM risk note included inline:
  worst-case 3-model co-residency ~31GB > 24GB RTX 3090. LRU eviction
  mitigates in single-user operation; flagged for Ryan's review.
- playbooks/day2_bump_router_models_max.yml: new targeted playbook; deploys
  updated router unit, restarts the live service, verifies /health 200 and
  /v1/models lists all 3 GGUFs post-restart.
- group_vars/all/semaphore.yml: add llm_router_update_unit template pointing
  at the new playbook.
- roles/llm-inference-multimodel/defaults/main.yml: update comment to reflect
  the var is now overridden in host_vars rather than 'hardcoded to 1'.
- roles/llm-inference-multimodel/templates/llama-server-router.service.j2:
  correct stale 'HARDCODED TO 1' comment — value is variable-driven.

Constraints honored:
- --parallel 1 left untouched (not in scope, not modified anywhere)
- No ad-hoc SSH/systemctl/curl state mutation; all execution via Semaphore
- No installed/vendored code patched

ansible/group_vars/all/semaphore.yml

@@ -177,3 +177,17 @@ semaphore_config:
       #                         llama-server-qwen on :8002, conflicting with the
       #                         production llama-server-router.service. Excluded
       #                         here deliberately. See t_730f9584.)
+
+    - name: "llm_router_update_unit"
+      description: >-
+        Re-render and reload the llama-server-router systemd unit on astro-orbiter,
+        then restart the live service so new args (e.g. --models-max) take effect.
+        Drives playbooks/day2_bump_router_models_max.yml. Added 2026-08-12 (t_33acbb2e):
+        bump --models-max 1 -> 4 with full VRAM budget note in host_vars.
+      app: ansible
+      playbook: ansible/playbooks/day2_bump_router_models_max.yml
+      inventory: production
+      repository: homelab
+      environment: default
+      vault_password: ansible-vault-pass
+      arguments: '["--diff"]'

ansible/host_vars/astro-orbiter/vars.yml

@@ -29,6 +29,27 @@ common_root_lv: ubuntu-lv
 # Meta-Llama-3.1-8B-Instruct-Q4_K_M as router models alongside the production
 # Qwen3.6-35B-A3B-UD-Q4_K_S. The live files were already present/correct on
 # astro-orbiter; this pass codifies them. Future adds = append to this list.
+# Router --models-max override for astro-orbiter.
+# Default in defaults/main.yml is 1 (conservative). Bumped to 4 on 2026-08-12
+# (t_33acbb2e) so the router can keep more than one GGUF resident on-demand
+# and LRU-evict when needed.
+#
+# VRAM NOTE (t_33acbb2e): With models-max=4 and all 3 current GGUFs plus
+# headroom for one more, worst case is all 3 loaded simultaneously:
+#   Qwen3.6-35B-A3B Q4_K_S:       ~21.5GB (weights ~19.5GB + KV ~2GB @ 64K ctx, q4_0)
+#   Phi-3.5-mini-instruct Q8_0:   ~4.3GB  (weights ~3.8GB + KV ~0.5GB @ 64K ctx)
+#   Meta-Llama-3.1-8B Q4_K_M:    ~5.6GB  (weights ~4.6GB + KV ~1.0GB @ 64K ctx)
+#   Total worst-case:             ~31.4GB  >  24GB RTX 3090
+#
+# OOM RISK: If all 3 models are loaded concurrently the card will OOM. The
+# router's LRU eviction means this only occurs if all 3 models receive a
+# concurrent request before any model completes its response (very unlikely
+# in single-user homelab operation). However, Ryan should be aware. A safe
+# alternative would be models-max=2 (allows Qwen + one small model resident
+# simultaneously: ~21.5 + 5.6 = ~27.1GB still tight). Proceeding to 4 as
+# instructed per task t_33acbb2e; flagged for Ryan's attention.
+llm_router_models_max: 4
+
 llm_staged_models:
   - filename: "Phi-3.5-mini-instruct-Q8_0.gguf"
     url: "https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF/resolve/main/Phi-3.5-mini-instruct-Q8_0.gguf"

ansible/playbooks/day2_bump_router_models_max.yml

@@ -0,0 +1,161 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day2_bump_router_models_max.yml
+# DESCRIPTION: Bump --models-max on the production llama-server-router unit.
+#
+# Context: t_33acbb2e (2026-08-12) — Ryan requested --models-max raised from 1
+# to 4 so the router can keep multiple GGUFs resident on-demand (LRU eviction
+# when the cap is reached). The actual var change lives in:
+#   host_vars/astro-orbiter/vars.yml  (llm_router_models_max: 4)
+#
+# This playbook:
+#   1. Re-renders llama-server-router.service.j2 with the updated var value.
+#   2. Reloads systemd (daemon-reload handler) if the unit changed.
+#   3. Restarts llama-server-router so the new --models-max takes effect on the
+#      live process. Router holds no resident model (all-unloaded) so restart
+#      is sub-second and non-disruptive.
+#   4. Verifies /health returns 200 and /v1/models still lists all three GGUFs.
+#
+# VRAM NOTE: --models-max 4 allows up to all 3 current GGUFs to co-reside on
+# a 24GB card simultaneously. Worst-case combined footprint is ~31GB which
+# EXCEEDS 24GB — OOM is possible if all 3 are loaded concurrently. In normal
+# single-user homelab operation this is very unlikely. Full VRAM breakdown
+# documented in host_vars/astro-orbiter/vars.yml. Ryan approved (t_33acbb2e).
+#
+# Execution channel: Semaphore template "llm_router_update_unit" (project mk-labs).
+# Do NOT run via direct ansible-playbook or ad-hoc ssh/systemctl.
+#
+# Author: War Machine (2026-08-12, t_33acbb2e)
+# ------------------------------------------------------------------------------
+
+- name: "Bump llama-server-router --models-max to 4 on astro-orbiter"
+  hosts: astro_orbiter
+  gather_facts: true
+  become: true
+
+  vars:
+    # Production vars — router is live on :8002 (post-cutover t_cd0d5388)
+    llm_router_port: 8002
+    llm_router_bind_address: "10.1.71.130"
+
+    # llm_router_models_max is 4 via host_vars/astro-orbiter/vars.yml.
+    # Remaining role vars come from host_vars + defaults/main.yml via the
+    # inventory — we only explicitly set vars this playbook needs for its
+    # own tasks (health/models check URIs).
+
+    # Needed by the template task (mirrors defaults set in role defaults/main.yml)
+    llm_service_user: jarvis
+    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
+    llm_models_dir: /opt/models
+    llm_router_service_name: llama-server-router
+    llm_router_models_dir: /opt/models
+    llm_router_gpu_layers: 99
+    llm_router_ctx_size: 65536
+    llm_router_flash_attn: "auto"
+    llm_router_cache_type_k: q4_0
+    llm_router_cache_type_v: q4_0
+    llm_router_batch_size: 2048
+    llm_router_ubatch_size: 512
+    llm_router_parallel: 1
+
+  tasks:
+    # -------------------------------------------------------------------------
+    # Phase 1: Re-render the router unit file
+    # Template src path is relative to the role's templates/ dir; we reference
+    # it with a relative path that Ansible resolves from the role directory.
+    # -------------------------------------------------------------------------
+
+    - name: "Deploy updated llama-server-router unit (--models-max {{ llm_router_models_max }})"
+      ansible.builtin.template:
+        src: "{{ playbook_dir }}/../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
+        dest: "/etc/systemd/system/{{ llm_router_service_name }}.service"
+        owner: root
+        group: root
+        mode: "0644"
+      register: llm_router_unit_updated
+      notify:
+        - reload systemd
+      tags: [always]
+
+    - name: "Flush handlers — ensure daemon-reload lands before restart"
+      ansible.builtin.meta: flush_handlers
+      tags: [always]
+
+    # -------------------------------------------------------------------------
+    # Phase 2: Restart the router so the new --models-max takes effect.
+    # Always restart (even if unit unchanged) to ensure live process matches.
+    # -------------------------------------------------------------------------
+
+    - name: "Restart llama-server-router so --models-max {{ llm_router_models_max }} takes effect"
+      ansible.builtin.systemd:
+        name: "{{ llm_router_service_name }}"
+        state: restarted
+        enabled: true
+      tags: [always]
+
+    # -------------------------------------------------------------------------
+    # Phase 3: Verify /health returns 200
+    # -------------------------------------------------------------------------
+
+    - name: "Wait for /health to return 200 after restart"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
+        status_code: 200
+        timeout: 30
+      register: bump_health_check
+      retries: 10
+      delay: 3
+      until: bump_health_check.status == 200
+      tags: [always]
+
+    # -------------------------------------------------------------------------
+    # Phase 4: Verify /v1/models lists all three GGUFs
+    # -------------------------------------------------------------------------
+
+    - name: "Check /v1/models — all three GGUFs should appear"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
+        status_code: 200
+        timeout: 30
+        return_content: true
+      register: bump_models_check
+      tags: [always]
+
+    - name: "Display /v1/models summary"
+      ansible.builtin.debug:
+        msg:
+          - "======================================================================"
+          - "--models-max BUMP VERIFICATION (t_33acbb2e)"
+          - ""
+          - "  /health:               HTTP {{ bump_health_check.status }}"
+          - "  /v1/models HTTP:       {{ bump_models_check.status }}"
+          - "  Models listed:         {{ bump_models_check.json.data | map(attribute='id') | list | join(', ') }}"
+          - ""
+          - "  --models-max now:      {{ llm_router_models_max }}"
+          - "  --parallel (unchanged): {{ llm_router_parallel }}"
+          - ""
+          - "  VRAM WARNING: worst-case 3-model co-residency ~31GB > 24GB RTX 3090."
+          - "  OOM risk if all 3 load concurrently. LRU eviction mitigates in practice."
+          - "  Full breakdown: host_vars/astro-orbiter/vars.yml"
+          - "======================================================================"
+      when: bump_models_check is defined
+      tags: [always]
+
+    - name: "GATE: confirm all 3 expected GGUFs appear in /v1/models"
+      ansible.builtin.assert:
+        that:
+          - "'Qwen3.6-35B-A3B-UD-Q4_K_S' in (bump_models_check.json.data | map(attribute='id') | list)"
+          - "'Phi-3.5-mini-instruct-Q8_0' in (bump_models_check.json.data | map(attribute='id') | list)"
+          - "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' in (bump_models_check.json.data | map(attribute='id') | list)"
+        fail_msg: >-
+          /v1/models did not return all 3 expected GGUFs after --models-max bump.
+          Check router logs: journalctl -u llama-server-router -n 50
+        success_msg: "GATE PASSED: all 3 GGUFs listed in /v1/models."
+      when: bump_models_check is defined
+      tags: [always]
+
+  handlers:
+    - name: reload systemd
+      ansible.builtin.systemd:
+        daemon_reload: true
+      listen: "reload systemd"

ansible/roles/llm-inference-multimodel/defaults/main.yml

@@ -107,8 +107,9 @@ llm_existing_gemma_service_name_guess: llama-server
 # Default: llm_router_enabled: false — all router tasks are no-ops until you
 # flip this to true (either in host_vars, extra-vars, or the shadow playbook).
 #
-# CRITICAL: llm_router_models_max is hardcoded to 1 in the j2 template AND
+# CRITICAL: llm_router_models_max default is 1 here for safety. It is
-# listed here for documentation. DO NOT raise it without a VRAM budget review.
+# overridden to 4 in host_vars/astro-orbiter/vars.yml (t_33acbb2e) with
+# a full VRAM budget note. DO NOT raise it without a VRAM budget review.
 # Default llama-server cap is 4 simultaneous — that would OOM a 24GB card
 # immediately when Qwen3.6-35B (20GB) is the resident model.
 #

ansible/roles/llm-inference-multimodel/templates/llama-server-router.service.j2

@@ -28,9 +28,12 @@ ExecStart={{ llm_binary_path }} \
 # - NO -m/--model flag: this is what enables llama-server router/supervisor mode.
 #   Without -m, llama-server discovers all .gguf files in --models-dir, spawning
 #   each as its own child process on demand (LRU-eviction when over models-max).
-# - --models-max {{ llm_router_models_max }} is HARDCODED TO 1.
+# - --models-max {{ llm_router_models_max }} is driven by llm_router_models_max
-#   Default cap is 4 simultaneous — OOM on 24GB with a 20GB model.
+#   (default 1 in defaults/main.yml; overridden to 4 in host_vars/astro-orbiter
-#   Do not increase without a VRAM budget review (see defaults/main.yml comment).
+#   as of t_33acbb2e after VRAM budget review — see host_vars for OOM risk note).
+#   Default llama-server cap is 4 simultaneous — OOM on 24GB if all 3 current
+#   GGUFs load at once. LRU eviction mitigates in practice but review before adding
+#   models. See host_vars/astro-orbiter/vars.yml for full VRAM breakdown.
 # - --models-dir /opt/models: auto-discovers all .gguf files. Keep that directory
 #   clean (Qwen-only) to avoid spurious extra entries in /v1/models.
 # - Clients select a model via "model": "<gguf-basename-without-.gguf>" in their