homelab/ansible/roles/llm-inference-multimodel/README.md

llm-inference-multimodel

Deploys two independent llama-server systemd services on astro-orbiter's RTX 3090 (24GB), alongside — not replacing — the existing llm-inference role:

Instance	Port	Model	Quant	ctx	parallel	~VRAM
llama-server-aux	8000	Phi-4-14B-Instruct	Q4_K_M	8192	2	~10.0GB
llama-server-toolcall	8001	Mistral-Small-24B-Instruct-2501	Q3_K_M	4096	1	~13.2GB

Combined estimate: ~23.2GB / 24GB (~0.8GB headroom). See /home/hermes/astro-orbiter-multi-model-plan.md for the full approved design (VRAM math, model selection rationale, rollback plan, validation harness).

Relationship to roles/llm-inference

This role does not replace llm-inference. It assumes that role's prerequisites are already satisfied on the host:

• NVIDIA driver installed
• /opt/llama.cpp cloned and built with CUDA (/opt/llama.cpp/build/bin/llama-server exists)
• jarvis service user + /home/jarvis present

The pre-existing single-model Gemma llama-server (however it is currently run) is never modified, restarted, or deleted by this role. It is the rollback target.

Phases

Run the whole role, or scope with --tags:

ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml
# or, once merged into a single play:
ansible-playbook -i inventory.yml <playbook>.yml --tags discover,models,systemd,firewall,verify

0. discover (tasks/discover.yml) — READ-ONLY. Confirms via service_facts + pgrep whether the existing Gemma llama-server actually runs as a systemd unit today, or some ad hoc way (nohup/screen/tmux). Does not assume a unit exists — this was an open unknown in the plan and is resolved here as a fact-gathering step, not an assumption. Also records baseline VRAM and current port 8000/8001 listeners for comparison later.

   If this reports no unit found, stop and read the debug message — it means plan §6's rollback story ("systemctl start the old unit to revert") isn't actually available yet, and that should be fixed (codify the existing process as a systemd unit) before proceeding to Phase 2.

1. models (tasks/models.yml) — Idempotent GGUF download to /opt/models/ with a stat + minimum-size guard (mirrors the pattern in roles/llm-inference/tasks/serve.yml and the llm-inference-homelab skill), so reruns don't re-pull 8.5GB/11.7GB files or mistake a truncated partial download for complete.

2. systemd (tasks/systemd.yml) — Templates and deploys both unit files to /etc/systemd/system/. Deliberately does not start or enable either service — units land on disk as a separately reviewable checkpoint. Two fully independent units (not one unit with two ExecStarts) so either instance can be restarted/stopped without affecting the other.

3. firewall (tasks/firewall.yml) — Scopes ports 8000 and 8001 via ufw to llm_allowed_source_cidr (default the Hermes LAN subnet), rather than leaving them open. Both unit templates also bind to llm_bind_address (default 10.1.71.130, the host's private LAN IP) — not 0.0.0.0 — which is a deliberate change from the pre-existing Gemma pattern flagged as insecure in the plan.

4. verify (tasks/verify.yml) — The only phase that actually starts + enables both services. Waits for /health on both ports, smoke-tests /v1/models and a trivial /v1/chat/completions call on each, checks nvidia-smi VRAM usage against the plan's design estimate, and greps dmesg for OOM-kill events.

   This smoke test is not the tool-calling validation harness. See below.

Key variables

Defined in defaults/main.yml (all overridable via host_vars/group_vars or -e):

• llm_service_user (jarvis), llm_binary_path, llm_models_dir, llm_bind_address, llm_allowed_source_cidr
• Aux: llm_aux_port, llm_aux_model_path, llm_aux_model_url, llm_aux_ctx_size, llm_aux_parallel, llm_aux_gpu_layers
• Tool-calling: llm_toolcall_port, llm_toolcall_model_path, llm_toolcall_model_url, llm_toolcall_ctx_size, llm_toolcall_parallel, llm_toolcall_gpu_layers

vars/main.yml holds constants not meant to be overridden per-host (HF token reference, expected-VRAM figures used only for the verify.yml report).

⚠️ Tool-calling validation is required before use

Port 8001 (Mistral-Small-24B) must pass the manual validation procedure described in plan §7 before any Claude Code / tool-calling-capable Hermes profile is pointed at it:

1. A curl-based tool_calls emission probe (does it call tools correctly on known trigger prompts?)
2. A hallucination stress test (does it fabricate tool_calls on prompts that shouldn't trigger any?)
3. A shadow-mode period (run parallel to the existing tool-calling path, compare outputs, before a hard cutover)

This is intentionally not automated into this role — it is a correctness/safety judgment call, not a repeatable infra check. See docs/validation-log.md in this role directory for the procedure reference and a place to log results once Ryan runs it.

Known gap: Semaphore execution path bypassed for this role (2026-08-05)

The normal execution/audit path (Semaphore) was believed non-functional at authoring time, so this role was run via direct ansible-playbook instead, executed personally by Ryan.

Confirmed 2026-08-05 (JARVIS, via Semaphore API — token vault_semaphore_api_token in the homelab Ansible vault): this was a misdiagnosis, not an outage. Semaphore's service, Postgres backend, and API (/api/ping returns pong) are all healthy on figment (10.1.71.37 — note the documented host city-hall/10.1.71.38 is stale; DNS for imagineering.local.mk-labs.cloud actually resolves through Traefik on lightning-lane to figment:3000). Queried /api/project/1/templates directly: only 6 templates exist project-wide (day0 baseline/root-LV checks, day1 Semaphore self-deploy, Traefik route updates) — none for this role, nor for the original single-model llm-inference role. Root cause confirmed: no Semaphore project template was ever created for LLM inference deployment, which presents identically to "Semaphore is broken" if you don't check the template list.

This is still a known gap — direct ansible-playbook execution bypasses the audit trail Semaphore normally provides. Create a project template for this role's playbook and retarget execution through Semaphore so runs are audited/logged there. Flag this in any future work that touches this role.

Rollback

Qwen2.5-14B shadow deployment (port 8002) — 2026-08-06

Added a third instance definition (llama-server-qwen) per /home/hermes/reports/local-llm-64k-context-recommendation.md, intended to eventually replace the llama-server-toolcall (8001) slot once validated — runs alongside 8000/8001 during the shadow-test window, does not stop or replace either.

VRAM GATE — service NOT started as of this commit. Live nvidia-smi check on 2026-08-06 showed Phi-4 (8000, ~10.4GB) + Mistral (8001, ~6.2GB) already consuming ~16.6GB / 24GB, leaving only ~7.5GB free. Qwen2.5-14B-Instruct Q5_K_M weights alone are ~10-12GB — does not fit concurrently with both existing instances at full GPU offload. The unit is deployed to disk (llm_qwen_service_enabled: false default in defaults/main.yml) but will not start until this is resolved. Options for the shadow-test window, none applied yet — pick one and flip llm_qwen_service_enabled: true:

1. Temporarily stop llama-server-toolcall (8001) for the duration of the shadow test — it's the model being superseded anyway, so this is low-risk and reversible (systemctl start llama-server-toolcall restores it).
2. Reduce Qwen's --n-gpu-layers (partial CPU offload) to fit the ~7.5GB remaining headroom — will materially hurt throughput, not recommended as first choice.
3. Reduce --ctx-size below 65536 — undermines the entire point of this exercise (Hermes's 64K floor), not recommended.

Recommended: option 1, coordinated with Ryan/JARVIS since it does touch a live service, even though 8001 was already flagged for retirement.

Rollback

The existing Gemma llama-server and its GGUF are untouched by every phase of this role. To roll back:

1. systemctl stop llama-server-aux llama-server-toolcall
2. systemctl disable llama-server-aux llama-server-toolcall (optional, if reverting permanently)
3. Confirm the original Gemma service (name determined by discover.yml, commonly llama-server.service) is (still) running: systemctl status llama-server
4. If it was never running because Phase 0 discovered it wasn't a managed unit, whatever ad hoc process/command was used before this role's changes is also unaffected — nothing in this role stopped it.

No files belonging to the existing Gemma deployment (GGUF, unit file, or otherwise) are ever written to or deleted by this role.