homelab/ansible/roles/llm-inference-multimodel

llm-inference-multimodel

Deploys two independent llama-server systemd services on astro-orbiter's RTX 3090 (24GB), alongside — not replacing — the existing llm-inference role:

Instance	Port	Model	Quant	ctx	parallel	~VRAM
llama-server-aux	8000	Phi-4-14B-Instruct	Q4_K_M	8192	2	~10.0GB
llama-server-toolcall	8001	Mistral-Small-24B-Instruct-2501	Q3_K_M	4096	1	~13.2GB

Combined estimate: ~23.2GB / 24GB (~0.8GB headroom). See /home/hermes/astro-orbiter-multi-model-plan.md for the full approved design (VRAM math, model selection rationale, rollback plan, validation harness).

Relationship to roles/llm-inference

This role does not replace llm-inference. It assumes that role's prerequisites are already satisfied on the host:

• NVIDIA driver installed
• /opt/llama.cpp cloned and built with CUDA (/opt/llama.cpp/build/bin/llama-server exists)
• jarvis service user + /home/jarvis present

The pre-existing single-model Gemma llama-server (however it is currently run) is never modified, restarted, or deleted by this role. It is the rollback target.

Phases

Run the whole role, or scope with --tags:

ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml
# or, once merged into a single play:
ansible-playbook -i inventory.yml <playbook>.yml --tags discover,models,systemd,firewall,verify

0. discover (tasks/discover.yml) — READ-ONLY. Confirms via service_facts + pgrep whether the existing Gemma llama-server actually runs as a systemd unit today, or some ad hoc way (nohup/screen/tmux). Does not assume a unit exists — this was an open unknown in the plan and is resolved here as a fact-gathering step, not an assumption. Also records baseline VRAM and current port 8000/8001 listeners for comparison later.

   If this reports no unit found, stop and read the debug message — it means plan §6's rollback story ("systemctl start the old unit to revert") isn't actually available yet, and that should be fixed (codify the existing process as a systemd unit) before proceeding to Phase 2.

1. models (tasks/models.yml) — Idempotent GGUF download to /opt/models/ with a stat + minimum-size guard (mirrors the pattern in roles/llm-inference/tasks/serve.yml and the llm-inference-homelab skill), so reruns don't re-pull 8.5GB/11.7GB files or mistake a truncated partial download for complete.

2. systemd (tasks/systemd.yml) — Templates and deploys both unit files to /etc/systemd/system/. Deliberately does not start or enable either service — units land on disk as a separately reviewable checkpoint. Two fully independent units (not one unit with two ExecStarts) so either instance can be restarted/stopped without affecting the other.

3. firewall (tasks/firewall.yml) — Scopes ports 8000 and 8001 via ufw to llm_allowed_source_cidr (default the Hermes LAN subnet), rather than leaving them open. Both unit templates also bind to llm_bind_address (default 10.1.71.130, the host's private LAN IP) — not 0.0.0.0 — which is a deliberate change from the pre-existing Gemma pattern flagged as insecure in the plan.

4. verify (tasks/verify.yml) — The only phase that actually starts + enables both services. Waits for /health on both ports, smoke-tests /v1/models and a trivial /v1/chat/completions call on each, checks nvidia-smi VRAM usage against the plan's design estimate, and greps dmesg for OOM-kill events.

   This smoke test is not the tool-calling validation harness. See below.

Key variables

Defined in defaults/main.yml (all overridable via host_vars/group_vars or -e):

• llm_service_user (jarvis), llm_binary_path, llm_models_dir, llm_bind_address, llm_allowed_source_cidr
• Aux: llm_aux_port, llm_aux_model_path, llm_aux_model_url, llm_aux_ctx_size, llm_aux_parallel, llm_aux_gpu_layers
• Tool-calling: llm_toolcall_port, llm_toolcall_model_path, llm_toolcall_model_url, llm_toolcall_ctx_size, llm_toolcall_parallel, llm_toolcall_gpu_layers

vars/main.yml holds constants not meant to be overridden per-host (HF token reference, expected-VRAM figures used only for the verify.yml report).

⚠️ Tool-calling validation is required before use

Port 8001 (Mistral-Small-24B) must pass the manual validation procedure described in plan §7 before any Claude Code / tool-calling-capable Hermes profile is pointed at it:

1. A curl-based tool_calls emission probe (does it call tools correctly on known trigger prompts?)
2. A hallucination stress test (does it fabricate tool_calls on prompts that shouldn't trigger any?)
3. A shadow-mode period (run parallel to the existing tool-calling path, compare outputs, before a hard cutover)

This is intentionally not automated into this role — it is a correctness/safety judgment call, not a repeatable infra check. See docs/validation-log.md in this role directory for the procedure reference and a place to log results once Ryan runs it.

Known gap: Semaphore is broken (as of 2026-08-05)

The normal execution/audit path (Semaphore) is currently non-functional. This role was authored to be run via direct ansible-playbook as an accepted interim stopgap, executed personally by Ryan. This is a known gap, not the intended long-term operational path — once Semaphore is repaired, retarget execution of this role (and future changes to it) through Semaphore so runs are audited/logged there again. Flag this in any future work that touches this role.

Rollback

The existing Gemma llama-server and its GGUF are untouched by every phase of this role. To roll back:

1. systemctl stop llama-server-aux llama-server-toolcall
2. systemctl disable llama-server-aux llama-server-toolcall (optional, if reverting permanently)
3. Confirm the original Gemma service (name determined by discover.yml, commonly llama-server.service) is (still) running: systemctl status llama-server
4. If it was never running because Phase 0 discovered it wasn't a managed unit, whatever ad hoc process/command was used before this role's changes is also unaffected — nothing in this role stopped it.

No files belonging to the existing Gemma deployment (GGUF, unit file, or otherwise) are ever written to or deleted by this role.