homelab/ansible/roles/llm-inference-multimodel/README.md

# llm-inference-multimodel

Deploys **two independent llama-server systemd services** on astro-orbiter's
RTX 3090 (24GB), alongside — not replacing — the existing `llm-inference` role:

| Instance | Port | Model | Quant | ctx | parallel | ~VRAM |
|---|---|---|---|---|---|---|
| `llama-server-aux` | 8000 | Phi-4-14B-Instruct | Q4_K_M | 8192 | 2 | ~10.0GB |
| `llama-server-toolcall` | 8001 | Mistral-Small-24B-Instruct-2501 | Q3_K_M | 4096 | 1 | ~13.2GB |

Combined estimate: **~23.2GB / 24GB** (~0.8GB headroom). See
`/home/hermes/astro-orbiter-multi-model-plan.md` for the full approved design
(VRAM math, model selection rationale, rollback plan, validation harness).

## Relationship to `roles/llm-inference`

This role does **not** replace `llm-inference`. It assumes that role's
prerequisites are already satisfied on the host:

- NVIDIA driver installed
- `/opt/llama.cpp` cloned and built with CUDA (`/opt/llama.cpp/build/bin/llama-server` exists)
- `jarvis` service user + `/home/jarvis` present

The pre-existing single-model Gemma llama-server (however it is currently
run) is **never modified, restarted, or deleted** by this role. It is the
rollback target.

## Phases

Run the whole role, or scope with `--tags`:

```
ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml
# or, once merged into a single play:
ansible-playbook -i inventory.yml <playbook>.yml --tags discover,models,systemd,firewall,verify
```

0. **discover** (`tasks/discover.yml`) — READ-ONLY. Confirms via
   `service_facts` + `pgrep` whether the existing Gemma llama-server actually
   runs as a systemd unit today, or some ad hoc way (nohup/screen/tmux). Does
   **not** assume a unit exists — this was an open unknown in the plan and is
   resolved here as a fact-gathering step, not an assumption. Also records
   baseline VRAM and current port 8000/8001 listeners for comparison later.

   **If this reports no unit found**, stop and read the debug message —
   it means plan §6's rollback story ("systemctl start the old unit to
   revert") isn't actually available yet, and that should be fixed (codify
   the existing process as a systemd unit) before proceeding to Phase 2.

1. **models** (`tasks/models.yml`) — Idempotent GGUF download to
   `/opt/models/` with a stat + minimum-size guard (mirrors the pattern in
   `roles/llm-inference/tasks/serve.yml` and the `llm-inference-homelab`
   skill), so reruns don't re-pull 8.5GB/11.7GB files or mistake a truncated
   partial download for complete.

2. **systemd** (`tasks/systemd.yml`) — Templates and deploys both unit files
   to `/etc/systemd/system/`. **Deliberately does not start or enable either
   service** — units land on disk as a separately reviewable checkpoint.
   Two fully independent units (not one unit with two ExecStarts) so either
   instance can be restarted/stopped without affecting the other.

3. **firewall** (`tasks/firewall.yml`) — Scopes ports 8000 and 8001 via `ufw`
   to `llm_allowed_source_cidr` (default the Hermes LAN subnet), rather than
   leaving them open. Both unit templates also bind to
   `llm_bind_address` (default `10.1.71.130`, the host's private LAN IP) —
   **not `0.0.0.0`** — which is a deliberate change from the pre-existing
   Gemma pattern flagged as insecure in the plan.

4. **verify** (`tasks/verify.yml`) — The only phase that actually starts +
   enables both services. Waits for `/health` on both ports, smoke-tests
   `/v1/models` and a trivial `/v1/chat/completions` call on each, checks
   `nvidia-smi` VRAM usage against the plan's design estimate, and greps
   `dmesg` for OOM-kill events.

   **This smoke test is not the tool-calling validation harness.** See
   below.

## Key variables

Defined in `defaults/main.yml` (all overridable via `host_vars`/`group_vars`
or `-e`):

- `llm_service_user` (jarvis), `llm_binary_path`, `llm_models_dir`, `llm_bind_address`, `llm_allowed_source_cidr`
- Aux: `llm_aux_port`, `llm_aux_model_path`, `llm_aux_model_url`, `llm_aux_ctx_size`, `llm_aux_parallel`, `llm_aux_gpu_layers`
- Tool-calling: `llm_toolcall_port`, `llm_toolcall_model_path`, `llm_toolcall_model_url`, `llm_toolcall_ctx_size`, `llm_toolcall_parallel`, `llm_toolcall_gpu_layers`

`vars/main.yml` holds constants not meant to be overridden per-host (HF token
reference, expected-VRAM figures used only for the verify.yml report).

## ⚠️ Tool-calling validation is required before use

Port 8001 (Mistral-Small-24B) **must** pass the manual validation procedure
described in plan §7 before any Claude Code / tool-calling-capable Hermes
profile is pointed at it:

1. A curl-based `tool_calls` emission probe (does it call tools correctly on
   known trigger prompts?)
2. A hallucination stress test (does it fabricate `tool_calls` on prompts
   that shouldn't trigger any?)
3. A shadow-mode period (run parallel to the existing tool-calling path,
   compare outputs, before a hard cutover)

This is **intentionally not automated into this role** — it is a
correctness/safety judgment call, not a repeatable infra check. See
`docs/validation-log.md` in this role directory for the procedure reference
and a place to log results once Ryan runs it.

## Known gap: Semaphore execution path bypassed for this role (2026-08-05)

The normal execution/audit path (Semaphore) was believed non-functional at authoring time, so this role was run via direct `ansible-playbook` instead, executed personally by Ryan.

**Confirmed 2026-08-05 (JARVIS, via Semaphore API — token `vault_semaphore_api_token` in the homelab Ansible vault):** this was a misdiagnosis, not an outage. Semaphore's service, Postgres backend, and API (`/api/ping` returns `pong`) are all healthy on figment (10.1.71.37 — note the documented host `city-hall`/10.1.71.38 is stale; DNS for `imagineering.local.mk-labs.cloud` actually resolves through Traefik on lightning-lane to figment:3000). Queried `/api/project/1/templates` directly: only 6 templates exist project-wide (day0 baseline/root-LV checks, day1 Semaphore self-deploy, Traefik route updates) — **none for this role, nor for the original single-model `llm-inference` role**. Root cause confirmed: no Semaphore project template was ever created for LLM inference deployment, which presents identically to "Semaphore is broken" if you don't check the template list.

**This is still a known gap** — direct `ansible-playbook` execution bypasses the audit trail Semaphore normally provides. Create a project template for this role's playbook and retarget execution through Semaphore so runs are audited/logged there. Flag this in any future work that touches this role.

## Rollback

## Qwen2.5-14B shadow deployment (port 8002) — 2026-08-06

Added a third instance definition (`llama-server-qwen`) per
`/home/hermes/reports/local-llm-64k-context-recommendation.md`, intended to
eventually replace the `llama-server-toolcall` (8001) slot once validated —
runs alongside 8000/8001 during the shadow-test window, does not stop or
replace either.

**VRAM GATE — service NOT started as of this commit.** Live `nvidia-smi` check
on 2026-08-06 showed Phi-4 (8000, ~10.4GB) + Mistral (8001, ~6.2GB) already
consuming ~16.6GB / 24GB, leaving only ~7.5GB free. Qwen2.5-14B-Instruct
Q5_K_M weights alone are ~10-12GB — **does not fit concurrently** with both
existing instances at full GPU offload. The unit is deployed to disk
(`llm_qwen_service_enabled: false` default in `defaults/main.yml`) but will
not start until this is resolved. Options for the shadow-test window,
none applied yet — pick one and flip `llm_qwen_service_enabled: true`:

1. Temporarily stop `llama-server-toolcall` (8001) for the duration of the
   shadow test — it's the model being superseded anyway, so this is low-risk
   and reversible (`systemctl start llama-server-toolcall` restores it).
2. Reduce Qwen's `--n-gpu-layers` (partial CPU offload) to fit the ~7.5GB
   remaining headroom — will materially hurt throughput, not recommended as
   first choice.
3. Reduce `--ctx-size` below 65536 — undermines the entire point of this
   exercise (Hermes's 64K floor), not recommended.

Recommended: option 1, coordinated with Ryan/JARVIS since it does touch a
live service, even though 8001 was already flagged for retirement.

## Rollback

The existing Gemma llama-server and its GGUF are untouched by every phase of
this role. To roll back:

1. `systemctl stop llama-server-aux llama-server-toolcall`
2. `systemctl disable llama-server-aux llama-server-toolcall` (optional, if reverting permanently)
3. Confirm the original Gemma service (name determined by `discover.yml`,
   commonly `llama-server.service`) is (still) running: `systemctl status llama-server`
4. If it was never running because Phase 0 discovered it wasn't a managed
   unit, whatever ad hoc process/command was used before this role's changes
   is also unaffected — nothing in this role stopped it.

No files belonging to the existing Gemma deployment (GGUF, unit file, or
otherwise) are ever written to or deleted by this role.