# llm-inference-multimodel

Deploys **two independent llama-server systemd services** on astro-orbiter's
RTX 3090 (24GB), alongside — not replacing — the existing `llm-inference` role:

| Instance | Port | Model | Quant | ctx | parallel | ~VRAM |
|---|---|---|---|---|---|---|
| `llama-server-aux` | 8000 | Phi-4-14B-Instruct | Q4_K_M | 8192 | 2 | ~10.0GB |
| `llama-server-toolcall` | 8001 | Mistral-Small-24B-Instruct-2501 | Q3_K_M | 4096 | 1 | ~13.2GB |

Combined estimate: **~23.2GB / 24GB** (~0.8GB headroom). See
`/home/hermes/astro-orbiter-multi-model-plan.md` for the full approved design
(VRAM math, model selection rationale, rollback plan, validation harness).

## Relationship to `roles/llm-inference`

This role does **not** replace `llm-inference`. It assumes that role's
prerequisites are already satisfied on the host:

- NVIDIA driver installed
- `/opt/llama.cpp` cloned and built with CUDA (`/opt/llama.cpp/build/bin/llama-server` exists)
- `jarvis` service user + `/home/jarvis` present

The pre-existing single-model Gemma llama-server (however it is currently
run) is **never modified, restarted, or deleted** by this role. It is the
rollback target.

## Phases

Run the whole role, or scope with `--tags`:

```
ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml
# or, once merged into a single play:
ansible-playbook -i inventory.yml <playbook>.yml --tags discover,models,systemd,firewall,verify
```

0. **discover** (`tasks/discover.yml`) — READ-ONLY. Confirms via
   `service_facts` + `pgrep` whether the existing Gemma llama-server actually
   runs as a systemd unit today, or some ad hoc way (nohup/screen/tmux). Does
   **not** assume a unit exists — this was an open unknown in the plan and is
   resolved here as a fact-gathering step, not an assumption. Also records
   baseline VRAM and current port 8000/8001 listeners for comparison later.

   **If this reports no unit found**, stop and read the debug message —
   it means plan §6's rollback story ("systemctl start the old unit to
   revert") isn't actually available yet, and that should be fixed (codify
   the existing process as a systemd unit) before proceeding to Phase 2.

1. **models** (`tasks/models.yml`) — Idempotent GGUF download to
   `/opt/models/` with a stat + minimum-size guard (mirrors the pattern in
   `roles/llm-inference/tasks/serve.yml` and the `llm-inference-homelab`
   skill), so reruns don't re-pull 8.5GB/11.7GB files or mistake a truncated
   partial download for complete.

2. **systemd** (`tasks/systemd.yml`) — Templates and deploys both unit files
   to `/etc/systemd/system/`. **Deliberately does not start or enable either
   service** — units land on disk as a separately reviewable checkpoint.
   Two fully independent units (not one unit with two ExecStarts) so either
   instance can be restarted/stopped without affecting the other.

3. **firewall** (`tasks/firewall.yml`) — Scopes ports 8000 and 8001 via `ufw`
   to `llm_allowed_source_cidr` (default the Hermes LAN subnet), rather than
   leaving them open. Both unit templates also bind to
   `llm_bind_address` (default `10.1.71.130`, the host's private LAN IP) —
   **not `0.0.0.0`** — which is a deliberate change from the pre-existing
   Gemma pattern flagged as insecure in the plan.

4. **verify** (`tasks/verify.yml`) — The only phase that actually starts +
   enables both services. Waits for `/health` on both ports, smoke-tests
   `/v1/models` and a trivial `/v1/chat/completions` call on each, checks
   `nvidia-smi` VRAM usage against the plan's design estimate, and greps
   `dmesg` for OOM-kill events.

   **This smoke test is not the tool-calling validation harness.** See
   below.

## Key variables

Defined in `defaults/main.yml` (all overridable via `host_vars`/`group_vars`
or `-e`):

- `llm_service_user` (jarvis), `llm_binary_path`, `llm_models_dir`, `llm_bind_address`, `llm_allowed_source_cidr`
- Aux: `llm_aux_port`, `llm_aux_model_path`, `llm_aux_model_url`, `llm_aux_ctx_size`, `llm_aux_parallel`, `llm_aux_gpu_layers`
- Tool-calling: `llm_toolcall_port`, `llm_toolcall_model_path`, `llm_toolcall_model_url`, `llm_toolcall_ctx_size`, `llm_toolcall_parallel`, `llm_toolcall_gpu_layers`

`vars/main.yml` holds constants not meant to be overridden per-host (HF token
reference, expected-VRAM figures used only for the verify.yml report).

## ⚠️ Tool-calling validation is required before use

Port 8001 (Mistral-Small-24B) **must** pass the manual validation procedure
described in plan §7 before any Claude Code / tool-calling-capable Hermes
profile is pointed at it:

1. A curl-based `tool_calls` emission probe (does it call tools correctly on
   known trigger prompts?)
2. A hallucination stress test (does it fabricate `tool_calls` on prompts
   that shouldn't trigger any?)
3. A shadow-mode period (run parallel to the existing tool-calling path,
   compare outputs, before a hard cutover)

This is **intentionally not automated into this role** — it is a
correctness/safety judgment call, not a repeatable infra check. See
`docs/validation-log.md` in this role directory for the procedure reference
and a place to log results once Ryan runs it.

## Known gap: Semaphore is broken (as of 2026-08-05)

The normal execution/audit path (Semaphore) is currently non-functional.
This role was authored to be run via direct `ansible-playbook` as an accepted
interim stopgap, executed personally by Ryan. **This is a known gap, not the
intended long-term operational path** — once Semaphore is repaired, retarget
execution of this role (and future changes to it) through Semaphore so runs
are audited/logged there again. Flag this in any future work that touches
this role.

## Rollback

The existing Gemma llama-server and its GGUF are untouched by every phase of
this role. To roll back:

1. `systemctl stop llama-server-aux llama-server-toolcall`
2. `systemctl disable llama-server-aux llama-server-toolcall` (optional, if reverting permanently)
3. Confirm the original Gemma service (name determined by `discover.yml`,
   commonly `llama-server.service`) is (still) running: `systemctl status llama-server`
4. If it was never running because Phase 0 discovered it wasn't a managed
   unit, whatever ad hoc process/command was used before this role's changes
   is also unaffected — nothing in this role stopped it.

No files belonging to the existing Gemma deployment (GGUF, unit file, or
otherwise) are ever written to or deleted by this role.