133 lines
6.4 KiB
Markdown
133 lines
6.4 KiB
Markdown
# llm-inference-multimodel
|
|
|
|
Deploys **two independent llama-server systemd services** on astro-orbiter's
|
|
RTX 3090 (24GB), alongside — not replacing — the existing `llm-inference` role:
|
|
|
|
| Instance | Port | Model | Quant | ctx | parallel | ~VRAM |
|
|
|---|---|---|---|---|---|---|
|
|
| `llama-server-aux` | 8000 | Phi-4-14B-Instruct | Q4_K_M | 8192 | 2 | ~10.0GB |
|
|
| `llama-server-toolcall` | 8001 | Mistral-Small-24B-Instruct-2501 | Q3_K_M | 4096 | 1 | ~13.2GB |
|
|
|
|
Combined estimate: **~23.2GB / 24GB** (~0.8GB headroom). See
|
|
`/home/hermes/astro-orbiter-multi-model-plan.md` for the full approved design
|
|
(VRAM math, model selection rationale, rollback plan, validation harness).
|
|
|
|
## Relationship to `roles/llm-inference`
|
|
|
|
This role does **not** replace `llm-inference`. It assumes that role's
|
|
prerequisites are already satisfied on the host:
|
|
|
|
- NVIDIA driver installed
|
|
- `/opt/llama.cpp` cloned and built with CUDA (`/opt/llama.cpp/build/bin/llama-server` exists)
|
|
- `jarvis` service user + `/home/jarvis` present
|
|
|
|
The pre-existing single-model Gemma llama-server (however it is currently
|
|
run) is **never modified, restarted, or deleted** by this role. It is the
|
|
rollback target.
|
|
|
|
## Phases
|
|
|
|
Run the whole role, or scope with `--tags`:
|
|
|
|
```
|
|
ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml
|
|
# or, once merged into a single play:
|
|
ansible-playbook -i inventory.yml <playbook>.yml --tags discover,models,systemd,firewall,verify
|
|
```
|
|
|
|
0. **discover** (`tasks/discover.yml`) — READ-ONLY. Confirms via
|
|
`service_facts` + `pgrep` whether the existing Gemma llama-server actually
|
|
runs as a systemd unit today, or some ad hoc way (nohup/screen/tmux). Does
|
|
**not** assume a unit exists — this was an open unknown in the plan and is
|
|
resolved here as a fact-gathering step, not an assumption. Also records
|
|
baseline VRAM and current port 8000/8001 listeners for comparison later.
|
|
|
|
**If this reports no unit found**, stop and read the debug message —
|
|
it means plan §6's rollback story ("systemctl start the old unit to
|
|
revert") isn't actually available yet, and that should be fixed (codify
|
|
the existing process as a systemd unit) before proceeding to Phase 2.
|
|
|
|
1. **models** (`tasks/models.yml`) — Idempotent GGUF download to
|
|
`/opt/models/` with a stat + minimum-size guard (mirrors the pattern in
|
|
`roles/llm-inference/tasks/serve.yml` and the `llm-inference-homelab`
|
|
skill), so reruns don't re-pull 8.5GB/11.7GB files or mistake a truncated
|
|
partial download for complete.
|
|
|
|
2. **systemd** (`tasks/systemd.yml`) — Templates and deploys both unit files
|
|
to `/etc/systemd/system/`. **Deliberately does not start or enable either
|
|
service** — units land on disk as a separately reviewable checkpoint.
|
|
Two fully independent units (not one unit with two ExecStarts) so either
|
|
instance can be restarted/stopped without affecting the other.
|
|
|
|
3. **firewall** (`tasks/firewall.yml`) — Scopes ports 8000 and 8001 via `ufw`
|
|
to `llm_allowed_source_cidr` (default the Hermes LAN subnet), rather than
|
|
leaving them open. Both unit templates also bind to
|
|
`llm_bind_address` (default `10.1.71.130`, the host's private LAN IP) —
|
|
**not `0.0.0.0`** — which is a deliberate change from the pre-existing
|
|
Gemma pattern flagged as insecure in the plan.
|
|
|
|
4. **verify** (`tasks/verify.yml`) — The only phase that actually starts +
|
|
enables both services. Waits for `/health` on both ports, smoke-tests
|
|
`/v1/models` and a trivial `/v1/chat/completions` call on each, checks
|
|
`nvidia-smi` VRAM usage against the plan's design estimate, and greps
|
|
`dmesg` for OOM-kill events.
|
|
|
|
**This smoke test is not the tool-calling validation harness.** See
|
|
below.
|
|
|
|
## Key variables
|
|
|
|
Defined in `defaults/main.yml` (all overridable via `host_vars`/`group_vars`
|
|
or `-e`):
|
|
|
|
- `llm_service_user` (jarvis), `llm_binary_path`, `llm_models_dir`, `llm_bind_address`, `llm_allowed_source_cidr`
|
|
- Aux: `llm_aux_port`, `llm_aux_model_path`, `llm_aux_model_url`, `llm_aux_ctx_size`, `llm_aux_parallel`, `llm_aux_gpu_layers`
|
|
- Tool-calling: `llm_toolcall_port`, `llm_toolcall_model_path`, `llm_toolcall_model_url`, `llm_toolcall_ctx_size`, `llm_toolcall_parallel`, `llm_toolcall_gpu_layers`
|
|
|
|
`vars/main.yml` holds constants not meant to be overridden per-host (HF token
|
|
reference, expected-VRAM figures used only for the verify.yml report).
|
|
|
|
## ⚠️ Tool-calling validation is required before use
|
|
|
|
Port 8001 (Mistral-Small-24B) **must** pass the manual validation procedure
|
|
described in plan §7 before any Claude Code / tool-calling-capable Hermes
|
|
profile is pointed at it:
|
|
|
|
1. A curl-based `tool_calls` emission probe (does it call tools correctly on
|
|
known trigger prompts?)
|
|
2. A hallucination stress test (does it fabricate `tool_calls` on prompts
|
|
that shouldn't trigger any?)
|
|
3. A shadow-mode period (run parallel to the existing tool-calling path,
|
|
compare outputs, before a hard cutover)
|
|
|
|
This is **intentionally not automated into this role** — it is a
|
|
correctness/safety judgment call, not a repeatable infra check. See
|
|
`docs/validation-log.md` in this role directory for the procedure reference
|
|
and a place to log results once Ryan runs it.
|
|
|
|
## Known gap: Semaphore is broken (as of 2026-08-05)
|
|
|
|
The normal execution/audit path (Semaphore) is currently non-functional.
|
|
This role was authored to be run via direct `ansible-playbook` as an accepted
|
|
interim stopgap, executed personally by Ryan. **This is a known gap, not the
|
|
intended long-term operational path** — once Semaphore is repaired, retarget
|
|
execution of this role (and future changes to it) through Semaphore so runs
|
|
are audited/logged there again. Flag this in any future work that touches
|
|
this role.
|
|
|
|
## Rollback
|
|
|
|
The existing Gemma llama-server and its GGUF are untouched by every phase of
|
|
this role. To roll back:
|
|
|
|
1. `systemctl stop llama-server-aux llama-server-toolcall`
|
|
2. `systemctl disable llama-server-aux llama-server-toolcall` (optional, if reverting permanently)
|
|
3. Confirm the original Gemma service (name determined by `discover.yml`,
|
|
commonly `llama-server.service`) is (still) running: `systemctl status llama-server`
|
|
4. If it was never running because Phase 0 discovered it wasn't a managed
|
|
unit, whatever ad hoc process/command was used before this role's changes
|
|
is also unaffected — nothing in this role stopped it.
|
|
|
|
No files belonging to the existing Gemma deployment (GGUF, unit file, or
|
|
otherwise) are ever written to or deleted by this role.
|