llm-inference-multimodel: role + day1 playbook (phase 0 discover approved)
This commit is contained in:
Hermes Agent service account
132
ansible/roles/llm-inference-multimodel/README.md
Normal file
132
ansible/roles/llm-inference-multimodel/README.md
Normal file
@@ -0,0 +1,132 @@
|
||||
# llm-inference-multimodel
|
||||
|
||||
Deploys **two independent llama-server systemd services** on astro-orbiter's
|
||||
RTX 3090 (24GB), alongside — not replacing — the existing `llm-inference` role:
|
||||
|
||||
| Instance | Port | Model | Quant | ctx | parallel | ~VRAM |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `llama-server-aux` | 8000 | Phi-4-14B-Instruct | Q4_K_M | 8192 | 2 | ~10.0GB |
|
||||
| `llama-server-toolcall` | 8001 | Mistral-Small-24B-Instruct-2501 | Q3_K_M | 4096 | 1 | ~13.2GB |
|
||||
|
||||
Combined estimate: **~23.2GB / 24GB** (~0.8GB headroom). See
|
||||
`/home/hermes/astro-orbiter-multi-model-plan.md` for the full approved design
|
||||
(VRAM math, model selection rationale, rollback plan, validation harness).
|
||||
|
||||
## Relationship to `roles/llm-inference`
|
||||
|
||||
This role does **not** replace `llm-inference`. It assumes that role's
|
||||
prerequisites are already satisfied on the host:
|
||||
|
||||
- NVIDIA driver installed
|
||||
- `/opt/llama.cpp` cloned and built with CUDA (`/opt/llama.cpp/build/bin/llama-server` exists)
|
||||
- `jarvis` service user + `/home/jarvis` present
|
||||
|
||||
The pre-existing single-model Gemma llama-server (however it is currently
|
||||
run) is **never modified, restarted, or deleted** by this role. It is the
|
||||
rollback target.
|
||||
|
||||
## Phases
|
||||
|
||||
Run the whole role, or scope with `--tags`:
|
||||
|
||||
```
|
||||
ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml
|
||||
# or, once merged into a single play:
|
||||
ansible-playbook -i inventory.yml <playbook>.yml --tags discover,models,systemd,firewall,verify
|
||||
```
|
||||
|
||||
0. **discover** (`tasks/discover.yml`) — READ-ONLY. Confirms via
|
||||
`service_facts` + `pgrep` whether the existing Gemma llama-server actually
|
||||
runs as a systemd unit today, or some ad hoc way (nohup/screen/tmux). Does
|
||||
**not** assume a unit exists — this was an open unknown in the plan and is
|
||||
resolved here as a fact-gathering step, not an assumption. Also records
|
||||
baseline VRAM and current port 8000/8001 listeners for comparison later.
|
||||
|
||||
**If this reports no unit found**, stop and read the debug message —
|
||||
it means plan §6's rollback story ("systemctl start the old unit to
|
||||
revert") isn't actually available yet, and that should be fixed (codify
|
||||
the existing process as a systemd unit) before proceeding to Phase 2.
|
||||
|
||||
1. **models** (`tasks/models.yml`) — Idempotent GGUF download to
|
||||
`/opt/models/` with a stat + minimum-size guard (mirrors the pattern in
|
||||
`roles/llm-inference/tasks/serve.yml` and the `llm-inference-homelab`
|
||||
skill), so reruns don't re-pull 8.5GB/11.7GB files or mistake a truncated
|
||||
partial download for complete.
|
||||
|
||||
2. **systemd** (`tasks/systemd.yml`) — Templates and deploys both unit files
|
||||
to `/etc/systemd/system/`. **Deliberately does not start or enable either
|
||||
service** — units land on disk as a separately reviewable checkpoint.
|
||||
Two fully independent units (not one unit with two ExecStarts) so either
|
||||
instance can be restarted/stopped without affecting the other.
|
||||
|
||||
3. **firewall** (`tasks/firewall.yml`) — Scopes ports 8000 and 8001 via `ufw`
|
||||
to `llm_allowed_source_cidr` (default the Hermes LAN subnet), rather than
|
||||
leaving them open. Both unit templates also bind to
|
||||
`llm_bind_address` (default `10.1.71.130`, the host's private LAN IP) —
|
||||
**not `0.0.0.0`** — which is a deliberate change from the pre-existing
|
||||
Gemma pattern flagged as insecure in the plan.
|
||||
|
||||
4. **verify** (`tasks/verify.yml`) — The only phase that actually starts +
|
||||
enables both services. Waits for `/health` on both ports, smoke-tests
|
||||
`/v1/models` and a trivial `/v1/chat/completions` call on each, checks
|
||||
`nvidia-smi` VRAM usage against the plan's design estimate, and greps
|
||||
`dmesg` for OOM-kill events.
|
||||
|
||||
**This smoke test is not the tool-calling validation harness.** See
|
||||
below.
|
||||
|
||||
## Key variables
|
||||
|
||||
Defined in `defaults/main.yml` (all overridable via `host_vars`/`group_vars`
|
||||
or `-e`):
|
||||
|
||||
- `llm_service_user` (jarvis), `llm_binary_path`, `llm_models_dir`, `llm_bind_address`, `llm_allowed_source_cidr`
|
||||
- Aux: `llm_aux_port`, `llm_aux_model_path`, `llm_aux_model_url`, `llm_aux_ctx_size`, `llm_aux_parallel`, `llm_aux_gpu_layers`
|
||||
- Tool-calling: `llm_toolcall_port`, `llm_toolcall_model_path`, `llm_toolcall_model_url`, `llm_toolcall_ctx_size`, `llm_toolcall_parallel`, `llm_toolcall_gpu_layers`
|
||||
|
||||
`vars/main.yml` holds constants not meant to be overridden per-host (HF token
|
||||
reference, expected-VRAM figures used only for the verify.yml report).
|
||||
|
||||
## ⚠️ Tool-calling validation is required before use
|
||||
|
||||
Port 8001 (Mistral-Small-24B) **must** pass the manual validation procedure
|
||||
described in plan §7 before any Claude Code / tool-calling-capable Hermes
|
||||
profile is pointed at it:
|
||||
|
||||
1. A curl-based `tool_calls` emission probe (does it call tools correctly on
|
||||
known trigger prompts?)
|
||||
2. A hallucination stress test (does it fabricate `tool_calls` on prompts
|
||||
that shouldn't trigger any?)
|
||||
3. A shadow-mode period (run parallel to the existing tool-calling path,
|
||||
compare outputs, before a hard cutover)
|
||||
|
||||
This is **intentionally not automated into this role** — it is a
|
||||
correctness/safety judgment call, not a repeatable infra check. See
|
||||
`docs/validation-log.md` in this role directory for the procedure reference
|
||||
and a place to log results once Ryan runs it.
|
||||
|
||||
## Known gap: Semaphore is broken (as of 2026-08-05)
|
||||
|
||||
The normal execution/audit path (Semaphore) is currently non-functional.
|
||||
This role was authored to be run via direct `ansible-playbook` as an accepted
|
||||
interim stopgap, executed personally by Ryan. **This is a known gap, not the
|
||||
intended long-term operational path** — once Semaphore is repaired, retarget
|
||||
execution of this role (and future changes to it) through Semaphore so runs
|
||||
are audited/logged there again. Flag this in any future work that touches
|
||||
this role.
|
||||
|
||||
## Rollback
|
||||
|
||||
The existing Gemma llama-server and its GGUF are untouched by every phase of
|
||||
this role. To roll back:
|
||||
|
||||
1. `systemctl stop llama-server-aux llama-server-toolcall`
|
||||
2. `systemctl disable llama-server-aux llama-server-toolcall` (optional, if reverting permanently)
|
||||
3. Confirm the original Gemma service (name determined by `discover.yml`,
|
||||
commonly `llama-server.service`) is (still) running: `systemctl status llama-server`
|
||||
4. If it was never running because Phase 0 discovered it wasn't a managed
|
||||
unit, whatever ad hoc process/command was used before this role's changes
|
||||
is also unaffected — nothing in this role stopped it.
|
||||
|
||||
No files belonging to the existing Gemma deployment (GGUF, unit file, or
|
||||
otherwise) are ever written to or deleted by this role.
|
||||
Reference in New Issue
Block a user