homelab/ansible/roles/multica_agent_host/README.md

# multica_agent_host

Provisions a Multica agent host from bare Linux to a **registered, running
daemon with its provider runtimes online**, and re-runs cleanly on a host that
is already provisioned.

Reaches "daemon running, runtimes registered" unattended. The one step it
cannot do is the provider CLI's own subscription login, which needs a browser
once per host.

## If your repo runs ansible-lint

This role keeps the variable names from its spec (`multica_version`,
`multica_server_url`, `multica_workspaces_root`, ...), which match the
product's own `MULTICA_*` namespace. ansible-lint's `var-naming[no-role-prefix]`
wants them prefixed with the role name instead — `multica_agent_host_multica_version`
at every call site.

The role ships the exemption at `roles/multica_agent_host/.ansible-lint`, but
**ansible-lint only reads that file from the directory you invoke it in**. If
you lint from your repo root, copy these two lines into your repo's own
`.ansible-lint`:

```yaml
skip_list:
  - var-naming[no-role-prefix]
```

Without it a repo-root lint run reports ~78 violations against a role that is
otherwise clean at the `production` profile.

## What it does

| Step | Handled |
|---|---|
| Dedicated `multica` service account (real home, linger) | yes |
| Node.js, pinned, from nodejs.org with SHASUMS256 verification | yes, when an npm-installed agent CLI is selected |
| `multica` CLI, pinned, from the GitHub release with `checksums.txt` verification | yes |
| Agent CLIs (`claude`, `hermes`, …) | yes |
| 1Password CLI (`op`), from 1Password's signed apt/yum repository | yes (`multica_op_install`) |
| 1Password service account token, as a 0600 file for the daemon user | opt-in (`multica_op_deliver_token`); not exported to the daemon |
| `workspaces_root`, on the intended mount, writable by the daemon user | yes |
| Multica authentication (`multica login --token`) | yes |
| CLI config (`server_url`, `app_url`, `workspaces_root`, `device_name`, …) | yes |
| Boot autostart (systemd user unit + `loginctl enable-linger`) | yes |
| Daemon start | yes |
| Runtime registration | follows automatically |
| **Provider CLI subscription login (e.g. `claude`)** | **no — browser, once per host** |
| Provider CLI credentials from a secret manager | out of scope; the role reports it as a remaining step |

## Requirements

- Linux target, `x86_64` or `aarch64`.
- systemd, for boot autostart. Without it the CLI falls back to an XDG autostart
  entry, which does not help a headless host — set `multica_autostart: false`
  and supervise the daemon yourself.
- Privilege escalation on the target (`become: true`).
- Ansible 2.14+.

## Secrets

**The role does not know where your secrets live.** It consumes one plain
variable:

```yaml
multica_pat: "{{ ... }}"   # mul_… user PAT, or mcn_… Cloud Node PAT
```

The playbook supplies it however your environment already does. In this
homelab that means an **`ansible-vault` encrypted var**, which is what the
example playbook uses; a `community.hashi_vault` lookup, a 1Password lookup and
a plain environment variable are written out beside it as commented
alternatives.

See [Getting and storing the token](#getting-and-storing-the-token) below for
the end-to-end procedure.

Three consequences worth stating:

1. The role is reusable across environments that do not share a secrets backend.
2. Consolidating providers later is a one-line playbook change; this role is
   untouched.
3. It sidesteps the HashiCorp Vault / `ansible-vault` ambiguity entirely — a
   role that takes a value rather than fetching one works with either today.

`no_log: true` is set on the login task regardless of where the token comes
from. Ansible logs command arguments by default; without it the PAT lands in
play output, in any CI log, and in every callback plugin.

This matters just as much with `ansible-vault` as with a remote lookup: an
encrypted var is decrypted in memory for the duration of the play, so
encryption at rest does nothing for the runtime path. `no_log` is what covers
it.

**Prefer a Cloud Node PAT (`mcn_…`) over a user PAT** where its scope is
narrower — a daemon host does not need everything a person can do. Confirm the
scope difference against your Multica deployment before deciding.

## Getting and storing the token

### Minting it

**There is no CLI command for this.** `multica user` only manages your profile
and `multica auth` only reports status and logs out; nothing in the command
tree mints a token of either kind — the CLI only ever consumes one. A PAT is
created in the Multica web app under account settings, in a Tokens section.
(The exact menu path could not be confirmed from a headless runtime, so treat
that as "where to look", not a click path.)

Two prefixes exist, and `multica login --token` accepts either:

| Prefix | Kind |
|---|---|
| `mul_` | User PAT — acts as you, with everything your account can reach |
| `mcn_` | "Cloud Node" PAT |

**On a self-hosted server, check whether `mcn_` is offered at all before
planning around it.** The string `mcn_` appears in exactly one place in the
shipped CLI — the `login --token` help text — and the only other place the
product uses "Cloud" as a noun is `multica setup cloud`, which points at
Multica Cloud (multica.ai) as opposed to `multica setup self-host`. That is
suggestive, not conclusive: Cloud Node PATs may well be a Multica Cloud
concept with no equivalent on a self-hosted deployment. The web UI's token
screen is the authority — whatever token types it offers you is the real
answer.

Either prefix satisfies the role; preflight validates both.

### Scoping a token on a self-hosted deployment

If `mul_` is the only kind available, a user PAT on a long-lived daemon host
means every run that host claims carries your full authority, and revoking it
later cuts off your own access too.

The lever that does exist is a **dedicated service member**. Workspace roles
are `owner`, `admin` and `member`:

```bash
multica workspace member list --output json         # see who exists and their roles
multica workspace member invite agents@example.com --role member
```

Have that account accept the invitation, log in as it, mint *its* PAT, and give
that to the host. `member` is coarser than a purpose-built node credential, but
it is strictly narrower than `owner`, and revoking it touches nothing you use.

Do **not** reuse the session token your desktop CLI already holds. It is not a
PAT, the server renews it on its own schedule, and it ties the host's identity
to your desktop login — revoking one breaks the other.

### Storing it with ansible-vault

One-time, if you do not already have a vault password file:

```bash
mkdir -p ~/.ansible
openssl rand -base64 48 > ~/.ansible/vault-pass-homelab
chmod 600 ~/.ansible/vault-pass-homelab
```

Point `ansible.cfg` at it so no play needs `--ask-vault-pass`:

```ini
[defaults]
vault_password_file = ~/.ansible/vault-pass-homelab
```

Then create the encrypted var file:

```bash
mkdir -p group_vars/multica_agent_hosts
ansible-vault create group_vars/multica_agent_hosts/vault.yml
```

with a single line in it:

```yaml
vault_multica_pat: "mul_...."
```

The playbook already reads `multica_pat: "{{ vault_multica_pat }}"`. The
indirection is deliberate: the `vault_` prefix makes it obvious at a glance
which variables are encrypted, and it keeps the name greppable without
decrypting anything.

**Alternative, if you would rather not keep a whole encrypted file:** encrypt
just the string and paste the block into an ordinary `group_vars` file.

```bash
ansible-vault encrypt_string --stdin-name 'vault_multica_pat'
```

Paste the token, press Ctrl-D. Because it reads stdin, the token never enters
your shell history — which `ansible-vault encrypt_string 'mul_...'` as an
argument would.

### Checks worth running once

```bash
# the file is actually encrypted, not plaintext you forgot to encrypt
head -1 group_vars/multica_agent_hosts/vault.yml   # => $ANSIBLE_VAULT;1.1;AES256

# the var resolves and the role's prefix check will pass, without printing it
ansible -i inventory/hosts.yml multica_agent_hosts -m debug \
  -a 'msg="pat resolves: {{ (vault_multica_pat | default("")) is match("^(mul_|mcn_)") }}"'
```

That second command prints only `true` or `false`, never the token.

Add the vault password file to `.gitignore` if it lives anywhere near the repo.
The encrypted `vault.yml` is safe to commit; the password file never is.

## Ansible owns the version

`multica_version` is pinned and `disable_auto_update` is set to `true` in CLI
config. Re-running the play **is** the upgrade path. Without this the CLI
self-updates out from under the role's declared state and the host silently
stops matching what the repo says it is.

The same reasoning applies to `multica_node_version` (full semver, not a major
track) and to `package_version` on each npm-installed agent CLI — left empty by
default, which tracks `latest`; set it to own that version too.

## Variables

See `defaults/main.yml` for the annotated list. The ones you will actually set:

| Variable | Default | Notes |
|---|---|---|
| `multica_server_url` | — | **Required.** |
| `multica_pat` | — | **Required.** See above. |
| `multica_version` | `0.6.5` | Pinned CLI release. |
| `multica_node_version` | `24.22.1` | Pinned, full semver. |
| `multica_workspaces_root` | `/opt/multica/workspaces` | Local disk. Repoint at a mount when one exists. |
| `multica_workspaces_root_mount` | `""` | Set it and the play aborts if the mount is missing. |
| `multica_agent_clis` | `[claude]` | Keys of `multica_agent_cli_catalog`. |
| `multica_device_name` | short inventory hostname | Shows up in `runtime list` → `device_info`. |
| `multica_user` | `multica` | Daemon service account. |
| `multica_profile` | `""` | `--profile`, to isolate config/daemon state/workspaces. |
| `multica_allow_remote_install_scripts` | `false` | Required for `hermes`; see below. |
| `multica_op_install` | `true` | 1Password CLI from 1Password's apt/yum repo. |
| `multica_op_version` | `""` | `""` tracks the repo (upgrades on every run); set e.g. `2.41.1` to pin. |
| `multica_op_deliver_token` | `false` | Write `multica_op_service_account_token` to `multica_op_token_file`. |

### Adding an agent CLI

Extend `multica_agent_cli_catalog`. Two install methods are implemented:

```yaml
multica_agent_cli_catalog:
  codex:
    provider: codex             # must match `provider` in `multica runtime list`
    binary: codex
    method: npm
    package: "@openai/codex"
    package_version: ""
    version_command: "codex --version"
    manual_login: true
    manual_login_hint: "run `codex login` once as the daemon user"
```

`method: script` fetches `install_url` and runs it as the daemon user. That is a
supply-chain decision, so the role **refuses** unless
`multica_allow_remote_install_scripts: true`. `hermes` uses this method
(`https://hermes-agent.nousresearch.com/install.sh`). Vendoring the installer
into your repo and switching to a local `script` source is the stricter option.

Agent CLI binaries are symlinked into `/usr/local/bin`. This matters: the daemon
runs from a systemd **user unit**, which gets systemd's default `PATH` and does
*not* source `/etc/profile.d`. A CLI reachable only from an interactive login
shell is invisible to the daemon. `MULTICA_<PROVIDER>_PATH` is the escape hatch
if you need the binary somewhere else.

Both install methods resolve the binary with `command -v` as the daemon user
and link *that* path, rather than assuming where they put it. This matters when
a CLI was already installed by other means — the install step is skipped, so
assuming the npm prefix would link to nothing. The role asserts the binary is
resolvable before linking, and linking the resolved path repairs a dangling
link left behind by an earlier run.

## Moving the workspaces root onto a mount later

`workspaces_root` holds per-run working directories, not durable state — the
daemon recreates what it needs. Moving it is therefore a drain-and-repoint, not
a migration:

```bash
multica daemon disk-usage --output json     # see what is actually there first
sudo -iu multica multica daemon stop        # let in-flight runs finish first
```

Then set `multica_workspaces_root` and `multica_workspaces_root_mount` together
in the playbook and re-run. The role creates the new directory, proves the
daemon user can write to it, updates `config set workspaces_root`, and restarts
the daemon.

**Set both variables in the same change.** Pointing the root at a mount path
without setting the guard means a play that runs while the mount is down
silently writes run workspaces onto the root filesystem — which nobody notices
until it fills. The guard turns that into a failed assertion instead.

The old directory is left behind deliberately; remove it by hand once the new
one has taken runs.

## Idempotency

Every step reads before it writes:

| Step | Guard |
|---|---|
| Node install | `node --version` equals the pin |
| CLI install | `multica --version` equals the pin |
| Agent CLI install | `command -v <binary>` and its version command |
| Config | `multica config show` parsed and compared key by key |
| Login | `multica auth status` already names `multica_server_url` (it prints to **stderr**; the guard reads both streams) |
| Autostart | `multica daemon autostart status` |
| Daemon start | `systemctl --user is-active <unit>`; a daemon running *outside* the unit is stopped first, then the unit is started |

A binary, config or credential change notifies a `systemctl --user restart`
handler, which is flushed *before* verification reads the daemon's state back.

The role never calls `multica daemon start` or `multica daemon restart`. Both
fork the daemon inside the caller's session (Ansible's SSH session scope),
outside the unit's cgroup. The unit then shows `inactive`, nothing supervises
the process, and a later `systemctl --user start` collides with it on
`daemon.lock`.

A converged host reports zero changed tasks.

## Verification

The role ends in assertions, not hope:

- `multica daemon status --output json` reports `status: running`, and its
  `cli_version` and `server_url` match the declared values.
- `multica runtime list --output json` contains an **online** runtime for every
  entry in `multica_agent_clis`, filtered to this host's `daemon_id`, with
  `device_info` naming `multica_device_name`. Retried, because registration is
  asynchronous after the daemon starts.
- `workspaces_root` exists, is a directory, is owned by the daemon user, is
  actually writable by it (ownership is not writability on a root-squashed
  network mount), and sits on `multica_workspaces_root_mount` when that is set.

It finishes by printing the manual steps that remain — the per-host provider CLI
logins.

## Testing against a throwaway VM

Molecule is fine but a scratch VM is proportionate:

```bash
# 1. clean host
ansible-playbook -i inventory/hosts.yml playbooks/provision-agent-host.yml -l scratch-vm

# 2. re-run: must report changed=0
ansible-playbook -i inventory/hosts.yml playbooks/provision-agent-host.yml -l scratch-vm

# 3. reboot, then confirm the daemon came back on its own
ssh scratch-vm 'sudo -iu multica multica daemon status --output json'

# 4. confirm from the workspace side
multica runtime list --output json | jq '.[] | select(.device_info | startswith("scratch-vm"))'
```

Then complete the browser login for each provider CLI once:

```bash
ssh -t scratch-vm 'sudo -iu multica claude'
```

That step only becomes automatable if the host moves from a subscription to an
API key.

## 1Password CLI

`op` lets an agent resolve `op://vault/item/field` references at the moment
it runs a command (`op run -- cmd`, `op inject -i tpl -o out`, `op read`), so
the secret value never has to appear in a prompt, an issue or a transcript.
The 1Password desktop-app integration (and the Claude Code plugin built on it)
needs a GUI session, so headless agent hosts use a **service account** instead.

Install, Debian/Ubuntu, exactly as 1Password documents it, with one addition:

- the key at `multica_op_key_url` is fingerprint-checked against
  `multica_op_key_fingerprint` before it is dearmored, so apt never trusts a
  substituted key;
- dearmored to `/usr/share/keyrings/1password-archive-keyring.gpg` (apt,
  `signed-by=`) and to `/usr/share/debsig/keyrings/AC2D62742012EA22/debsig.gpg`
  with the policy in `/etc/debsig/policies/AC2D62742012EA22/` (debsig-verify);
- `1password-cli` from `downloads.1password.com/linux/debian/<arch> stable`.

EL uses 1Password's yum repository with `gpgcheck` and `repo_gpgcheck`.
The play then runs `op --version` as the daemon user, in the daemon user's
environment.

### The service account token

The role can deliver the token, but by default does not:

```yaml
multica_op_deliver_token: true
multica_op_service_account_token: "{{ vault_op_service_account_tokens[inventory_hostname_short] }}"
```

writes it to `multica_op_token_file`
(`~multica/.config/op-service-account/token`, directory 0700, file 0600,
owned by the daemon user). That is all it does. **The role does not put the
token into the daemon's environment**, for two reasons:

1. The daemon's systemd user unit is written and owned by the multica CLI
   (`multica daemon autostart`). The role must not template or edit it, or it
   silently freezes the CLI's `ExecStart`, environment and restart semantics.
2. Anything in the daemon's environment is inherited by every agent process
   it spawns, where a casual `env` or `printenv` puts it in a transcript.

How the token reaches `op` is a separate decision, not yet made. Until then
an agent can use it explicitly:

```bash
OP_SERVICE_ACCOUNT_TOKEN="$(cat ~/.config/op-service-account/token)" op run -- <cmd>
```

`op` reads the token only from `OP_SERVICE_ACCOUNT_TOKEN`; it has no
token-file option.

## Known gaps

- **Provider CLI login.** Unavoidable for subscription auth.
- **Hermes credentials.** Hermes reads 1Password natively; wiring that is not
  implemented here because the right item layout is deployment-specific. The
  role installs Hermes and reports the credential step as outstanding.
- **Non-systemd hosts.** Autostart degrades to an XDG entry that a headless host
  never triggers.

## Tags

| Tag | Covers |
|---|---|
| `multica_workspaces` | workspaces root |
| `multica_node` | Node.js |
| `multica_cli` | multica CLI |
| `multica_agents` | agent CLIs |
| `multica_config` | `multica config set` |
| `multica_auth` | `multica login` |
| `multica_daemon` | autostart + start |
| `multica_verify` | the assertions |
| `multica_op` | 1Password CLI, and the token file when enabled |
| `multica` | all of the above |

Preflight and the service-account step are tagged `always`: they build
`_multica_uid` and `_multica_user_env`, which every `become_user` task below
depends on, so a `--tags` subset still runs them. Both are idempotent.

## Offline checks

From the repo root:

```bash
ansible-lint playbooks/provision-agent-host.yml roles/multica_agent_host
ansible-playbook --syntax-check playbooks/provision-agent-host.yml
ansible-playbook -i localhost, tests/expressions.yml
```

`tests/expressions.yml` runs the role's filter chains — `config show` parsing,
version extraction, the online-runtime difference, the mount containment check
— against fixtures captured from a real `multica` CLI, with no target host.
It catches the class of bug where a filter chain silently returns the wrong
type or matches nothing; it does not replace the scratch-VM run above.