homelab/ansible/roles/multica_agent_host/README.md

multica_agent_host

Provisions a Multica agent host from bare Linux to a registered, running daemon with its provider runtimes online, and re-runs cleanly on a host that is already provisioned.

Reaches "daemon running, runtimes registered" unattended. The one step it cannot do is the provider CLI's own subscription login, which needs a browser once per host.

If your repo runs ansible-lint

This role keeps the variable names from its spec (multica_version, multica_server_url, multica_workspaces_root, ...), which match the product's own MULTICA_* namespace. ansible-lint's var-naming[no-role-prefix] wants them prefixed with the role name instead — multica_agent_host_multica_version at every call site.

The role ships the exemption at roles/multica_agent_host/.ansible-lint, but ansible-lint only reads that file from the directory you invoke it in. If you lint from your repo root, copy these two lines into your repo's own .ansible-lint:

skip_list:
  - var-naming[no-role-prefix]

Without it a repo-root lint run reports ~78 violations against a role that is otherwise clean at the production profile.

What it does

Step	Handled
Dedicated multica service account (real home, linger)	yes
Node.js, pinned, from nodejs.org with SHASUMS256 verification	yes, when an npm-installed agent CLI is selected
multica CLI, pinned, from the GitHub release with checksums.txt verification	yes
Agent CLIs (claude, hermes, …)	yes
1Password CLI (op), from 1Password's signed apt/yum repository	yes (multica_op_install)
1Password service account token, as a 0600 file for the daemon user	opt-in (multica_op_deliver_token); not exported to the daemon
workspaces_root, on the intended mount, writable by the daemon user	yes
Multica authentication (multica login --token)	yes
CLI config (server_url, app_url, workspaces_root, device_name, …)	yes
Boot autostart (systemd user unit + loginctl enable-linger)	yes
Daemon start	yes
Runtime registration	follows automatically
Provider CLI subscription login (e.g. claude)	no — browser, once per host
Provider CLI credentials from a secret manager	out of scope; the role reports it as a remaining step

Requirements

• Linux target, x86_64 or aarch64.
• systemd, for boot autostart. Without it the CLI falls back to an XDG autostart entry, which does not help a headless host — set multica_autostart: false and supervise the daemon yourself.
• Privilege escalation on the target (become: true).
• Ansible 2.14+.

Secrets

The role does not know where your secrets live. It consumes one plain variable:

multica_pat: "{{ ... }}"   # mul_… user PAT, or mcn_… Cloud Node PAT

The playbook supplies it however your environment already does. In this homelab that means an ansible-vault encrypted var, which is what the example playbook uses; a community.hashi_vault lookup, a 1Password lookup and a plain environment variable are written out beside it as commented alternatives.

See Getting and storing the token below for the end-to-end procedure.

Three consequences worth stating:

1. The role is reusable across environments that do not share a secrets backend.
2. Consolidating providers later is a one-line playbook change; this role is untouched.
3. It sidesteps the HashiCorp Vault / ansible-vault ambiguity entirely — a role that takes a value rather than fetching one works with either today.

no_log: true is set on the login task regardless of where the token comes from. Ansible logs command arguments by default; without it the PAT lands in play output, in any CI log, and in every callback plugin.

This matters just as much with ansible-vault as with a remote lookup: an encrypted var is decrypted in memory for the duration of the play, so encryption at rest does nothing for the runtime path. no_log is what covers it.

Prefer a Cloud Node PAT (mcn_…) over a user PAT where its scope is narrower — a daemon host does not need everything a person can do. Confirm the scope difference against your Multica deployment before deciding.

Getting and storing the token

Minting it

There is no CLI command for this. multica user only manages your profile and multica auth only reports status and logs out; nothing in the command tree mints a token of either kind — the CLI only ever consumes one. A PAT is created in the Multica web app under account settings, in a Tokens section. (The exact menu path could not be confirmed from a headless runtime, so treat that as "where to look", not a click path.)

Two prefixes exist, and multica login --token accepts either:

Prefix	Kind
mul_	User PAT — acts as you, with everything your account can reach
mcn_	"Cloud Node" PAT

On a self-hosted server, check whether mcn_ is offered at all before planning around it. The string mcn_ appears in exactly one place in the shipped CLI — the login --token help text — and the only other place the product uses "Cloud" as a noun is multica setup cloud, which points at Multica Cloud (multica.ai) as opposed to multica setup self-host. That is suggestive, not conclusive: Cloud Node PATs may well be a Multica Cloud concept with no equivalent on a self-hosted deployment. The web UI's token screen is the authority — whatever token types it offers you is the real answer.

Either prefix satisfies the role; preflight validates both.

Scoping a token on a self-hosted deployment

If mul_ is the only kind available, a user PAT on a long-lived daemon host means every run that host claims carries your full authority, and revoking it later cuts off your own access too.

The lever that does exist is a dedicated service member. Workspace roles are owner, admin and member:

multica workspace member list --output json         # see who exists and their roles
multica workspace member invite agents@example.com --role member

Have that account accept the invitation, log in as it, mint its PAT, and give that to the host. member is coarser than a purpose-built node credential, but it is strictly narrower than owner, and revoking it touches nothing you use.

Do not reuse the session token your desktop CLI already holds. It is not a PAT, the server renews it on its own schedule, and it ties the host's identity to your desktop login — revoking one breaks the other.

Storing it with ansible-vault

One-time, if you do not already have a vault password file:

mkdir -p ~/.ansible
openssl rand -base64 48 > ~/.ansible/vault-pass-homelab
chmod 600 ~/.ansible/vault-pass-homelab

Point ansible.cfg at it so no play needs --ask-vault-pass:

[defaults]
vault_password_file = ~/.ansible/vault-pass-homelab

Then create the encrypted var file:

mkdir -p group_vars/multica_agent_hosts
ansible-vault create group_vars/multica_agent_hosts/vault.yml

with a single line in it:

vault_multica_pat: "mul_...."

The playbook already reads multica_pat: "{{ vault_multica_pat }}". The indirection is deliberate: the vault_ prefix makes it obvious at a glance which variables are encrypted, and it keeps the name greppable without decrypting anything.

Alternative, if you would rather not keep a whole encrypted file: encrypt just the string and paste the block into an ordinary group_vars file.

ansible-vault encrypt_string --stdin-name 'vault_multica_pat'

Paste the token, press Ctrl-D. Because it reads stdin, the token never enters your shell history — which ansible-vault encrypt_string 'mul_...' as an argument would.

Checks worth running once

# the file is actually encrypted, not plaintext you forgot to encrypt
head -1 group_vars/multica_agent_hosts/vault.yml   # => $ANSIBLE_VAULT;1.1;AES256

# the var resolves and the role's prefix check will pass, without printing it
ansible -i inventory/hosts.yml multica_agent_hosts -m debug \
  -a 'msg="pat resolves: {{ (vault_multica_pat | default("")) is match("^(mul_|mcn_)") }}"'

That second command prints only true or false, never the token.

Add the vault password file to .gitignore if it lives anywhere near the repo. The encrypted vault.yml is safe to commit; the password file never is.

Ansible owns the version

multica_version is pinned and disable_auto_update is set to true in CLI config. Re-running the play is the upgrade path. Without this the CLI self-updates out from under the role's declared state and the host silently stops matching what the repo says it is.

The same reasoning applies to multica_node_version (full semver, not a major track) and to package_version on each npm-installed agent CLI — left empty by default, which tracks latest; set it to own that version too.

Variables

See defaults/main.yml for the annotated list. The ones you will actually set:

Variable	Default	Notes
multica_server_url	—	Required.
multica_pat	—	Required. See above.
multica_version	0.6.5	Pinned CLI release.
multica_node_version	24.22.1	Pinned, full semver.
multica_workspaces_root	/opt/multica/workspaces	Local disk. Repoint at a mount when one exists.
multica_workspaces_root_mount	""	Set it and the play aborts if the mount is missing.
multica_agent_clis	[claude]	Keys of multica_agent_cli_catalog.
multica_device_name	short inventory hostname	Shows up in runtime list → device_info.
multica_user	multica	Daemon service account.
multica_profile	""	--profile, to isolate config/daemon state/workspaces.
multica_allow_remote_install_scripts	false	Required for hermes; see below.
multica_op_install	true	1Password CLI from 1Password's apt/yum repo.
multica_op_version	""	"" tracks the repo (upgrades on every run); set e.g. 2.41.1 to pin.
multica_op_deliver_token	false	Write multica_op_service_account_token to multica_op_token_file.

Adding an agent CLI

Extend multica_agent_cli_catalog. Two install methods are implemented:

multica_agent_cli_catalog:
  codex:
    provider: codex             # must match `provider` in `multica runtime list`
    binary: codex
    method: npm
    package: "@openai/codex"
    package_version: ""
    version_command: "codex --version"
    manual_login: true
    manual_login_hint: "run `codex login` once as the daemon user"

method: script fetches install_url and runs it as the daemon user. That is a supply-chain decision, so the role refuses unless multica_allow_remote_install_scripts: true. hermes uses this method (https://hermes-agent.nousresearch.com/install.sh). Vendoring the installer into your repo and switching to a local script source is the stricter option.

Agent CLI binaries are symlinked into /usr/local/bin. This matters: the daemon runs from a systemd user unit, which gets systemd's default PATH and does not source /etc/profile.d. A CLI reachable only from an interactive login shell is invisible to the daemon. MULTICA_<PROVIDER>_PATH is the escape hatch if you need the binary somewhere else.

Both install methods resolve the binary with command -v as the daemon user and link that path, rather than assuming where they put it. This matters when a CLI was already installed by other means — the install step is skipped, so assuming the npm prefix would link to nothing. The role asserts the binary is resolvable before linking, and linking the resolved path repairs a dangling link left behind by an earlier run.

Moving the workspaces root onto a mount later

workspaces_root holds per-run working directories, not durable state — the daemon recreates what it needs. Moving it is therefore a drain-and-repoint, not a migration:

multica daemon disk-usage --output json     # see what is actually there first
sudo -iu multica multica daemon stop        # let in-flight runs finish first

Then set multica_workspaces_root and multica_workspaces_root_mount together in the playbook and re-run. The role creates the new directory, proves the daemon user can write to it, updates config set workspaces_root, and restarts the daemon.

Set both variables in the same change. Pointing the root at a mount path without setting the guard means a play that runs while the mount is down silently writes run workspaces onto the root filesystem — which nobody notices until it fills. The guard turns that into a failed assertion instead.

The old directory is left behind deliberately; remove it by hand once the new one has taken runs.

Idempotency

Every step reads before it writes:

Step	Guard
Node install	node --version equals the pin
CLI install	multica --version equals the pin
Agent CLI install	command -v <binary> and its version command
Config	multica config show parsed and compared key by key
Login	multica auth status already names multica_server_url (it prints to stderr; the guard reads both streams)
Autostart	multica daemon autostart status
Daemon start	systemctl --user is-active <unit>; a daemon running outside the unit is stopped first, then the unit is started

A binary, config or credential change notifies a systemctl --user restart handler, which is flushed before verification reads the daemon's state back.

The role never calls multica daemon start or multica daemon restart. Both fork the daemon inside the caller's session (Ansible's SSH session scope), outside the unit's cgroup. The unit then shows inactive, nothing supervises the process, and a later systemctl --user start collides with it on daemon.lock.

A converged host reports zero changed tasks.

Verification

The role ends in assertions, not hope:

• multica daemon status --output json reports status: running, and its cli_version and server_url match the declared values.
• multica runtime list --output json contains an online runtime for every entry in multica_agent_clis, filtered to this host's daemon_id, with device_info naming multica_device_name. Retried, because registration is asynchronous after the daemon starts.
• workspaces_root exists, is a directory, is owned by the daemon user, is actually writable by it (ownership is not writability on a root-squashed network mount), and sits on multica_workspaces_root_mount when that is set.

It finishes by printing the manual steps that remain — the per-host provider CLI logins.

Testing against a throwaway VM

Molecule is fine but a scratch VM is proportionate:

# 1. clean host
ansible-playbook -i inventory/hosts.yml playbooks/provision-agent-host.yml -l scratch-vm

# 2. re-run: must report changed=0
ansible-playbook -i inventory/hosts.yml playbooks/provision-agent-host.yml -l scratch-vm

# 3. reboot, then confirm the daemon came back on its own
ssh scratch-vm 'sudo -iu multica multica daemon status --output json'

# 4. confirm from the workspace side
multica runtime list --output json | jq '.[] | select(.device_info | startswith("scratch-vm"))'

Then complete the browser login for each provider CLI once:

ssh -t scratch-vm 'sudo -iu multica claude'

That step only becomes automatable if the host moves from a subscription to an API key.

1Password CLI

op lets an agent resolve op://vault/item/field references at the moment it runs a command (op run -- cmd, op inject -i tpl -o out, op read), so the secret value never has to appear in a prompt, an issue or a transcript. The 1Password desktop-app integration (and the Claude Code plugin built on it) needs a GUI session, so headless agent hosts use a service account instead.

Install, Debian/Ubuntu, exactly as 1Password documents it, with one addition:

• the key at multica_op_key_url is fingerprint-checked against multica_op_key_fingerprint before it is dearmored, so apt never trusts a substituted key;
• dearmored to /usr/share/keyrings/1password-archive-keyring.gpg (apt, signed-by=) and to /usr/share/debsig/keyrings/AC2D62742012EA22/debsig.gpg with the policy in /etc/debsig/policies/AC2D62742012EA22/ (debsig-verify);
• 1password-cli from downloads.1password.com/linux/debian/<arch> stable.

EL uses 1Password's yum repository with gpgcheck and repo_gpgcheck. The play then runs op --version as the daemon user, in the daemon user's environment.

The service account token

The role can deliver the token, but by default does not:

multica_op_deliver_token: true
multica_op_service_account_token: "{{ vault_op_service_account_tokens[inventory_hostname_short] }}"

writes it to multica_op_token_file (~multica/.config/op-service-account/token, directory 0700, file 0600, owned by the daemon user). That is all it does. The role does not put the token into the daemon's environment, for two reasons:

1. The daemon's systemd user unit is written and owned by the multica CLI (multica daemon autostart). The role must not template or edit it, or it silently freezes the CLI's ExecStart, environment and restart semantics.
2. Anything in the daemon's environment is inherited by every agent process it spawns, where a casual env or printenv puts it in a transcript.

How the token reaches op is a separate decision, not yet made. Until then an agent can use it explicitly:

OP_SERVICE_ACCOUNT_TOKEN="$(cat ~/.config/op-service-account/token)" op run -- <cmd>

op reads the token only from OP_SERVICE_ACCOUNT_TOKEN; it has no token-file option.

Known gaps

• Provider CLI login. Unavoidable for subscription auth.
• Hermes credentials. Hermes reads 1Password natively; wiring that is not implemented here because the right item layout is deployment-specific. The role installs Hermes and reports the credential step as outstanding.
• Non-systemd hosts. Autostart degrades to an XDG entry that a headless host never triggers.

Tags

Tag	Covers
multica_workspaces	workspaces root
multica_node	Node.js
multica_cli	multica CLI
multica_agents	agent CLIs
multica_config	multica config set
multica_auth	multica login
multica_daemon	autostart + start
multica_verify	the assertions
multica_op	1Password CLI, and the token file when enabled
multica	all of the above

Preflight and the service-account step are tagged always: they build _multica_uid and _multica_user_env, which every become_user task below depends on, so a --tags subset still runs them. Both are idempotent.

Offline checks

From the repo root:

ansible-lint playbooks/provision-agent-host.yml roles/multica_agent_host
ansible-playbook --syntax-check playbooks/provision-agent-host.yml
ansible-playbook -i localhost, tests/expressions.yml

tests/expressions.yml runs the role's filter chains — config show parsing, version extraction, the online-runtime difference, the mount containment check — against fixtures captured from a real multica CLI, with no target host. It catches the class of bug where a filter chain silently returns the wrong type or matches nothing; it does not replace the scratch-VM run above.