Content multica creates in inbox/ is owned multica:multica, so obsync fell
into "other" (---) and Sync could neither read nor upload Wong's
submissions. Found after the real Sync sign-in on spaceship-earth; the
earlier verify only tested the obsync -> multica direction.
Shared folders now carry user:obsync:rwx as an access and a default entry
alongside the shared user's, and verify tests the multica -> obsync
direction (read + rewrite, as merge needs). Second run changed=0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The npm-installed Claude Code on spaceship-earth updated itself from the
pinned 2.1.304 to 2.1.308 right after its first login, so every play would
downgrade it and it would update again. New multica_claude_settings is
merged (recursively, not templated, since Claude writes the file too) into
~multica/.claude/settings.json.
provision-agent-host.yml sets DISABLE_AUTOUPDATER=1 and mirrors the model
mix already live on haunted-mansion: model sonnet, subagents haiku,
MAX_THINKING_TOKENS 4000, BASH_MAX_OUTPUT_LENGTH 20000, thinking off.
spaceship-earth: back on 2.1.304 and staying there; second run changed=0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found provisioning spaceship-earth:
- Login was not idempotent: `multica auth status` prints to stderr (0.6.5),
and the guard read stdout only, so every run re-logged in and restarted
the daemon. The guard now reads both streams; the status read is no_log
because it echoes a token prefix.
- `multica daemon start` / `daemon restart` fork the daemon inside the
caller's session (Ansible's SSH session scope), outside the systemd user
unit, which stays inactive: the same orphan seen on haunted-mansion on
2026-10-21. Start/restart now go through `systemctl --user`, after a
daemon-reload when autostart was (re)registered. A daemon found running
outside its unit is stopped first. The unit name is read from
`autostart status`, so --profile hosts resolve their own unit.
spaceship-earth: second run changed=0, daemon in multica-daemon.service.
Not yet run against haunted-mansion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- provision-agent-host.yml: multica_pat now comes from
vault_multica_pats[inventory_hostname_short], one daemon-<host> token per
host. A pre_task asserts the key exists (never prints the value) so a
missing entry fails by host name instead of as a no_log "censored".
- provision-wong-host.yml: spaceship-earth only. Grows the root partition,
PV and LV/filesystem with growpart + community.general.lvg/lvol;
idempotent (second run changed=0). Then applies obsidian_headless.
- roles/obsidian_headless: Obsidian's official headless Sync client
(obsidian-headless 0.0.14) as system user obsync under a sandboxed
systemd unit with UMask=0077. Vault at /srv/obsidian/vault: everpure/
and wip/ obsync-only, inbox/ granted to multica via access + default
ACLs, vault root traverse-only. Verifies the isolation as multica.
Sign-in to Obsidian Sync stays manual (see the role README).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nginx-proxy (new role):
- Shared nginx + certbot under /opt/nginx, replacing the copies that
lived inside the Paperclip compose project
- Apps publish via tasks_from: add_vhost (cert marker -> issue cert ->
write vhost -> nginx -t -> reload); a vhost that fails nginx -t is
rolled back
- Certs issued one per domain via Cloudflare DNS challenge; domains
registered as marker files in cert-domains/; optional LE staging
- Stock nginx.conf plus conf.d (directory mounts, not single-file
mounts); catch-all server drops unknown Host/SNI
- Containers renamed to nginx and certbot
hermes_docker (new role):
- Official Hermes image, host networking, gateway and dashboard bound
to loopback and published through nginx-proxy
- Dashboard login via basic-auth provider, secrets from the vault
- Main model set from hermes_model (fantasmic endpoint, key from vault);
config.yaml overrides are merged from a 0600 staged file that is
removed afterwards, so the key never appears in a command line
- Data dir owner-only (HERMES_HOME_MODE=0700)
paperclip-docker:
- Remove nginx, certbot and docker-compose download tasks; deploy with
community.docker.docker_compose_v2 (idempotent, remove_orphans)
- Stop passing the Cloudflare token and certbot email to the Paperclip
container via env_file
- Bind Postgres to 127.0.0.1 (host networking exposed it on all
interfaces); stop chowning pgdata, which the postgres image owns
- Handler recreates the paperclip service so .env changes apply
- Domain m-v.wdw-labs.cloud -> cranium-command.wdw-labs.cloud
Playbooks:
- Add day1_deploy_nginx_proxy.yml and day1_deploy_hermes_docker.yml
- deploy_paperclip_stack.yml: run nginx-proxy before paperclip-docker,
CLI-based container checks (no Docker SDK dependency)
- Remove deploy-hermes.yml
Host vars: add nginx_proxy_certbot_email and nginx_proxy_certbot_staging
(staging stays on until the switch to real certs).
- Create playbooks/build_paperclip_image.yml to build image locally from repo
- Clone/update paperclipai/paperclip main branch on deploy host
- Build with --target production for lean, current, self-managed image
- Update docker-compose to use paperclip-local:latest instead of reeoss image
- Gives full control over Paperclip version and dependency updates
- OpenCode v1.2.24 local adapter pre-installed in source build
- Enables rapid iteration and custom patches on Paperclip platform
Deployment flow:
1. ansible-playbook playbooks/build_paperclip_image.yml # Build from source
2. ansible-playbook playbooks/deploy_paperclip_stack.yml # Deploy with local image
- Create roles/paperclip-docker with full Docker Compose stack deployment
- Template OpenCode v1.2.24 config (provider structure) with Gemma-4-26B model
- Integrate vLLM endpoint (localhost:8000) with vault-injected API key auth
- Wire secrets from ansible vault: vllm_api_key, cloudflare_token, paperclip auth secrets
- Add pre-deployment checks (PostgreSQL, nginx, certbot cert management)
- Implement idempotent stack lifecycle (containers, volumes, networking)
- Deploy to RTX 3090 via muppet-vision.wdw-labs.cloud with nginx SSL termination
Paperclip now serves as CEO reasoning engine with local Gemma inference,
accessible to Hermes platform agents via OpenCode adapter on shared vLLM.
Set OPENAI_API_BASE_URL (config.py default) to our local vLLM endpoint.
Open WebUI was using hardcoded 'https://api.openai.com/v1' for model discovery.
ExternalSecret only provides vllm-api-key from 1Password.
Using same key for both purposes is acceptable for now.
Avoids need for manual secret patching.
The open-webui item doesn't exist in 1Password yet.
webui-secret-key is manually managed via Kubernetes secret patch.
Template now only pulls vllm-api-key from 1Password.
- Change emptyDir to persistent volume (open-webui-data PVC)
Fixes data loss on pod restart (admin user disappears)
- Add ENABLE_OLLAMA_API=false
Disable broken Ollama endpoint probing
vLLM uses OpenAI-compatible API, not Ollama
- Create pvc.yaml for persistent data storage
10Gi NFS-backed storage via nfs-emporium StorageClass
The 'open-webui' item doesn't exist in 1Password yet.
webui-secret-key is manually patched in the secret for now.
Re-enable ExternalSecret pull once the 1Password item is created.
- Add ExternalName Service for astro-orbiter (10.1.71.130:8000)
- Remove Ollama-specific configuration (OLLAMA_BASE_URLS)
- Keep only OpenAI-compatible configuration (OPENAI_BASE_URL)
Fixes: Open WebUI pod was attempting to use Ollama endpoints (/api/tags)
against vLLM which only supports OpenAI-compatible API (/v1/models).
Pod now reaches astro-orbiter via Kubernetes Service DNS name.
Resolves: HTTP 404 errors from Upstream ollama request failed
rblundon