Add shared nginx-proxy role; deploy Hermes in Docker; rename Paperclip

nginx-proxy (new role):
- Shared nginx + certbot under /opt/nginx, replacing the copies that
  lived inside the Paperclip compose project
- Apps publish via tasks_from: add_vhost (cert marker -> issue cert ->
  write vhost -> nginx -t -> reload); a vhost that fails nginx -t is
  rolled back
- Certs issued one per domain via Cloudflare DNS challenge; domains
  registered as marker files in cert-domains/; optional LE staging
- Stock nginx.conf plus conf.d (directory mounts, not single-file
  mounts); catch-all server drops unknown Host/SNI
- Containers renamed to nginx and certbot

hermes_docker (new role):
- Official Hermes image, host networking, gateway and dashboard bound
  to loopback and published through nginx-proxy
- Dashboard login via basic-auth provider, secrets from the vault
- Main model set from hermes_model (fantasmic endpoint, key from vault);
  config.yaml overrides are merged from a 0600 staged file that is
  removed afterwards, so the key never appears in a command line
- Data dir owner-only (HERMES_HOME_MODE=0700)

paperclip-docker:
- Remove nginx, certbot and docker-compose download tasks; deploy with
  community.docker.docker_compose_v2 (idempotent, remove_orphans)
- Stop passing the Cloudflare token and certbot email to the Paperclip
  container via env_file
- Bind Postgres to 127.0.0.1 (host networking exposed it on all
  interfaces); stop chowning pgdata, which the postgres image owns
- Handler recreates the paperclip service so .env changes apply
- Domain m-v.wdw-labs.cloud -> cranium-command.wdw-labs.cloud

Playbooks:
- Add day1_deploy_nginx_proxy.yml and day1_deploy_hermes_docker.yml
- deploy_paperclip_stack.yml: run nginx-proxy before paperclip-docker,
  CLI-based container checks (no Docker SDK dependency)
- Remove deploy-hermes.yml

Host vars: add nginx_proxy_certbot_email and nginx_proxy_certbot_staging
(staging stays on until the switch to real certs).

ansible/roles/hermes_docker/defaults/main.yml

@@ -0,0 +1,91 @@
+---
+hermes_image: nousresearch/hermes-agent:latest
+# missing = only pull if absent (no surprise upgrades on redeploy).
+# Set to "always" when you actually want to upgrade the image.
+hermes_pull_policy: missing
+
+hermes_container_name: hermes
+hermes_project_dir: /opt/hermes
+hermes_data_dir: "{{ hermes_project_dir }}/data"
+# The container's hermes user. The image's start-up hook chowns the data
+# volume to it; the role only needs the UID to stage files it can read.
+hermes_uid: 10000
+# config.yaml, .env and sessions live here and can hold API keys, so keep it
+# owner-only. Passed to the container as HERMES_HOME_MODE (see compose).
+hermes_data_dir_mode: "0700"
+hermes_secrets_env_file: "{{ hermes_project_dir }}/hermes-secrets.env"
+
+# Same network namespace as Paperclip and nginx
+hermes_network_mode: host
+hermes_memory_limit: 8G
+hermes_cpu_limit: "4.0"
+# Playwright/Chromium needs shared memory (upstream troubleshooting note)
+hermes_shm_size: 1g
+
+# --- Gateway / API server (loopback only; Paperclip reaches it locally) ---
+hermes_gateway_port: 8642
+hermes_api_server_enabled: true
+hermes_api_server_host: 127.0.0.1
+# Leave empty to run keyless on loopback. If you set one, Paperclip (or any
+# other client) must send it as a bearer token.
+# e.g. hermes_api_server_key: "{{ vault_hermes_api_server_key }}"
+hermes_api_server_key: ""
+
+# --- Dashboard (bound to loopback, published through nginx) ---
+hermes_dashboard_enabled: true
+hermes_dashboard_host: 127.0.0.1
+hermes_dashboard_port: 9119
+hermes_domain: haunted-mansion.wdw-labs.cloud
+# Publish the dashboard through the nginx-proxy role (cert + vhost + reload).
+# Requires day1_deploy_nginx_proxy.yml to have run first.
+hermes_publish_vhost: true
+hermes_public_url: "https://{{ hermes_domain }}"
+hermes_dashboard_username: admin
+
+# Vault variables this role requires (see README block at bottom of file)
+hermes_required_vault_vars:
+  - vault_hermes_dashboard_password_hash
+  - vault_hermes_dashboard_secret
+
+# --- Preflight ---
+hermes_preflight_ports:
+  - "{{ hermes_gateway_port }}"
+  - "{{ hermes_dashboard_port }}"
+hermes_wait_timeout: 120
+
+# --- config.yaml overrides (deep-merged, only the keys listed here) ---
+# nginx runs with host networking, so it reaches the dashboard from loopback.
+# Model settings are NOT managed by default (your wizard-written config stays
+# untouched). To manage them, add a model block, for example:
+#   model:
+#     provider: custom
+#     model: <served-model-name>
+#     base_url: http://10.1.71.130:8000/v1
+#     api_key: "{{ vault_vllm_api_key }}"
+hermes_config_overrides:
+  dashboard:
+    trusted_proxies:
+      - "127.0.0.1"
+
+# Main model (written to config.yaml under model:). Leave empty to manage it
+# by hand with `hermes model` or the dashboard. Example:
+#   hermes_model:
+#     provider: custom
+#     default: <served model name from the endpoint's /v1/models>
+#     base_url: http://fantasmic.wdw-labs.cloud:8000/v1
+#     api_key: "{{ vault_fantasmic_api_key }}"
+#     # context_length: 65536   # only if Hermes complains about the context window
+# The api_key lands in config.yaml in plain text (owner-only, see hermes_data_dir_mode).
+hermes_model: {}
+
+# What the merge task actually applies: base overrides + the model block.
+hermes_effective_overrides: >-
+  {{ hermes_config_overrides
+     | combine({'model': hermes_model} if (hermes_model | length > 0) else {}, recursive=True) }}
+
+# Required vault vars:
+#   vault_hermes_dashboard_password_hash  - output of hash_password() (see below)
+#   vault_hermes_dashboard_secret         - openssl rand -base64 32
+# Generate the hash with:
+#   docker exec -it -w /opt/hermes hermes /opt/hermes/.venv/bin/python -c \
+#     "from plugins.dashboard_auth.basic import hash_password; print(hash_password('PW'))"

ansible/roles/hermes_docker/files/merge_config.py

@@ -0,0 +1,40 @@
+"""Deep-merge overrides (a JSON file, path in argv[1]) into config.yaml.
+
+Run inside the hermes container as the hermes user. Only keys present in the
+overrides are touched. Prints CHANGED when the file was rewritten.
+"""
+import json
+import os
+import sys
+
+import yaml
+
+PATH = "/opt/data/config.yaml"
+with open(sys.argv[1]) as fh:
+    overrides = json.load(fh)
+
+if os.path.exists(PATH):
+    with open(PATH) as fh:
+        current = yaml.safe_load(fh) or {}
+else:
+    current = {}
+
+
+def merge(dst, src):
+    changed = False
+    for key, val in src.items():
+        if isinstance(val, dict) and isinstance(dst.get(key), dict):
+            changed |= merge(dst[key], val)
+        elif dst.get(key) != val:
+            dst[key] = val
+            changed = True
+    return changed
+
+
+if merge(current, overrides):
+    with open(PATH, "w") as fh:
+        yaml.safe_dump(current, fh, sort_keys=False)
+    print("CHANGED")
+else:
+    print("UNCHANGED")
+sys.exit(0)

ansible/roles/hermes_docker/handlers/main.yml

@@ -0,0 +1,5 @@
+---
+# Config.yaml is read at start, so a plain restart is enough.
+# Compose/env changes are handled by docker compose itself (it recreates).
+- name: Restart hermes
+  ansible.builtin.command: "docker restart {{ hermes_container_name }}"

ansible/roles/hermes_docker/tasks/main.yml

@@ -0,0 +1,148 @@
+---
+- name: Preflight | Required vault variables are defined
+  ansible.builtin.assert:
+    that: "{{ item }} is defined and ({{ item }} | length) > 0"
+    fail_msg: "{{ item }} is not defined. Add it to your ansible vault."
+    quiet: true
+  loop: "{{ hermes_required_vault_vars }}"
+  tags: [hermes, preflight]
+
+- name: Preflight | Confirm docker CLI is available
+  ansible.builtin.command: docker --version
+  changed_when: false
+  tags: [hermes, preflight]
+
+- name: Preflight | Get existing container state
+  ansible.builtin.command: "docker inspect -f '{% raw %}{{.State.Running}}{% endraw %}' {{ hermes_container_name }}"
+  register: hermes_existing
+  failed_when: false
+  changed_when: false
+  tags: [hermes, preflight]
+
+# Only enforce free ports when our container is not already running,
+# so redeploys stay idempotent.
+- name: Preflight | Ensure required ports are free
+  ansible.builtin.wait_for:
+    host: 127.0.0.1
+    port: "{{ item }}"
+    state: stopped
+    timeout: 3
+  loop: "{{ hermes_preflight_ports }}"
+  when: hermes_existing.stdout | default('') | trim != 'true'
+  tags: [hermes, preflight]
+
+- name: Create project directory
+  ansible.builtin.file:
+    path: "{{ hermes_project_dir }}"
+    state: directory
+    mode: "0755"
+  tags: [hermes]
+
+# No owner here: the image's start-up hook chowns the volume to the hermes user.
+- name: Create data directory (owner-only)
+  ansible.builtin.file:
+    path: "{{ hermes_data_dir }}"
+    state: directory
+    mode: "{{ hermes_data_dir_mode }}"
+  tags: [hermes]
+
+# Compose reads this as root on the host; the container never sees the file.
+- name: Template secrets env file
+  ansible.builtin.template:
+    src: hermes-secrets.env.j2
+    dest: "{{ hermes_secrets_env_file }}"
+    owner: root
+    group: root
+    mode: "0600"
+  no_log: true
+  tags: [hermes]
+
+- name: Template docker compose file
+  ansible.builtin.template:
+    src: docker-compose.yml.j2
+    dest: "{{ hermes_project_dir }}/docker-compose.yml"
+    mode: "0644"
+  tags: [hermes]
+
+- name: Deploy hermes
+  community.docker.docker_compose_v2:
+    project_src: "{{ hermes_project_dir }}"
+    pull: "{{ hermes_pull_policy }}"
+    state: present
+  tags: [hermes]
+
+- name: Verify gateway health
+  ansible.builtin.uri:
+    url: "http://127.0.0.1:{{ hermes_gateway_port }}/health"
+    status_code: 200
+  register: hermes_health
+  until: hermes_health.status == 200
+  retries: "{{ (hermes_wait_timeout | int) // 5 }}"
+  delay: 5
+  when: hermes_api_server_enabled | bool
+  tags: [hermes, verify]
+
+- name: Wait for dashboard port
+  ansible.builtin.wait_for:
+    host: "{{ hermes_dashboard_host }}"
+    port: "{{ hermes_dashboard_port }}"
+    timeout: "{{ hermes_wait_timeout }}"
+  when: hermes_dashboard_enabled | bool
+  tags: [hermes, verify]
+
+# Overrides can contain secrets (the model api_key), so they are staged in a
+# 0600 file owned by the container user instead of travelling in a command
+# line (visible in `ps`), and removed again afterwards. The merge script runs
+# inside the container with the image's own venv (PyYAML is available there),
+# as the hermes user, so config.yaml stays owned by it.
+- name: Apply config.yaml overrides
+  when: hermes_effective_overrides | length > 0
+  tags: [hermes, config]
+  block:
+    - name: Stage overrides file
+      ansible.builtin.copy:
+        content: "{{ hermes_effective_overrides | to_json }}"
+        dest: "{{ hermes_data_dir }}/.ansible-overrides.json"
+        owner: "{{ hermes_uid }}"
+        group: "{{ hermes_uid }}"
+        mode: "0600"
+      changed_when: false   # transient file; only the merge result should count as a change
+      no_log: true
+
+    - name: Merge into config.yaml
+      ansible.builtin.command:
+        argv:
+          - docker
+          - exec
+          - -i
+          - --user
+          - hermes
+          - "{{ hermes_container_name }}"
+          - /opt/hermes/.venv/bin/python
+          - "-"
+          - /opt/data/.ansible-overrides.json
+        stdin: "{{ lookup('ansible.builtin.file', 'merge_config.py') }}"
+      register: hermes_config_merge
+      changed_when: "'CHANGED' in hermes_config_merge.stdout and 'UNCHANGED' not in hermes_config_merge.stdout"
+      notify: Restart hermes
+
+  always:
+    - name: Remove staged overrides file
+      ansible.builtin.file:
+        path: "{{ hermes_data_dir }}/.ansible-overrides.json"
+        state: absent
+      changed_when: false
+
+- name: Publish dashboard through nginx-proxy
+  ansible.builtin.include_role:
+    name: nginx-proxy
+    tasks_from: add_vhost
+  vars:
+    vhost_name: hermes
+    vhost_domain: "{{ hermes_domain }}"
+    vhost_upstream_port: "{{ hermes_dashboard_port }}"
+    vhost_proxy_timeout: 3600s   # long-lived dashboard WebSockets
+  when:
+    - hermes_publish_vhost | bool
+    - hermes_dashboard_enabled | bool
+  tags: [hermes, nginx]

ansible/roles/hermes_docker/templates/docker-compose.yml.j2

@@ -0,0 +1,31 @@
+# Managed by Ansible (hermes_docker role)
+services:
+  hermes:
+    image: {{ hermes_image }}
+    container_name: {{ hermes_container_name }}
+    restart: unless-stopped
+    network_mode: {{ hermes_network_mode }}
+    command: gateway run
+    shm_size: "{{ hermes_shm_size }}"
+    env_file:
+      - {{ hermes_secrets_env_file }}
+    environment:
+      HERMES_HOME_MODE: "{{ hermes_data_dir_mode }}"
+{% if hermes_dashboard_enabled | bool %}
+      HERMES_DASHBOARD: "1"
+      HERMES_DASHBOARD_HOST: "{{ hermes_dashboard_host }}"
+      HERMES_DASHBOARD_PORT: "{{ hermes_dashboard_port }}"
+      HERMES_DASHBOARD_PUBLIC_URL: "{{ hermes_public_url }}"
+{% endif %}
+{% if hermes_api_server_enabled | bool %}
+      API_SERVER_ENABLED: "true"
+      API_SERVER_HOST: "{{ hermes_api_server_host }}"
+      API_SERVER_PORT: "{{ hermes_gateway_port }}"
+{% endif %}
+    volumes:
+      - {{ hermes_data_dir }}:/opt/data
+    deploy:
+      resources:
+        limits:
+          memory: {{ hermes_memory_limit }}
+          cpus: "{{ hermes_cpu_limit }}"

ansible/roles/hermes_docker/templates/hermes-secrets.env.j2

@@ -0,0 +1,8 @@
+# Managed by Ansible (hermes_docker role) - do not edit
+# Single quotes keep compose from interpolating '$' inside hashes/secrets.
+HERMES_DASHBOARD_BASIC_AUTH_USERNAME={{ hermes_dashboard_username }}
+HERMES_DASHBOARD_BASIC_AUTH_PASSWORD_HASH='{{ vault_hermes_dashboard_password_hash }}'
+HERMES_DASHBOARD_BASIC_AUTH_SECRET='{{ vault_hermes_dashboard_secret }}'
+{% if hermes_api_server_key | length > 0 %}
+API_SERVER_KEY='{{ hermes_api_server_key }}'
+{% endif %}

ansible/roles/hermes_docker_backup/defaults/main.yml

@@ -0,0 +1,66 @@
+---
+hermes_image: nousresearch/hermes-agent:latest
+# missing = only pull if absent (no surprise upgrades on redeploy).
+# Set to "always" when you actually want to upgrade the image.
+hermes_pull_policy: missing
+
+hermes_container_name: hermes
+hermes_project_dir: /opt/hermes
+hermes_data_dir: "{{ hermes_project_dir }}/data"
+hermes_secrets_env_file: "{{ hermes_project_dir }}/hermes-secrets.env"
+
+# Same network namespace as Paperclip and nginx
+hermes_network_mode: host
+hermes_memory_limit: 8G
+hermes_cpu_limit: "4.0"
+# Playwright/Chromium needs shared memory (upstream troubleshooting note)
+hermes_shm_size: 1g
+
+# --- Gateway / API server (loopback only; Paperclip reaches it locally) ---
+hermes_gateway_port: 8642
+hermes_api_server_enabled: true
+hermes_api_server_host: 127.0.0.1
+# Leave empty to run keyless on loopback. If you set one, Paperclip (or any
+# other client) must send it as a bearer token.
+# e.g. hermes_api_server_key: "{{ vault_hermes_api_server_key }}"
+hermes_api_server_key: "{{ vault_hermes_api_server_key }}"
+
+# --- Dashboard (bound to loopback, published through nginx) ---
+hermes_dashboard_enabled: true
+hermes_dashboard_host: 127.0.0.1
+hermes_dashboard_port: 9119
+hermes_domain: haunted-mansion.wdw-labs.cloud
+hermes_public_url: "https://{{ hermes_domain }}"
+hermes_dashboard_username: admin
+
+# Vault variables this role requires (see README block at bottom of file)
+hermes_required_vault_vars:
+  - vault_hermes_dashboard_password_hash
+  - vault_hermes_dashboard_secret
+
+# --- Preflight ---
+hermes_preflight_ports:
+  - "{{ hermes_gateway_port }}"
+  - "{{ hermes_dashboard_port }}"
+hermes_wait_timeout: 120
+
+# --- config.yaml overrides (deep-merged, only the keys listed here) ---
+# nginx runs with host networking, so it reaches the dashboard from loopback.
+# Model settings are NOT managed by default (your wizard-written config stays
+# untouched). To manage them, add a model block, for example:
+#   model:
+#     provider: custom
+#     model: <served-model-name>
+#     base_url: http://10.1.71.130:8000/v1
+#     api_key: "{{ vault_vllm_api_key }}"
+hermes_config_overrides:
+  dashboard:
+    trusted_proxies:
+      - "127.0.0.1"
+
+# Required vault vars:
+#   vault_hermes_dashboard_password_hash  - output of hash_password() (see below)
+#   vault_hermes_dashboard_secret         - openssl rand -base64 32
+# Generate the hash with:
+#   docker exec -it -w /opt/hermes hermes /opt/hermes/.venv/bin/python -c \
+#     "from plugins.dashboard_auth.basic import hash_password; print(hash_password('PW'))"

ansible/roles/hermes_docker_backup/files/merge_config.py

@@ -0,0 +1,39 @@
+"""Deep-merge overrides (JSON in $HERMES_CONFIG_OVERRIDES) into config.yaml.
+
+Run inside the hermes container as the hermes user. Only keys present in the
+overrides are touched. Prints CHANGED when the file was rewritten.
+"""
+import json
+import os
+import sys
+
+import yaml
+
+PATH = "/opt/data/config.yaml"
+overrides = json.loads(os.environ["HERMES_CONFIG_OVERRIDES"])
+
+if os.path.exists(PATH):
+    with open(PATH) as fh:
+        current = yaml.safe_load(fh) or {}
+else:
+    current = {}
+
+
+def merge(dst, src):
+    changed = False
+    for key, val in src.items():
+        if isinstance(val, dict) and isinstance(dst.get(key), dict):
+            changed |= merge(dst[key], val)
+        elif dst.get(key) != val:
+            dst[key] = val
+            changed = True
+    return changed
+
+
+if merge(current, overrides):
+    with open(PATH, "w") as fh:
+        yaml.safe_dump(current, fh, sort_keys=False)
+    print("CHANGED")
+else:
+    print("UNCHANGED")
+sys.exit(0)

ansible/roles/hermes_docker_backup/handlers/main.yml

@@ -0,0 +1,5 @@
+---
+# Config.yaml is read at start, so a plain restart is enough.
+# Compose/env changes are handled by docker compose itself (it recreates).
+- name: Restart hermes
+  ansible.builtin.command: "docker restart {{ hermes_container_name }}"

ansible/roles/hermes_docker_backup/nginx-hermes-vhost.conf.j2

@@ -0,0 +1,49 @@
+    # ---- Hermes dashboard: paste inside the http { } block of nginx.conf.j2 ----
+    # Redirect HTTP to HTTPS
+    server {
+        listen 80;
+        server_name {{ hermes_domain }};
+        return 301 https://$server_name$request_uri;
+    }
+
+    server {
+        listen 443 ssl http2;
+        server_name {{ hermes_domain }};
+
+        ssl_certificate /etc/nginx/certs/live/{{ hermes_domain }}/fullchain.pem;
+        ssl_certificate_key /etc/nginx/certs/live/{{ hermes_domain }}/privkey.pem;
+
+        ssl_protocols TLSv1.2 TLSv1.3;
+        ssl_ciphers HIGH:!aNULL:!MD5;
+        ssl_prefer_server_ciphers on;
+        ssl_session_cache shared:SSL:10m;
+        ssl_session_timeout 10m;
+
+        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
+        add_header X-Content-Type-Options "nosniff" always;
+        add_header X-Frame-Options "SAMEORIGIN" always;
+        add_header X-XSS-Protection "1; mode=block" always;
+
+        # No auth_basic here: the dashboard has its own login (basic auth provider).
+        location / {
+            proxy_pass http://127.0.0.1:{{ hermes_dashboard_port }};
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header X-Forwarded-Host $server_name;
+            proxy_set_header X-Forwarded-Port $server_port;
+
+            # WebSocket support
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+
+            proxy_connect_timeout 60s;
+            proxy_send_timeout 3600s;
+            proxy_read_timeout 3600s;
+
+            proxy_buffering off;
+            proxy_request_buffering off;
+        }
+    }

ansible/roles/hermes_docker_backup/tasks/main.yml

@@ -0,0 +1,108 @@
+---
+- name: Preflight | Required vault variables are defined
+  ansible.builtin.assert:
+    that: "{{ item }} is defined and ({{ item }} | length) > 0"
+    fail_msg: "{{ item }} is not defined. Add it to your ansible vault."
+    quiet: true
+  loop: "{{ hermes_required_vault_vars }}"
+  tags: [hermes, preflight]
+
+- name: Preflight | Confirm docker CLI is available
+  ansible.builtin.command: docker --version
+  changed_when: false
+  tags: [hermes, preflight]
+
+- name: Preflight | Get existing container state
+  community.docker.docker_container_info:
+    name: "{{ hermes_container_name }}"
+  register: hermes_existing
+  tags: [hermes, preflight]
+
+# Only enforce free ports when our container is not already running,
+# so redeploys stay idempotent.
+- name: Preflight | Ensure required ports are free
+  ansible.builtin.wait_for:
+    host: 127.0.0.1
+    port: "{{ item }}"
+    state: stopped
+    timeout: 3
+  loop: "{{ hermes_preflight_ports }}"
+  when: not (hermes_existing.container.State.Running | default(false))
+  tags: [hermes, preflight]
+
+- name: Create project and data directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    mode: "0755"
+  loop:
+    - "{{ hermes_project_dir }}"
+    - "{{ hermes_data_dir }}"
+  tags: [hermes]
+
+# Compose reads this as root on the host; the container never sees the file.
+- name: Template secrets env file
+  ansible.builtin.template:
+    src: hermes-secrets.env.j2
+    dest: "{{ hermes_secrets_env_file }}"
+    owner: root
+    group: root
+    mode: "0600"
+  no_log: true
+  tags: [hermes]
+
+- name: Template docker compose file
+  ansible.builtin.template:
+    src: docker-compose.yml.j2
+    dest: "{{ hermes_project_dir }}/docker-compose.yml"
+    mode: "0644"
+  tags: [hermes]
+
+- name: Deploy hermes
+  community.docker.docker_compose_v2:
+    project_src: "{{ hermes_project_dir }}"
+    pull: "{{ hermes_pull_policy }}"
+    state: present
+  tags: [hermes]
+
+- name: Verify gateway health
+  ansible.builtin.uri:
+    url: "http://127.0.0.1:{{ hermes_gateway_port }}/health"
+    status_code: 200
+  register: hermes_health
+  until: hermes_health.status == 200
+  retries: "{{ (hermes_wait_timeout | int) // 5 }}"
+  delay: 5
+  when: hermes_api_server_enabled | bool
+  tags: [hermes, verify]
+
+- name: Wait for dashboard port
+  ansible.builtin.wait_for:
+    host: "{{ hermes_dashboard_host }}"
+    port: "{{ hermes_dashboard_port }}"
+    timeout: "{{ hermes_wait_timeout }}"
+  when: hermes_dashboard_enabled | bool
+  tags: [hermes, verify]
+
+# Runs inside the container with the image's own venv (PyYAML is available
+# there) as the hermes user, so config.yaml stays owned by UID 10000.
+- name: Merge config.yaml overrides
+  ansible.builtin.command:
+    argv:
+      - docker
+      - exec
+      - -i
+      - --user
+      - hermes
+      - -e
+      - "HERMES_CONFIG_OVERRIDES={{ hermes_config_overrides | to_json }}"
+      - "{{ hermes_container_name }}"
+      - /opt/hermes/.venv/bin/python
+      - "-"
+    stdin: "{{ lookup('ansible.builtin.file', 'merge_config.py') }}"
+  register: hermes_config_merge
+  changed_when: "'CHANGED' in hermes_config_merge.stdout and 'UNCHANGED' not in hermes_config_merge.stdout"
+  no_log: true
+  when: hermes_config_overrides | length > 0
+  notify: Restart hermes
+  tags: [hermes, config]

ansible/roles/hermes_docker_backup/templates/docker-compose.yml.j2

@@ -0,0 +1,30 @@
+# Managed by Ansible (hermes_docker role)
+services:
+  hermes:
+    image: {{ hermes_image }}
+    container_name: {{ hermes_container_name }}
+    restart: unless-stopped
+    network_mode: {{ hermes_network_mode }}
+    command: gateway run
+    shm_size: "{{ hermes_shm_size }}"
+    env_file:
+      - {{ hermes_secrets_env_file }}
+    environment:
+{% if hermes_dashboard_enabled | bool %}
+      HERMES_DASHBOARD: "1"
+      HERMES_DASHBOARD_HOST: "{{ hermes_dashboard_host }}"
+      HERMES_DASHBOARD_PORT: "{{ hermes_dashboard_port }}"
+      HERMES_DASHBOARD_PUBLIC_URL: "{{ hermes_public_url }}"
+{% endif %}
+{% if hermes_api_server_enabled | bool %}
+      API_SERVER_ENABLED: "true"
+      API_SERVER_HOST: "{{ hermes_api_server_host }}"
+      API_SERVER_PORT: "{{ hermes_gateway_port }}"
+{% endif %}
+    volumes:
+      - {{ hermes_data_dir }}:/opt/data
+    deploy:
+      resources:
+        limits:
+          memory: {{ hermes_memory_limit }}
+          cpus: "{{ hermes_cpu_limit }}"

ansible/roles/hermes_docker_backup/templates/hermes-secrets.env.j2

@@ -0,0 +1,8 @@
+# Managed by Ansible (hermes_docker role) - do not edit
+# Single quotes keep compose from interpolating '$' inside hashes/secrets.
+HERMES_DASHBOARD_BASIC_AUTH_USERNAME={{ hermes_dashboard_username }}
+HERMES_DASHBOARD_BASIC_AUTH_PASSWORD_HASH='{{ vault_hermes_dashboard_password_hash }}'
+HERMES_DASHBOARD_BASIC_AUTH_SECRET='{{ vault_hermes_dashboard_secret }}'
+{% if hermes_api_server_key | length > 0 %}
+API_SERVER_KEY='{{ hermes_api_server_key }}'
+{% endif %}

ansible/roles/nginx-proxy/README.md

@@ -0,0 +1,28 @@
+# nginx-proxy
+
+Shared nginx + certbot (Cloudflare DNS challenge) under /opt/nginx. Apps publish
+themselves with `tasks_from: add_vhost`; see the header of tasks/add_vhost.yml.
+
+Layout on the host:
+
+    /opt/nginx/
+      docker-compose.yml  .env (0600)
+      conf.d/        00-global.conf + one <app>.conf per vhost   -> /etc/nginx/conf.d
+      certs/         certbot state (live/, archive/, renewal/)   -> /etc/letsencrypt, /etc/nginx/certs
+      cert-domains/  one empty file per domain (filename = domain)
+      scripts/       certbot-init.sh
+
+Required variables: `nginx_proxy_certbot_email`, and `vault_cloudflare_api_token`
+(or `nginx_proxy_cloudflare_api_token`).
+
+Prerequisite (manual): a DNS record for each app domain pointing at this host.
+
+Staging: set `nginx_proxy_certbot_staging: true` to test the DNS challenge without
+using Let's Encrypt rate limits. Staging certs are skipped-as-existing afterwards, so
+to move a domain to a real cert: `docker exec certbot certbot delete --cert-name <domain>`
+and re-run its play.
+
+Retire a domain: delete its cert-domains/ marker and conf.d/<app>.conf, reload nginx,
+then `docker exec certbot certbot delete --cert-name <domain>`.
+
+Pin nginx older than 1.25.1? Set `nginx_proxy_http2: false` (the template uses `http2 on;`).

ansible/roles/nginx-proxy/defaults/main.yml

@@ -0,0 +1,27 @@
+---
+nginx_proxy_base_path: /opt/nginx
+nginx_proxy_certs_dir: "{{ nginx_proxy_base_path }}/certs"
+nginx_proxy_conf_dir: "{{ nginx_proxy_base_path }}/conf.d"
+nginx_proxy_domains_dir: "{{ nginx_proxy_base_path }}/cert-domains"
+nginx_proxy_scripts_dir: "{{ nginx_proxy_base_path }}/scripts"
+
+nginx_proxy_nginx_container: nginx
+nginx_proxy_certbot_container: certbot
+nginx_proxy_nginx_image: nginx:latest
+nginx_proxy_certbot_image: certbot/dns-cloudflare:latest
+
+# Required: set these in group_vars/host_vars (token comes from the vault)
+nginx_proxy_certbot_email: ""
+nginx_proxy_cloudflare_api_token: "{{ vault_cloudflare_api_token }}"
+
+# true = Let's Encrypt staging (untrusted certs, no rate-limit risk). Use while iterating.
+nginx_proxy_certbot_staging: false
+
+# nginx reloads this often so renewed certs get picked up
+nginx_proxy_reload_interval: 6h
+nginx_proxy_client_max_body_size: 100M
+
+# "http2 on;" needs nginx >= 1.25.1 (fine for nginx:latest). Set false if you pin older.
+nginx_proxy_http2: true
+
+nginx_proxy_pull_policy: missing

ansible/roles/nginx-proxy/files/certbot-init.sh

@@ -0,0 +1,92 @@
+#!/bin/sh
+# Issues certificates (one per domain) via the Cloudflare DNS challenge.
+#
+#   certbot-init.sh                 -> every domain with a marker file in $DOMAINS_DIR
+#   certbot-init.sh a.example.com   -> only the domains given as arguments
+#
+# Idempotent: domains that already have a certificate are skipped.
+# Renewal is NOT handled here; `certbot renew` covers every existing lineage.
+set -eu
+
+DOMAINS_DIR="${DOMAINS_DIR:-/etc/cert-domains.d}"
+EMAIL="${CERTBOT_EMAIL:-}"
+TOKEN="${CLOUDFLARE_API_TOKEN:-}"
+
+STAGING_ARG=""
+if [ "${CERTBOT_STAGING:-0}" = "1" ]; then
+    STAGING_ARG="--server https://acme-staging-v02.api.letsencrypt.org/directory"
+    echo "NOTE: using the Let's Encrypt STAGING server (certificates will not be trusted)"
+fi
+
+if [ "$#" -gt 0 ]; then
+    DOMAINS="$*"
+elif [ -d "$DOMAINS_DIR" ]; then
+    DOMAINS=$(ls -1 "$DOMAINS_DIR" | grep -E '^[A-Za-z0-9]([A-Za-z0-9.-]*[A-Za-z0-9])?\.[A-Za-z]{2,}$' || true)
+else
+    DOMAINS=""
+fi
+DOMAINS=$(printf '%s\n' "$DOMAINS" | tr ', \t' '\n\n\n' | grep -v '^$' | sort -u || true)
+
+if [ -z "$DOMAINS" ]; then
+    echo "No domains registered in $DOMAINS_DIR; nothing to do"
+    exit 0
+fi
+
+if [ -z "$EMAIL" ]; then
+    echo "ERROR: CERTBOT_EMAIL not set"
+    exit 1
+fi
+if [ -z "$TOKEN" ]; then
+    echo "ERROR: CLOUDFLARE_API_TOKEN not set"
+    exit 1
+fi
+
+# Temporary credentials file, shared by all requests and always cleaned up
+CREDS_FILE=$(mktemp /tmp/cloudflare.XXXXXX)
+trap 'rm -f "$CREDS_FILE"' EXIT
+chmod 600 "$CREDS_FILE"
+printf 'dns_cloudflare_api_token = %s\n' "$TOKEN" > "$CREDS_FILE"
+
+FAILED=""
+ISSUED=0
+SKIPPED=0
+
+for DOMAIN in $DOMAINS; do
+    CERT_PATH="/etc/letsencrypt/live/${DOMAIN}/fullchain.pem"
+
+    if [ -f "$CERT_PATH" ]; then
+        echo "Certificate for $DOMAIN already exists at $CERT_PATH"
+        SKIPPED=$((SKIPPED + 1))
+        continue
+    fi
+
+    echo "Requesting certificate for $DOMAIN via Cloudflare DNS challenge..."
+
+    # --cert-name pins the lineage directory to the domain name so the
+    # live/<domain>/ path nginx expects never gets a -0001 suffix.
+    # STAGING_ARG is intentionally unquoted (it expands to two words).
+    # shellcheck disable=SC2086
+    if certbot certonly \
+        --dns-cloudflare \
+        --dns-cloudflare-credentials "$CREDS_FILE" \
+        --agree-tos \
+        --no-eff-email \
+        --email "$EMAIL" \
+        --non-interactive \
+        --cert-name "$DOMAIN" \
+        $STAGING_ARG \
+        -d "$DOMAIN" && [ -f "$CERT_PATH" ]; then
+        echo "Certificate obtained and verified for $DOMAIN at $CERT_PATH"
+        ISSUED=$((ISSUED + 1))
+    else
+        echo "ERROR: Failed to obtain certificate for $DOMAIN"
+        FAILED="$FAILED $DOMAIN"
+    fi
+done
+
+echo "Summary: $ISSUED issued, $SKIPPED already present"
+
+if [ -n "$FAILED" ]; then
+    echo "ERROR: certificate request failed for:$FAILED"
+    exit 1
+fi

ansible/roles/nginx-proxy/handlers/main.yml

@@ -0,0 +1,6 @@
+---
+- name: Reload nginx
+  ansible.builtin.shell: >
+    docker exec {{ nginx_proxy_nginx_container }} nginx -t &&
+    docker exec {{ nginx_proxy_nginx_container }} nginx -s reload
+  changed_when: true

ansible/roles/nginx-proxy/tasks/add_vhost.yml

@@ -0,0 +1,118 @@
+---
+# Publish an app through nginx-proxy. Usage:
+#
+#   - ansible.builtin.include_role:
+#       name: nginx-proxy
+#       tasks_from: add_vhost
+#     vars:
+#       vhost_name: paperclip              # required, becomes conf.d/<name>.conf
+#       vhost_domain: "{{ paperclip_domain }}"   # required
+#       vhost_upstream_port: 3100          # required
+#       vhost_upstream_host: 127.0.0.1     # optional
+#       vhost_proxy_timeout: 60s           # optional (use 3600s for WebSockets)
+#       vhost_health_path: /health         # optional, proxied without access log
+#
+# Order: marker -> certificate -> vhost file -> nginx -t -> reload.
+# The vhost is only written once the certificate exists, and a vhost that
+# fails nginx -t is rolled back, so one bad app cannot take the others down.
+- name: Validate vhost parameters
+  ansible.builtin.assert:
+    that:
+      - vhost_name is defined
+      - vhost_domain is defined
+      - vhost_upstream_port is defined
+    fail_msg: "add_vhost needs vhost_name, vhost_domain and vhost_upstream_port"
+    quiet: true
+
+- name: Check that the nginx-proxy containers are running
+  ansible.builtin.command: "docker inspect -f '{% raw %}{{.State.Running}}{% endraw %}' {{ item }}"
+  register: nginx_proxy_running
+  failed_when: nginx_proxy_running.rc != 0 or nginx_proxy_running.stdout | trim != 'true'
+  changed_when: false
+  loop:
+    - "{{ nginx_proxy_nginx_container }}"
+    - "{{ nginx_proxy_certbot_container }}"
+  loop_control:
+    label: "{{ item }}"
+
+- name: "Register cert marker for {{ vhost_domain }}"
+  ansible.builtin.file:
+    path: "{{ nginx_proxy_domains_dir }}/{{ vhost_domain }}"
+    state: touch
+    modification_time: preserve
+    access_time: preserve
+    mode: "0644"
+
+- name: "Check for existing certificate for {{ vhost_domain }}"
+  ansible.builtin.stat:
+    path: "{{ nginx_proxy_certs_dir }}/live/{{ vhost_domain }}/fullchain.pem"
+  register: nginx_proxy_cert_before
+
+# Targeted run: only this domain, so another app's failed issuance
+# cannot fail this play.
+- name: "Issue certificate for {{ vhost_domain }}"
+  ansible.builtin.command: >-
+    docker exec {{ nginx_proxy_certbot_container }}
+    /scripts/certbot-init.sh {{ vhost_domain }}
+  register: nginx_proxy_cert_issue
+  changed_when: "'Certificate obtained' in nginx_proxy_cert_issue.stdout"
+  when: not nginx_proxy_cert_before.stat.exists
+
+- name: "Confirm certificate exists for {{ vhost_domain }}"
+  ansible.builtin.stat:
+    path: "{{ nginx_proxy_certs_dir }}/live/{{ vhost_domain }}/fullchain.pem"
+  register: nginx_proxy_cert_after
+  failed_when: not nginx_proxy_cert_after.stat.exists
+
+- name: "Write {{ vhost_name }} vhost, test, and reload"
+  block:
+    - name: "Template conf.d/{{ vhost_name }}.conf"
+      ansible.builtin.template:
+        src: vhost.conf.j2
+        dest: "{{ nginx_proxy_conf_dir }}/{{ vhost_name }}.conf"
+        mode: "0644"
+        backup: true
+      register: nginx_proxy_vhost_file
+
+    - name: Test nginx config
+      ansible.builtin.command: "docker exec {{ nginx_proxy_nginx_container }} nginx -t"
+      changed_when: false
+
+    - name: Reload nginx
+      ansible.builtin.command: "docker exec {{ nginx_proxy_nginx_container }} nginx -s reload"
+      when: nginx_proxy_vhost_file.changed
+
+    - name: Remove superseded backup
+      ansible.builtin.file:
+        path: "{{ nginx_proxy_vhost_file.backup_file }}"
+        state: absent
+      when: nginx_proxy_vhost_file.backup_file is defined
+
+  rescue:
+    - name: Restore previous vhost
+      ansible.builtin.copy:
+        src: "{{ nginx_proxy_vhost_file.backup_file }}"
+        dest: "{{ nginx_proxy_conf_dir }}/{{ vhost_name }}.conf"
+        remote_src: true
+        mode: "0644"
+      when: nginx_proxy_vhost_file.backup_file | default('') | length > 0
+
+    - name: Remove backup after restoring it
+      ansible.builtin.file:
+        path: "{{ nginx_proxy_vhost_file.backup_file }}"
+        state: absent
+      when: nginx_proxy_vhost_file.backup_file | default('') | length > 0
+
+    - name: Remove the new vhost (there was no previous one)
+      ansible.builtin.file:
+        path: "{{ nginx_proxy_conf_dir }}/{{ vhost_name }}.conf"
+        state: absent
+      when:
+        - nginx_proxy_vhost_file is defined
+        - nginx_proxy_vhost_file.backup_file | default('') | length == 0
+
+    - name: Fail with context
+      ansible.builtin.fail:
+        msg: >-
+          nginx rejected the {{ vhost_name }} vhost; it was rolled back and
+          the running config is unchanged.

ansible/roles/nginx-proxy/tasks/main.yml

@@ -0,0 +1,70 @@
+---
+- name: Assert required variables
+  ansible.builtin.assert:
+    that:
+      - nginx_proxy_certbot_email | length > 0
+      - nginx_proxy_cloudflare_api_token | length > 0
+    fail_msg: >-
+      Set nginx_proxy_certbot_email, and make sure vault_cloudflare_api_token
+      (or nginx_proxy_cloudflare_api_token) is defined.
+    quiet: true
+  tags: [nginx-proxy, preflight]
+
+- name: Create directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    mode: "0755"
+  loop:
+    - "{{ nginx_proxy_base_path }}"
+    - "{{ nginx_proxy_certs_dir }}"
+    - "{{ nginx_proxy_conf_dir }}"
+    - "{{ nginx_proxy_domains_dir }}"
+    - "{{ nginx_proxy_scripts_dir }}"
+  tags: [nginx-proxy]
+
+- name: Install certbot script
+  ansible.builtin.copy:
+    src: certbot-init.sh
+    dest: "{{ nginx_proxy_scripts_dir }}/certbot-init.sh"
+    mode: "0755"
+  tags: [nginx-proxy]
+
+- name: Template global nginx settings
+  ansible.builtin.template:
+    src: 00-global.conf.j2
+    dest: "{{ nginx_proxy_conf_dir }}/00-global.conf"
+    mode: "0644"
+  notify: Reload nginx
+  tags: [nginx-proxy]
+
+- name: Template compose env file
+  ansible.builtin.template:
+    src: env.j2
+    dest: "{{ nginx_proxy_base_path }}/.env"
+    mode: "0600"
+  no_log: true
+  tags: [nginx-proxy]
+
+- name: Template compose file
+  ansible.builtin.template:
+    src: docker-compose.yml.j2
+    dest: "{{ nginx_proxy_base_path }}/docker-compose.yml"
+    mode: "0644"
+  tags: [nginx-proxy]
+
+- name: Deploy nginx and certbot
+  community.docker.docker_compose_v2:
+    project_src: "{{ nginx_proxy_base_path }}"
+    pull: "{{ nginx_proxy_pull_policy }}"
+    state: present
+  tags: [nginx-proxy]
+
+- name: Verify nginx config is valid and the container is serving
+  ansible.builtin.command: "docker exec {{ nginx_proxy_nginx_container }} nginx -t"
+  register: nginx_proxy_test
+  until: nginx_proxy_test.rc == 0
+  retries: 6
+  delay: 5
+  changed_when: false
+  tags: [nginx-proxy, verify]

ansible/roles/nginx-proxy/templates/00-global.conf.j2

@@ -0,0 +1,31 @@
+# Managed by Ansible (nginx-proxy role)
+# Included inside the stock nginx.conf http { } block, so these are http-level.
+server_tokens off;
+client_max_body_size {{ nginx_proxy_client_max_body_size }};
+
+ssl_protocols TLSv1.2 TLSv1.3;
+ssl_ciphers HIGH:!aNULL:!MD5;
+ssl_prefer_server_ciphers on;
+ssl_session_cache shared:SSL:10m;
+ssl_session_timeout 10m;
+
+# Catch-all: unknown Host headers get nothing, and unknown SNI is rejected
+# before any certificate is presented.
+server {
+    listen 80 default_server;
+    server_name _;
+    access_log off;
+
+    location = /nginx-health {
+        return 200 "ok\n";
+    }
+    location / {
+        return 444;
+    }
+}
+
+server {
+    listen 443 ssl default_server;
+    server_name _;
+    ssl_reject_handshake on;
+}

ansible/roles/nginx-proxy/templates/docker-compose.yml.j2

@@ -0,0 +1,46 @@
+# Managed by Ansible (nginx-proxy role)
+services:
+  certbot:
+    image: {{ nginx_proxy_certbot_image }}
+    container_name: {{ nginx_proxy_certbot_container }}
+    volumes:
+      - {{ nginx_proxy_certs_dir }}:/etc/letsencrypt
+      - {{ nginx_proxy_scripts_dir }}:/scripts:ro
+      - {{ nginx_proxy_domains_dir }}:/etc/cert-domains.d:ro
+    environment:
+      CLOUDFLARE_API_TOKEN: ${CLOUDFLARE_API_TOKEN}
+      CERTBOT_EMAIL: ${CERTBOT_EMAIL}
+      CERTBOT_STAGING: "{{ '1' if nginx_proxy_certbot_staging | bool else '0' }}"
+    entrypoint: /bin/sh
+    command:
+      - -c
+      - |
+        while true; do
+          /scripts/certbot-init.sh || echo "issuance failed; will retry next cycle"
+          certbot renew --quiet
+          sleep 12h
+        done
+    restart: unless-stopped
+
+  nginx:
+    image: {{ nginx_proxy_nginx_image }}
+    container_name: {{ nginx_proxy_nginx_container }}
+    network_mode: host
+    volumes:
+      # Directory mounts (not single files) so Ansible's atomic file
+      # replacement is visible inside the container.
+      - {{ nginx_proxy_conf_dir }}:/etc/nginx/conf.d:ro
+      - {{ nginx_proxy_certs_dir }}:/etc/nginx/certs:ro
+    command:
+      - /bin/sh
+      - -c
+      - |
+        ( while :; do sleep {{ nginx_proxy_reload_interval }}; nginx -s reload; done ) &
+        exec nginx -g 'daemon off;'
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost/nginx-health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 30s

ansible/roles/nginx-proxy/templates/env.j2

@@ -0,0 +1,3 @@
+# Managed by Ansible (nginx-proxy role)
+CLOUDFLARE_API_TOKEN={{ nginx_proxy_cloudflare_api_token }}
+CERTBOT_EMAIL={{ nginx_proxy_certbot_email }}

ansible/roles/nginx-proxy/templates/vhost.conf.j2

@@ -0,0 +1,51 @@
+# Managed by Ansible (nginx-proxy role) - vhost: {{ vhost_name }}
+server {
+    listen 80;
+    server_name {{ vhost_domain }};
+    return 301 https://{{ vhost_domain }}$request_uri;
+}
+
+server {
+    listen 443 ssl;
+{% if nginx_proxy_http2 | bool %}
+    http2 on;
+{% endif %}
+    server_name {{ vhost_domain }};
+
+    ssl_certificate /etc/nginx/certs/live/{{ vhost_domain }}/fullchain.pem;
+    ssl_certificate_key /etc/nginx/certs/live/{{ vhost_domain }}/privkey.pem;
+
+    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
+    add_header X-Content-Type-Options "nosniff" always;
+    add_header X-Frame-Options "SAMEORIGIN" always;
+    add_header X-XSS-Protection "1; mode=block" always;
+
+    location / {
+        proxy_pass http://{{ vhost_upstream_host | default('127.0.0.1') }}:{{ vhost_upstream_port }};
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-Host $host;
+        proxy_set_header X-Forwarded-Port $server_port;
+
+        # WebSocket support
+        proxy_http_version 1.1;
+        proxy_set_header Upgrade $http_upgrade;
+        proxy_set_header Connection "upgrade";
+
+        proxy_connect_timeout 60s;
+        proxy_send_timeout {{ vhost_proxy_timeout | default('60s') }};
+        proxy_read_timeout {{ vhost_proxy_timeout | default('60s') }};
+
+        proxy_buffering off;
+        proxy_request_buffering off;
+    }
+{% if vhost_health_path | default('') | length > 0 %}
+
+    location {{ vhost_health_path }} {
+        proxy_pass http://{{ vhost_upstream_host | default('127.0.0.1') }}:{{ vhost_upstream_port }}{{ vhost_health_path }};
+        access_log off;
+    }
+{% endif %}
+}

ansible/roles/paperclip-docker/defaults/main.yml

@@ -2,31 +2,30 @@
 # Paperclip deployment paths and configuration
 paperclip_base_path: /opt/paperclip
 paperclip_data_dir: "{{ paperclip_base_path }}/data"
-paperclip_certs_dir: "{{ paperclip_base_path }}/certs"
 paperclip_port: 3100
-paperclip_domain: muppet-vision.wdw-labs.cloud
-paperclip_host: muppet-vision.wdw-labs.cloud
+paperclip_domain: cranium-command.wdw-labs.cloud
+paperclip_host: cranium-command.wdw-labs.cloud
+# paperclip_listen_host: 127.0.0.1 # Bootstrap local deployment
+paperclip_listen_host: 0.0.0.0 # Switch to this host after CEO is functional on local model
 
 # Paperclip allowed hostnames (space or comma-separated)
 # Includes: domain name, localhost (for internal/local access), and client IP
 paperclip_allowed_hostnames: "{{ paperclip_domain }} localhost 192.168.10.50"
 
 # Deployment mode: authenticated (requires login) or local_trusted (no auth, trusted network)
-paperclip_deployment_mode: "authenticated"
+# paperclip_deployment_mode: "local_trusted" # Bootstrap local deployment
+paperclip_deployment_mode: "authenticated" # Switch to this model after CEO is functional on local model
 
-# Reverse proxy configuration
-nginx_port: 443
-nginx_http_port: 80
+# TLS / reverse proxy are handled by the nginx-proxy role (/opt/nginx).
+# Set nginx_proxy_certbot_email in host_vars; run day1_deploy_nginx_proxy.yml first.
 
-# HTTPS/SSL configuration
-certbot_email: ryan.blundon@protonmail.com
-certbot_provider: cloudflare
-
-# Inference endpoint (vLLM or remote OpenAI-compatible)
-vllm_endpoint: "http://127.0.0.1:8000"
+# Inference endpoints (vLLM or remote OpenAI-compatible)
+illuminations_endpoint: "http://illuminations:8000"
 fantasmic_endpoint: "http://fantasmic.wdw-labs.cloud:8000"
-wishes_endpoint: "http://fantasmic.wdw-labs.cloud:8000"
+wishes_endpoint: "http://wishes.wdw-labs.cloud:8000"
 
-# NOTE: Authentication secrets (BETTER_AUTH_SECRET, PAPERCLIP_AGENT_JWT_SECRET)
-#       are auto-generated during deployment and stored in .env (mode 0600)
-#       No need to specify them here — they're created with: openssl rand -hex 24
+# NOTE: Secrets come from the ansible vault and are written to .env (mode 0600):
+#   vault_better_auth_secret, vault_paperclip_agent_jwt_secret,
+#   vault_illuminations_api_key, vault_fantasmic_api_key
+# Everything in .env is passed into the Paperclip container (env_file), so only
+# put values here that Paperclip itself should be able to read.

ansible/roles/paperclip-docker/handlers/main.yml

@@ -1,19 +1,12 @@
 ---
+# Recreate (not just restart) so .env / env_file changes are actually applied.
+# Skipped when the deploy task already created or recreated containers this run.
 - name: Restart paperclip stack
-  shell: |
-    cd {{ paperclip_base_path }}
-    /usr/local/bin/docker-compose restart
-  become: yes
-
-- name: Reload nginx
-  shell: |
-    docker exec paperclip-nginx nginx -s reload
-  become: yes
-  ignore_errors: yes
-
-- name: Reload certbot
-  shell: |
-    docker restart paperclip-certbot
-  become: yes
-  ignore_errors: yes
-  
+  community.docker.docker_compose_v2:
+    project_src: "{{ paperclip_base_path }}"
+    services:
+      - paperclip
+    state: present
+    recreate: always
+  become: true
+  when: not (paperclip_deploy.changed | default(false))

ansible/roles/paperclip-docker/tasks/main.yml

@@ -1,27 +1,6 @@
 ---
-- name: Install Docker SDK for Python
-  apt:
-    name: python3-docker
-    state: present
-  become: true
-
-- name: Download latest docker-compose from GitHub
-  shell: |
-    curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
-    chmod +x /usr/local/bin/docker-compose
-  become: true
-
-- name: Verify docker-compose installation
-  shell: /usr/local/bin/docker-compose --version
-  register: compose_version
-  become: true
-
-- name: Show docker-compose version
-  debug:
-    msg: "{{ compose_version.stdout }}"
-
 - name: Create paperclip directories
-  file:
+  ansible.builtin.file:
     path: "{{ item }}"
     state: directory
     mode: '0755'
@@ -29,97 +8,77 @@
     group: root
   loop:
     - "{{ paperclip_base_path }}"
-    - "{{ paperclip_data_dir }}"
-    - "{{ paperclip_certs_dir }}"
+    # The postgres image chowns its own data dir on start; do not chown it here.
     - "{{ paperclip_base_path }}/pgdata"
   become: true
+  tags: [paperclip, files]
 
-- name: Create OpenCode config directory
-  file:
-    path: "{{ paperclip_base_path }}/.config/opencode"
+# Container runs as UID 1000. Deliberately NOT recursive: recurse + mode would
+# chmod every file the app writes under these paths on each run. A fresh
+# deploy has no root-owned content to repair; if you ever need to, run a
+# one-off `chown -R 1000:1000 <dir>` by hand.
+- name: Create container-owned directories (UID 1000)
+  ansible.builtin.file:
+    path: "{{ item }}"
     state: directory
     mode: '0755'
     owner: 1000
     group: 1000
+  loop:
+    - "{{ paperclip_data_dir }}"
+    - "{{ paperclip_base_path }}/.config/opencode"
+    - "{{ paperclip_base_path }}/.local/share/opencode"
   become: true
+  tags: [paperclip, files]
 
-- name: Create OpenCode .local/share directory
-  file:
-    path: "{{ paperclip_base_path }}/.local/share/opencode"
-    state: directory
-    mode: '0755'
-    owner: 1000
-    group: 1000
-  become: yes
-
-- name: Deploy OpenCode auth.json with vLLM credentials
-  template:
-    src: opencode-auth.json.j2
-    dest: "{{ paperclip_base_path }}/.local/share/opencode/auth.json"
-    mode: '0600'
-    owner: 1000
-    group: 1000
-  become: true
-  no_log: true
-
-- name: Fix directory ownership for container user (UID 1000)
-  shell: |
-    chown -R 1000:1000 {{ paperclip_data_dir }}
-    chown -R 1000:1000 {{ paperclip_base_path }}/pgdata
-    chown -R 1000:1000 {{ paperclip_base_path }}/.config
-  become: true
+# - name: Deploy OpenCode auth.json with vLLM credentials
+#   template:
+#     src: opencode-auth.json.j2
+#     dest: "{{ paperclip_base_path }}/.local/share/opencode/auth.json"
+#     mode: '0600'
+#     owner: 1000
+#     group: 1000
+#   become: true
+#   no_log: true
 
+# This file is both the compose interpolation source and the container's
+# env_file, so everything here is visible inside the Paperclip container.
+# Infrastructure secrets (Cloudflare token, certbot email) live in
+# /opt/nginx/.env instead.
 - name: Write secrets and configuration to .env
-  copy:
+  ansible.builtin.copy:
     content: |
       # Authentication secrets (from vault)
       BETTER_AUTH_SECRET={{ vault_better_auth_secret }}
       PAPERCLIP_AGENT_JWT_SECRET={{ vault_paperclip_agent_jwt_secret }}
-      
-      # Infrastructure configuration
-      CLOUDFLARE_API_TOKEN={{ vault_cloudflare_api_token }}
-      CERTBOT_EMAIL={{ certbot_email }}
-      PAPERCLIP_DOMAIN={{ paperclip_domain }}
-      VLLM_ENDPOINT={{ vllm_endpoint }}
-      VLLM_API_KEY={{ vault_vllm_api_key }}
+
+      # Inference endpoints
+      ILLUMINATIONS_ENDPOINT={{ illuminations_endpoint }}
+      ILLUMINATIONS_API_KEY={{ vault_illuminations_api_key }}
+      FANTASMIC_ENDPOINT={{ fantasmic_endpoint }}
+      FANTASMIC_API_KEY={{ vault_fantasmic_api_key }}
     dest: "{{ paperclip_base_path }}/.env"
     mode: '0600'
     owner: root
     group: root
   become: true
   no_log: true
+  notify: Restart paperclip stack
+  tags: [paperclip, files]
 
+# No notify: the deploy task below recreates whatever the compose file changed.
 - name: Template docker-compose.yml
-  template:
+  ansible.builtin.template:
     src: docker-compose.yml.j2
     dest: "{{ paperclip_base_path }}/docker-compose.yml"
     mode: '0644'
     owner: root
     group: root
   become: true
-  notify: Restart paperclip stack
-
-- name: Template nginx config
-  template:
-    src: nginx.conf.j2
-    dest: "{{ paperclip_base_path }}/nginx.conf"
-    mode: '0644'
-    owner: root
-    group: root
-  become: true
-  notify: Reload nginx
-
-- name: Template certbot init script
-  template:
-    src: certbot-init.sh.j2
-    dest: "{{ paperclip_base_path }}/certbot-init.sh"
-    mode: '0755'
-    owner: root
-    group: root
-  become: true
+  tags: [paperclip, files]
 
 - name: Template OpenCode config
-  template:
+  ansible.builtin.template:
     src: opencode.jsonc.j2
     dest: "{{ paperclip_base_path }}/.config/opencode/opencode.jsonc"
     mode: '0644'
@@ -127,43 +86,71 @@
     group: 1000
   become: true
   notify: Restart paperclip stack
+  tags: [paperclip, files]
 
-- name: Stop and remove old containers
-  shell: |
-    cd {{ paperclip_base_path }}
-    /usr/local/bin/docker-compose down --remove-orphans 2>/dev/null || true
+# Idempotent: only changed services are recreated. remove_orphans also cleans
+# up the old paperclip-nginx / paperclip-certbot containers if they still exist.
+- name: Deploy paperclip stack
+  community.docker.docker_compose_v2:
+    project_src: "{{ paperclip_base_path }}"
+    pull: missing
+    remove_orphans: true
+    state: present
+  register: paperclip_deploy
   become: true
-  ignore_errors: true
-
-- name: Deploy docker compose stack
-  shell: |
-    cd {{ paperclip_base_path }}
-    /usr/local/bin/docker-compose up -d
-  become: true
-  register: compose_result
+  tags: [paperclip, deploy]
 
 - name: Wait for Paperclip to be ready
-  uri:
+  ansible.builtin.uri:
     url: "http://localhost:{{ paperclip_port }}/health"
     status_code: 200
-  register: result
-  until: result.status == 200
+  register: paperclip_health
+  until: paperclip_health.status == 200
   retries: 30
   delay: 5
   ignore_errors: true
+  tags: [paperclip, deploy, verify]
 
-- name: Wait for nginx to be ready
-  uri:
-    url: "https://{{ paperclip_domain }}/"
-    validate_certs: no
-    status_code: [200, 301, 302]
-  register: result
-  until: result.status in [200, 301, 302]
-  retries: 30
+- name: Warn if Paperclip never became healthy
+  ansible.builtin.debug:
+    msg: "WARNING: Paperclip did not report healthy on port {{ paperclip_port }}. Check: docker logs paperclip"
+  when: paperclip_health.status | default(0) != 200
+  tags: [paperclip, verify]
+
+# Requires day1_deploy_nginx_proxy.yml to have run first.
+- name: Publish Paperclip through nginx-proxy
+  ansible.builtin.include_role:
+    name: nginx-proxy
+    tasks_from: add_vhost
+    apply:
+      become: true
+      tags: [paperclip, nginx]
+  vars:
+    vhost_name: paperclip
+    vhost_domain: "{{ paperclip_domain }}"
+    vhost_upstream_port: "{{ paperclip_port }}"
+    vhost_health_path: /health
+  tags: [paperclip, nginx]
+
+# --resolve keeps this independent of DNS: it tests nginx, the cert and the
+# upstream. DNS is a separate (manual) prerequisite.
+- name: Verify HTTPS through nginx
+  ansible.builtin.command: >-
+    curl -sk -o /dev/null -w '%{http_code}'
+    --resolve {{ paperclip_domain }}:443:127.0.0.1
+    https://{{ paperclip_domain }}/
+  register: paperclip_https
+  until: paperclip_https.stdout in ['200', '301', '302']
+  retries: 6
   delay: 5
-  ignore_errors: true
+  changed_when: false
+  failed_when: false
+  tags: [paperclip, verify]
 
-- name: Verify Paperclip is accessible via https
-  debug:
-    msg: "Paperclip is accessible at https://{{ paperclip_domain }}"
-  when: result.status is defined and result.status in [200, 301, 302]
+- name: Report HTTPS result
+  ansible.builtin.debug:
+    msg: >-
+      {{ 'Paperclip is serving at https://' ~ paperclip_domain
+         if paperclip_https.stdout in ['200', '301', '302']
+         else 'WARNING: https://' ~ paperclip_domain ~ ' returned ' ~ (paperclip_https.stdout | default('no response')) ~ ' (502 usually means Paperclip is not up yet)' }}
+  tags: [paperclip, verify]

ansible/roles/paperclip-docker/templates/certbot-init.sh.j2

@@ -1,65 +0,0 @@
-#!/bin/sh
-set -e
-
-DOMAIN="${PAPERCLIP_DOMAIN}"
-CERT_PATH="/etc/letsencrypt/live/${DOMAIN}/fullchain.pem"
-EMAIL="${CERTBOT_EMAIL}"
-TOKEN="${CLOUDFLARE_API_TOKEN}"
-
-# Verify environment variables
-if [ -z "$DOMAIN" ]; then
-    echo "ERROR: PAPERCLIP_DOMAIN not set"
-    exit 1
-fi
-
-if [ -z "$EMAIL" ]; then
-    echo "ERROR: CERTBOT_EMAIL not set"
-    exit 1
-fi
-
-if [ -z "$TOKEN" ]; then
-    echo "ERROR: CLOUDFLARE_API_TOKEN not set"
-    exit 1
-fi
-
-# Check if certificate already exists
-if [ -f "$CERT_PATH" ]; then
-    echo "Certificate for $DOMAIN already exists at $CERT_PATH"
-    exit 0
-fi
-
-echo "Requesting certificate for $DOMAIN via Cloudflare DNS challenge..."
-
-# Create a temporary credentials file (more reliable than env var with certbot)
-CREDS_FILE="/tmp/cloudflare.ini"
-cat > "$CREDS_FILE" << EOF
-dns_cloudflare_api_token = ${TOKEN}
-EOF
-chmod 600 "$CREDS_FILE"
-
-# Request certificate with Cloudflare DNS plugin
-if certbot certonly \
-    --dns-cloudflare \
-    --dns-cloudflare-credentials "$CREDS_FILE" \
-    --agree-tos \
-    --no-eff-email \
-    --email "$EMAIL" \
-    --non-interactive \
-    -d "$DOMAIN"; then
-    echo "Certificate obtained successfully for $DOMAIN"
-else
-    echo "ERROR: Failed to obtain certificate for $DOMAIN"
-    rm -f "$CREDS_FILE"
-    exit 1
-fi
-
-# Clean up credentials file
-rm -f "$CREDS_FILE"
-
-# Verify certificate
-if [ -f "$CERT_PATH" ]; then
-    echo "Certificate verified at $CERT_PATH"
-else
-    echo "ERROR: Certificate not found at $CERT_PATH after successful certbot run"
-    exit 1
-fi

ansible/roles/paperclip-docker/templates/docker-compose.yml.j2

@@ -3,6 +3,8 @@ services:
     image: postgres:16-alpine
     container_name: paperclip-db
     network_mode: host
+    # network_mode: host would otherwise expose Postgres on every interface
+    command: ["postgres", "-c", "listen_addresses=127.0.0.1"]
     environment:
       POSTGRES_USER: paperclip
       POSTGRES_PASSWORD: paperclip
@@ -23,12 +25,12 @@ services:
     env_file:
       - {{ paperclip_base_path }}/.env
     environment:
-      HOST: "127.0.0.1"
+      HOST: {{ paperclip_listen_host }}
       PAPERCLIP_HOME: /paperclip
       PAPERCLIP_PORT: {{ paperclip_port }}
       PAPERCLIP_PUBLIC_URL: "https://{{ paperclip_domain }}"
       PAPERCLIP_DEPLOYMENT_MODE: "{{ paperclip_deployment_mode }}"
-      PAPERCLIP_DEPLOYMENT_EXPOSURE: "private"
+ #     PAPERCLIP_DEPLOYMENT_EXPOSURE: "private"
       PAPERCLIP_ALLOWED_HOSTNAMES: "{{ paperclip_allowed_hostnames }}"
       DATABASE_URL: "postgresql://paperclip:paperclip@localhost:5432/paperclip"
       BETTER_AUTH_SECRET: ${BETTER_AUTH_SECRET}
@@ -50,43 +52,3 @@ services:
       timeout: 10s
       retries: 3
       start_period: 60s
-
-  certbot:
-    image: certbot/dns-cloudflare:latest
-    container_name: paperclip-certbot
-    volumes:
-      - {{ paperclip_certs_dir }}:/etc/letsencrypt
-      - {{ paperclip_base_path }}/certbot-init.sh:/certbot-init.sh:ro
-      - /var/run/docker.sock:/var/run/docker.sock:ro
-    environment:
-      CLOUDFLARE_API_TOKEN: ${CLOUDFLARE_API_TOKEN}
-      CERTBOT_EMAIL: ${CERTBOT_EMAIL}
-      PAPERCLIP_DOMAIN: ${PAPERCLIP_DOMAIN}
-    entrypoint: /bin/sh
-    command:
-      - -c
-      - |
-        /certbot-init.sh && while true; do certbot renew --quiet; sleep 12h; done
-    restart: unless-stopped
-
-  nginx:
-    image: nginx:latest
-    container_name: paperclip-nginx
-    network_mode: host
-    volumes:
-      - {{ paperclip_base_path }}/nginx.conf:/etc/nginx/nginx.conf:ro
-      - {{ paperclip_certs_dir }}:/etc/nginx/certs:ro
-    depends_on:
-      - certbot
-    restart: unless-stopped
-    healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:80/"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-      start_period: 30s
-
-volumes:
-  paperclip_data:
-    driver: local
-    

ansible/roles/paperclip-docker/templates/nginx.conf.j2

@@ -1,86 +0,0 @@
-user nginx;
-worker_processes auto;
-error_log /var/log/nginx/error.log warn;
-pid /var/run/nginx.pid;
-
-events {
-    worker_connections 1024;
-}
-
-http {
-    include /etc/nginx/mime.types;
-    default_type application/octet-stream;
-
-    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
-                    '$status $body_bytes_sent "$http_referer" '
-                    '"$http_user_agent" "$http_x_forwarded_for"';
-
-    access_log /var/log/nginx/access.log main;
-
-    sendfile on;
-    tcp_nopush on;
-    tcp_nodelay on;
-    keepalive_timeout 65;
-    types_hash_max_size 2048;
-    client_max_body_size 100M;
-
-    # Redirect HTTP to HTTPS
-    server {
-        listen 80;
-        server_name {{ paperclip_domain }};
-        return 301 https://$server_name$request_uri;
-    }
-
-    # HTTPS server
-    server {
-        listen 443 ssl http2;
-        server_name {{ paperclip_domain }};
-
-        # SSL certificates from certbot
-        ssl_certificate /etc/nginx/certs/live/{{ paperclip_domain }}/fullchain.pem;
-        ssl_certificate_key /etc/nginx/certs/live/{{ paperclip_domain }}/privkey.pem;
-
-        # SSL configuration
-        ssl_protocols TLSv1.2 TLSv1.3;
-        ssl_ciphers HIGH:!aNULL:!MD5;
-        ssl_prefer_server_ciphers on;
-        ssl_session_cache shared:SSL:10m;
-        ssl_session_timeout 10m;
-
-        # Security headers
-        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
-        add_header X-Content-Type-Options "nosniff" always;
-        add_header X-Frame-Options "SAMEORIGIN" always;
-        add_header X-XSS-Protection "1; mode=block" always;
-
-        location / {
-            proxy_pass http://127.0.0.1:{{ paperclip_port }};
-            proxy_set_header Host $host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto $scheme;
-            proxy_set_header X-Forwarded-Host $server_name;
-            proxy_set_header X-Forwarded-Port $server_port;
-
-            # WebSocket support
-            proxy_http_version 1.1;
-            proxy_set_header Upgrade $http_upgrade;
-            proxy_set_header Connection "upgrade";
-
-            # Timeouts
-            proxy_connect_timeout 60s;
-            proxy_send_timeout 60s;
-            proxy_read_timeout 60s;
-
-            # Buffering
-            proxy_buffering off;
-            proxy_request_buffering off;
-        }
-
-        # Health check endpoint (no auth)
-        location /health {
-            proxy_pass http://127.0.0.1:{{ paperclip_port }}/health;
-            access_log off;
-        }
-    }
-}

ansible/roles/paperclip-docker/templates/opencode-auth.json.j2

@@ -2,7 +2,7 @@
   "vllm": {
     "type": "api",
     "key": "{{ vault_vllm_api_key }}"
-  }
+  },
   "fantasmic": {
     "type": "api",
     "key": "{{ vault_fantasmic_api_key }}"

ansible/roles/paperclip-docker/templates/opencode.jsonc.j2

@@ -1,9 +1,9 @@
 {
   "$schema": "https://opencode.ai/config.json",
   "provider": {
-    "vllm": {
+    "illuminations": {
       "npm": "@ai-sdk/openai-compatible",
-      "name": "vLLM (Local)",
+      "name": "illuminations",
       "models": {
         "gemma-4-26b": {
           "id": "Gemma-4-26B-A4B-it-AWQ",
@@ -11,9 +11,9 @@
         }
       },
       "options": {
-        "baseURL": "{{ vllm_endpoint }}/v1",
+        "baseURL": "{{ illuminations_endpoint }}/v1",
         "headers": {
-          "Authorization": "Bearer {{ vault_vllm_api_key }}"
+          "Authorization": "Bearer {env:ILLUMINATIONS_API_KEY}"
         }
       }
     }