Add shared nginx-proxy role; deploy Hermes in Docker; rename Paperclip

nginx-proxy (new role):
- Shared nginx + certbot under /opt/nginx, replacing the copies that
  lived inside the Paperclip compose project
- Apps publish via tasks_from: add_vhost (cert marker -> issue cert ->
  write vhost -> nginx -t -> reload); a vhost that fails nginx -t is
  rolled back
- Certs issued one per domain via Cloudflare DNS challenge; domains
  registered as marker files in cert-domains/; optional LE staging
- Stock nginx.conf plus conf.d (directory mounts, not single-file
  mounts); catch-all server drops unknown Host/SNI
- Containers renamed to nginx and certbot

hermes_docker (new role):
- Official Hermes image, host networking, gateway and dashboard bound
  to loopback and published through nginx-proxy
- Dashboard login via basic-auth provider, secrets from the vault
- Main model set from hermes_model (fantasmic endpoint, key from vault);
  config.yaml overrides are merged from a 0600 staged file that is
  removed afterwards, so the key never appears in a command line
- Data dir owner-only (HERMES_HOME_MODE=0700)

paperclip-docker:
- Remove nginx, certbot and docker-compose download tasks; deploy with
  community.docker.docker_compose_v2 (idempotent, remove_orphans)
- Stop passing the Cloudflare token and certbot email to the Paperclip
  container via env_file
- Bind Postgres to 127.0.0.1 (host networking exposed it on all
  interfaces); stop chowning pgdata, which the postgres image owns
- Handler recreates the paperclip service so .env changes apply
- Domain m-v.wdw-labs.cloud -> cranium-command.wdw-labs.cloud

Playbooks:
- Add day1_deploy_nginx_proxy.yml and day1_deploy_hermes_docker.yml
- deploy_paperclip_stack.yml: run nginx-proxy before paperclip-docker,
  CLI-based container checks (no Docker SDK dependency)
- Remove deploy-hermes.yml

Host vars: add nginx_proxy_certbot_email and nginx_proxy_certbot_staging
(staging stays on until the switch to real certs).

ansible/roles/hermes_docker_backup/defaults/main.yml

@@ -0,0 +1,66 @@
+---
+hermes_image: nousresearch/hermes-agent:latest
+# missing = only pull if absent (no surprise upgrades on redeploy).
+# Set to "always" when you actually want to upgrade the image.
+hermes_pull_policy: missing
+
+hermes_container_name: hermes
+hermes_project_dir: /opt/hermes
+hermes_data_dir: "{{ hermes_project_dir }}/data"
+hermes_secrets_env_file: "{{ hermes_project_dir }}/hermes-secrets.env"
+
+# Same network namespace as Paperclip and nginx
+hermes_network_mode: host
+hermes_memory_limit: 8G
+hermes_cpu_limit: "4.0"
+# Playwright/Chromium needs shared memory (upstream troubleshooting note)
+hermes_shm_size: 1g
+
+# --- Gateway / API server (loopback only; Paperclip reaches it locally) ---
+hermes_gateway_port: 8642
+hermes_api_server_enabled: true
+hermes_api_server_host: 127.0.0.1
+# Leave empty to run keyless on loopback. If you set one, Paperclip (or any
+# other client) must send it as a bearer token.
+# e.g. hermes_api_server_key: "{{ vault_hermes_api_server_key }}"
+hermes_api_server_key: "{{ vault_hermes_api_server_key }}"
+
+# --- Dashboard (bound to loopback, published through nginx) ---
+hermes_dashboard_enabled: true
+hermes_dashboard_host: 127.0.0.1
+hermes_dashboard_port: 9119
+hermes_domain: haunted-mansion.wdw-labs.cloud
+hermes_public_url: "https://{{ hermes_domain }}"
+hermes_dashboard_username: admin
+
+# Vault variables this role requires (see README block at bottom of file)
+hermes_required_vault_vars:
+  - vault_hermes_dashboard_password_hash
+  - vault_hermes_dashboard_secret
+
+# --- Preflight ---
+hermes_preflight_ports:
+  - "{{ hermes_gateway_port }}"
+  - "{{ hermes_dashboard_port }}"
+hermes_wait_timeout: 120
+
+# --- config.yaml overrides (deep-merged, only the keys listed here) ---
+# nginx runs with host networking, so it reaches the dashboard from loopback.
+# Model settings are NOT managed by default (your wizard-written config stays
+# untouched). To manage them, add a model block, for example:
+#   model:
+#     provider: custom
+#     model: <served-model-name>
+#     base_url: http://10.1.71.130:8000/v1
+#     api_key: "{{ vault_vllm_api_key }}"
+hermes_config_overrides:
+  dashboard:
+    trusted_proxies:
+      - "127.0.0.1"
+
+# Required vault vars:
+#   vault_hermes_dashboard_password_hash  - output of hash_password() (see below)
+#   vault_hermes_dashboard_secret         - openssl rand -base64 32
+# Generate the hash with:
+#   docker exec -it -w /opt/hermes hermes /opt/hermes/.venv/bin/python -c \
+#     "from plugins.dashboard_auth.basic import hash_password; print(hash_password('PW'))"

ansible/roles/hermes_docker_backup/files/merge_config.py

@@ -0,0 +1,39 @@
+"""Deep-merge overrides (JSON in $HERMES_CONFIG_OVERRIDES) into config.yaml.
+
+Run inside the hermes container as the hermes user. Only keys present in the
+overrides are touched. Prints CHANGED when the file was rewritten.
+"""
+import json
+import os
+import sys
+
+import yaml
+
+PATH = "/opt/data/config.yaml"
+overrides = json.loads(os.environ["HERMES_CONFIG_OVERRIDES"])
+
+if os.path.exists(PATH):
+    with open(PATH) as fh:
+        current = yaml.safe_load(fh) or {}
+else:
+    current = {}
+
+
+def merge(dst, src):
+    changed = False
+    for key, val in src.items():
+        if isinstance(val, dict) and isinstance(dst.get(key), dict):
+            changed |= merge(dst[key], val)
+        elif dst.get(key) != val:
+            dst[key] = val
+            changed = True
+    return changed
+
+
+if merge(current, overrides):
+    with open(PATH, "w") as fh:
+        yaml.safe_dump(current, fh, sort_keys=False)
+    print("CHANGED")
+else:
+    print("UNCHANGED")
+sys.exit(0)

ansible/roles/hermes_docker_backup/handlers/main.yml

@@ -0,0 +1,5 @@
+---
+# Config.yaml is read at start, so a plain restart is enough.
+# Compose/env changes are handled by docker compose itself (it recreates).
+- name: Restart hermes
+  ansible.builtin.command: "docker restart {{ hermes_container_name }}"

ansible/roles/hermes_docker_backup/nginx-hermes-vhost.conf.j2

@@ -0,0 +1,49 @@
+    # ---- Hermes dashboard: paste inside the http { } block of nginx.conf.j2 ----
+    # Redirect HTTP to HTTPS
+    server {
+        listen 80;
+        server_name {{ hermes_domain }};
+        return 301 https://$server_name$request_uri;
+    }
+
+    server {
+        listen 443 ssl http2;
+        server_name {{ hermes_domain }};
+
+        ssl_certificate /etc/nginx/certs/live/{{ hermes_domain }}/fullchain.pem;
+        ssl_certificate_key /etc/nginx/certs/live/{{ hermes_domain }}/privkey.pem;
+
+        ssl_protocols TLSv1.2 TLSv1.3;
+        ssl_ciphers HIGH:!aNULL:!MD5;
+        ssl_prefer_server_ciphers on;
+        ssl_session_cache shared:SSL:10m;
+        ssl_session_timeout 10m;
+
+        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
+        add_header X-Content-Type-Options "nosniff" always;
+        add_header X-Frame-Options "SAMEORIGIN" always;
+        add_header X-XSS-Protection "1; mode=block" always;
+
+        # No auth_basic here: the dashboard has its own login (basic auth provider).
+        location / {
+            proxy_pass http://127.0.0.1:{{ hermes_dashboard_port }};
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header X-Forwarded-Host $server_name;
+            proxy_set_header X-Forwarded-Port $server_port;
+
+            # WebSocket support
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+
+            proxy_connect_timeout 60s;
+            proxy_send_timeout 3600s;
+            proxy_read_timeout 3600s;
+
+            proxy_buffering off;
+            proxy_request_buffering off;
+        }
+    }

ansible/roles/hermes_docker_backup/tasks/main.yml

@@ -0,0 +1,108 @@
+---
+- name: Preflight | Required vault variables are defined
+  ansible.builtin.assert:
+    that: "{{ item }} is defined and ({{ item }} | length) > 0"
+    fail_msg: "{{ item }} is not defined. Add it to your ansible vault."
+    quiet: true
+  loop: "{{ hermes_required_vault_vars }}"
+  tags: [hermes, preflight]
+
+- name: Preflight | Confirm docker CLI is available
+  ansible.builtin.command: docker --version
+  changed_when: false
+  tags: [hermes, preflight]
+
+- name: Preflight | Get existing container state
+  community.docker.docker_container_info:
+    name: "{{ hermes_container_name }}"
+  register: hermes_existing
+  tags: [hermes, preflight]
+
+# Only enforce free ports when our container is not already running,
+# so redeploys stay idempotent.
+- name: Preflight | Ensure required ports are free
+  ansible.builtin.wait_for:
+    host: 127.0.0.1
+    port: "{{ item }}"
+    state: stopped
+    timeout: 3
+  loop: "{{ hermes_preflight_ports }}"
+  when: not (hermes_existing.container.State.Running | default(false))
+  tags: [hermes, preflight]
+
+- name: Create project and data directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    mode: "0755"
+  loop:
+    - "{{ hermes_project_dir }}"
+    - "{{ hermes_data_dir }}"
+  tags: [hermes]
+
+# Compose reads this as root on the host; the container never sees the file.
+- name: Template secrets env file
+  ansible.builtin.template:
+    src: hermes-secrets.env.j2
+    dest: "{{ hermes_secrets_env_file }}"
+    owner: root
+    group: root
+    mode: "0600"
+  no_log: true
+  tags: [hermes]
+
+- name: Template docker compose file
+  ansible.builtin.template:
+    src: docker-compose.yml.j2
+    dest: "{{ hermes_project_dir }}/docker-compose.yml"
+    mode: "0644"
+  tags: [hermes]
+
+- name: Deploy hermes
+  community.docker.docker_compose_v2:
+    project_src: "{{ hermes_project_dir }}"
+    pull: "{{ hermes_pull_policy }}"
+    state: present
+  tags: [hermes]
+
+- name: Verify gateway health
+  ansible.builtin.uri:
+    url: "http://127.0.0.1:{{ hermes_gateway_port }}/health"
+    status_code: 200
+  register: hermes_health
+  until: hermes_health.status == 200
+  retries: "{{ (hermes_wait_timeout | int) // 5 }}"
+  delay: 5
+  when: hermes_api_server_enabled | bool
+  tags: [hermes, verify]
+
+- name: Wait for dashboard port
+  ansible.builtin.wait_for:
+    host: "{{ hermes_dashboard_host }}"
+    port: "{{ hermes_dashboard_port }}"
+    timeout: "{{ hermes_wait_timeout }}"
+  when: hermes_dashboard_enabled | bool
+  tags: [hermes, verify]
+
+# Runs inside the container with the image's own venv (PyYAML is available
+# there) as the hermes user, so config.yaml stays owned by UID 10000.
+- name: Merge config.yaml overrides
+  ansible.builtin.command:
+    argv:
+      - docker
+      - exec
+      - -i
+      - --user
+      - hermes
+      - -e
+      - "HERMES_CONFIG_OVERRIDES={{ hermes_config_overrides | to_json }}"
+      - "{{ hermes_container_name }}"
+      - /opt/hermes/.venv/bin/python
+      - "-"
+    stdin: "{{ lookup('ansible.builtin.file', 'merge_config.py') }}"
+  register: hermes_config_merge
+  changed_when: "'CHANGED' in hermes_config_merge.stdout and 'UNCHANGED' not in hermes_config_merge.stdout"
+  no_log: true
+  when: hermes_config_overrides | length > 0
+  notify: Restart hermes
+  tags: [hermes, config]

ansible/roles/hermes_docker_backup/templates/docker-compose.yml.j2

@@ -0,0 +1,30 @@
+# Managed by Ansible (hermes_docker role)
+services:
+  hermes:
+    image: {{ hermes_image }}
+    container_name: {{ hermes_container_name }}
+    restart: unless-stopped
+    network_mode: {{ hermes_network_mode }}
+    command: gateway run
+    shm_size: "{{ hermes_shm_size }}"
+    env_file:
+      - {{ hermes_secrets_env_file }}
+    environment:
+{% if hermes_dashboard_enabled | bool %}
+      HERMES_DASHBOARD: "1"
+      HERMES_DASHBOARD_HOST: "{{ hermes_dashboard_host }}"
+      HERMES_DASHBOARD_PORT: "{{ hermes_dashboard_port }}"
+      HERMES_DASHBOARD_PUBLIC_URL: "{{ hermes_public_url }}"
+{% endif %}
+{% if hermes_api_server_enabled | bool %}
+      API_SERVER_ENABLED: "true"
+      API_SERVER_HOST: "{{ hermes_api_server_host }}"
+      API_SERVER_PORT: "{{ hermes_gateway_port }}"
+{% endif %}
+    volumes:
+      - {{ hermes_data_dir }}:/opt/data
+    deploy:
+      resources:
+        limits:
+          memory: {{ hermes_memory_limit }}
+          cpus: "{{ hermes_cpu_limit }}"

ansible/roles/hermes_docker_backup/templates/hermes-secrets.env.j2

@@ -0,0 +1,8 @@
+# Managed by Ansible (hermes_docker role) - do not edit
+# Single quotes keep compose from interpolating '$' inside hashes/secrets.
+HERMES_DASHBOARD_BASIC_AUTH_USERNAME={{ hermes_dashboard_username }}
+HERMES_DASHBOARD_BASIC_AUTH_PASSWORD_HASH='{{ vault_hermes_dashboard_password_hash }}'
+HERMES_DASHBOARD_BASIC_AUTH_SECRET='{{ vault_hermes_dashboard_secret }}'
+{% if hermes_api_server_key | length > 0 %}
+API_SERVER_KEY='{{ hermes_api_server_key }}'
+{% endif %}