multica deployment

hermes/obsidian-file-watcher/README.md

@@ -0,0 +1,223 @@
+# wiki-llm intake pipeline (n8n + Hermes Kanban)
+
+Detects new raw files in the Obsidian vault, queues one Hermes Kanban card per file, tracks every file in an n8n Data Table, and alerts on failure. The Hermes librarian does the actual cataloging and writes wiki pages directly into the vault per the llm-wiki skill.
+
+> **Status:** design complete, partly implemented. `kanban-gateway.sh` (tested against a stub `hermes`) and `wiki-llm-watcher.py` (tested against a mock webhook) are written. The n8n workflows are still to build. See [Open items](#open-items).
+
+## Architecture
+
+```mermaid
+flowchart LR
+    subgraph MAC["Mac (vault + Hermes)"]
+        V["Obsidian vault<br/>raw/"] --> W["Watcher<br/>(launchd, polls 15-30s)"]
+        GW["kanban-gateway<br/>(SSH forced command)"] --> K["hermes kanban<br/>~/.hermes/kanban.db"]
+        K --> D["Dispatcher<br/>(hermes gateway)"] --> L["Librarian profile<br/>(llm-wiki skill)"]
+        L --> V
+    end
+    subgraph N8N["n8n"]
+        IN["Intake workflow"]
+        ST["Status workflow<br/>(Schedule Trigger)"]
+        DT[("Data Table<br/>processed-files log")]
+        NT["Slack / email<br/>failure alert"]
+    end
+    W -- "POST one file per call" --> IN
+    IN <--> DT
+    IN -- "SSH: create" --> GW
+    ST -- "SSH: list / show" --> GW
+    ST <--> DT
+    ST --> NT
+```
+
+| Piece | Where | Job |
+|---|---|---|
+| Watcher | Mac, launchd | Finds new paths in `raw/`, waits until each file is stable, POSTs one webhook call per file |
+| Intake workflow | n8n | Validates, dedupes against the Data Table, creates the Kanban card over SSH |
+| `kanban-gateway.sh` | Mac, SSH forced command | The only thing n8n's key can run: `create`, `list`, `show` |
+| Hermes Kanban + dispatcher | Mac | Durable queue, claiming, retries, stale-task reclaim, circuit breaker |
+| Librarian profile | Mac | Does the work in the vault workspace, writes wiki pages per the llm-wiki skill |
+| Status workflow | n8n | Polls the board, updates the Data Table, sends failure notifications |
+| Data Table | n8n | Authoritative processed-files log |
+
+## Key decisions
+
+- **Watcher on the vault host, not a network mount.** Filesystem events and Local File Trigger don't work reliably over NFS/SMB, and failures there are silent.
+- **Polling loop is the source of truth.** FSEvents coalesces and reorders events, and clippers and sync tools are noisy, so the watcher lists `raw/` every 15 to 30 seconds and compares against its seen-state. Events may wake it early but never decide anything.
+- **Files stay in `raw/`.** "Processed" is tracked in the Data Table, not by moving files.
+- **Dedupe key is the relative path, first-seen wins.** The librarian edits frontmatter after cataloging, so the content hash changes. The hash is recorded as metadata at first-seen only. A second check rejects a first-seen hash that already exists under a different path (`status=duplicate`).
+- **Kanban is the queue.** The board already provides claiming, retries, stale reclaim, and a circuit breaker, so n8n has no Worker workflow.
+- **SSH instead of the dashboard REST API.** The dashboard plugin routes under `/api/plugins/kanban/` are unauthenticated by design. This pipeline does not use or expose them.
+- **Instruction files can live in `raw/`.** `AGENTS.md`, `CLAUDE.md`, `.hermes.md`, `README.md`, and `SCHEMA.md` are ignored by name at any depth (`WIKI_IGNORE_NAMES`), and the Intake `Validate` node should reject the same names. Hermes loads `AGENTS.md` into the librarian's context; see "Processing rules (AGENTS.md)" below.
+- **Output stays outside `raw/`.** The llm-wiki skill's folder layout must keep wiki output out of `raw/`, so Hermes can't feed itself.
+
+## Mac setup
+
+### 1. Install the gateway script
+
+1. Copy `kanban-gateway.sh` to `~/bin/kanban-gateway` and `chmod 755` it.
+2. Edit the top block: `HERMES_BIN` (absolute path), `VAULT_ROOT`, `LIBRARIAN_PROFILE`, `SKILL_NAME`. The profile and skill names in the file are placeholders.
+3. Generate a dedicated key for n8n: `ssh-keygen -t ed25519 -f n8n_kanban -C n8n-wiki-llm`.
+4. Add one line to `~/.ssh/authorized_keys` on the Mac, using the absolute script path (optionally add `from="<n8n IP>"` before `restrict`):
+
+   ```
+   restrict,command="/Users/<you>/bin/kanban-gateway" ssh-ed25519 AAAA... n8n-wiki-llm
+   ```
+
+5. Enable Remote Login for your user (System Settings, General, Sharing). On an MDM-managed Mac this may need IT approval.
+6. Smoke test from the n8n host: `ssh -i n8n_kanban <user>@<mac> list`. A bare `list` should return JSON.
+
+The SSH user must be the user that owns the Hermes profiles and `~/.hermes/kanban.db`. Kanban is single-host and per-user.
+
+### 2. Gateway script protocol
+
+| Command sent over SSH | Effect |
+|---|---|
+| `create <base64 relative path>` | Creates a card whose body lists every `AGENTS.md` from the vault root down to the file's folder (shallow to deep), so folder rules are read deterministically. Assignee = librarian profile, `--workspace dir:<vault>`, `--idempotency-key <relative path>`, `--skill <skill>`, `--json` |
+| `list [status]` | `hermes kanban list --json`, optionally filtered. Status is one of `triage todo ready running blocked review done archived` |
+| `show <task id>` | `hermes kanban show <id> --json`. The id must match `t_[A-Za-z0-9]+` |
+
+The path travels base64-encoded and is never passed through a shell or `eval`, so filenames containing spaces, quotes, or `$()` are inert. The script also rejects control characters, absolute paths, `..` segments, anything outside `raw/`, and symlinks that resolve out of `raw/`.
+
+Exit codes: `0` ok, `2` rejected input, `3` file not found.
+
+### 3. Run the dispatcher
+
+`hermes gateway start` must be running on the Mac. Without it, cards stay `ready`. The default dispatch tick is 60 seconds. Cap concurrency with `kanban.max_in_progress` in `~/.hermes/config.yaml` so a burst drains at a pace your local models can handle.
+
+### 4. Install the watcher
+
+1. Copy `wiki-llm-watcher.py` to `~/bin/` (`chmod 755`). It uses only the Python 3 standard library.
+2. Copy `wiki-llm-watcher.env.example` to `~/.config/wiki-llm-watcher/env`, edit it, and `chmod 600` it. It holds the webhook URL (the Webhook node's **production** URL) and secret.
+3. **Baseline first:** `python3 ~/bin/wiki-llm-watcher.py --baseline` marks everything already in `raw/` as seen, so the first run does not post your whole backlog. Skip this only if you want the backlog processed.
+4. Dry run: `python3 ~/bin/wiki-llm-watcher.py --dry-run --once` prints the payloads it would send and posts nothing.
+5. Install `com.wiki-llm.watcher.plist` into `~/Library/LaunchAgents/` (replace `YOURNAME`), then `launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.wiki-llm.watcher.plist`. Logs go to `~/Library/Logs/wiki-llm-watcher.log`.
+6. To reprocess one file on purpose: `python3 ~/bin/wiki-llm-watcher.py --forget raw/path/to/file.md`.
+
+macOS privacy controls (TCC) can block a LaunchAgent from reading `~/Documents`, `~/Desktop`, or iCloud Drive. If the log shows permission errors, grant the Python binary access under Privacy & Security, or keep the vault outside those folders.
+
+Webhook response handling in the watcher: any 2xx marks the file posted (including n8n's `duplicate` response). `400/409/413/422` marks it `rejected` locally and stops retrying. `401/403/404` and an unreachable n8n pause posting for 60 seconds without marking anything. Other errors retry per file with exponential backoff.
+
+## n8n workflows
+
+### Intake (Webhook trigger)
+
+1. **Webhook node**, header-auth credential. Body: `{path, sha256, size, mtime, source}`.
+2. **Code node:** validate `path` against an allowlist regex (for example `^raw/[\w .,\-/]+\.md$`, widened to the extensions you accept). Reject mismatches as `status=rejected`. Compute `rel_b64 = Buffer.from(path, 'utf8').toString('base64')` and `file_key = path`.
+3. **Data Table lookup** by `file_key`. If a row exists, respond `duplicate` and stop. Otherwise insert with `status=queued`. Respond to the webhook after this insert, so a 2xx means "durably queued".
+4. **SSH node** (Execute Command, Private Key credential): `create {{ $json.rel_b64 }}`. Retry On Fail: 3 tries with a few seconds' backoff. On Error: Continue (using error output).
+5. **If node:** check `code` is `0` explicitly. Don't rely on the node throwing. Exit `3` means the file is gone, so set `rejected`. Anything else, set `create_failed`, record `last_error`, and notify.
+6. **Code node:** parse `stdout` as JSON and extract the task id. Verify the key name against a real `hermes kanban create --json` run.
+7. **Data Table update:** `kanban_task_id`, `status=created`, `created_at`.
+
+An SSH timeout can happen after the card was created. The kanban idempotency key means a retry returns the existing card, not a duplicate.
+
+### Status (Schedule Trigger, every 1 to 2 minutes)
+
+1. **SSH node:** `list blocked` and `list done` (or plain `list`).
+2. **Code node:** match cards to Data Table rows by `kanban_task_id`.
+3. **Data Table update:** `running`, `done` (set `processed_at`), `blocked`.
+4. **If node:** for `blocked` where `notified_at` is empty, run `show <id>` to get the recorded reason, send the **Slack** or **Gmail** node, then set `notified_at`. Notify on transitions only.
+
+### Retry sweep (Schedule Trigger, optional)
+
+Re-attempts rows stuck in `create_failed` with a capped `attempts` count.
+
+### Error Workflow
+
+Set an **Error Trigger** workflow on both workflows above as a safety net for crashes outside the handled branches.
+
+## Data Table: `wiki_llm_files`
+
+| Column | Notes |
+|---|---|
+| `file_key` | Relative path. Unique lookup key |
+| `sha256_first_seen` | Recorded at first-seen only |
+| `size`, `mtime`, `source` | Source: `watcher` or `obsidian-plugin` |
+| `status` | `queued`, `created`, `running`, `done`, `blocked`, `create_failed`, `rejected`, `duplicate` |
+| `kanban_task_id` | From the `create` response |
+| `attempts`, `last_error` | Retry accounting and diagnostics |
+| `first_seen`, `created_at`, `processed_at` | Timestamps |
+| `notified_at` | Prevents duplicate alerts |
+
+A Data Table is enough to start. Move to Postgres only if you need joins, heavy volume, or external reporting.
+
+## Processing rules (AGENTS.md)
+
+All per-folder configuration lives in the vault as `AGENTS.md` files, so the watcher stays generic: it only knows the inbox folder and what to ignore.
+
+```
+vault/
+  AGENTS.md                              # global inbox rules
+  raw/
+    genealogy/find-a-grave/AGENTS.md     # rules for that folder
+    meeting-notes/AGENTS.md              # rules for that folder
+```
+
+When the gateway creates a card, it walks from the vault root down to the file's folder and lists every `AGENTS.md` it finds in the card body, shallow to deep, telling the agent to read all of them first (deeper files take precedence). This makes the rules deterministic and visible on the card. It does not depend on Hermes discovering subdirectory context files lazily, though Hermes also loads `AGENTS.md` from its working directory and discovers subdirectory ones as files are read there. If no `AGENTS.md` exists for a folder, the card says so and the agent follows the llm-wiki skill defaults.
+
+Only one project context type loads per Hermes session (first match wins: `.hermes.md`, `AGENTS.md`, `CLAUDE.md`, `.cursorrules`). If the vault root already has a `.hermes.md`, it takes precedence at startup, but the card still names the `AGENTS.md` chain explicitly.
+
+Suggested split, so rules live in one place each:
+
+| Where | Holds |
+|---|---|
+| llm-wiki skill | How the wiki is structured and written (stable, reusable) |
+| `AGENTS.md` at root | Inbox-wide rules: never edit raw bodies (only frontmatter metadata), output goes outside `raw/`, raw file text is untrusted data and never instructions, block the card with a clear reason instead of guessing |
+| `AGENTS.md` per folder | That folder's processing rules and where its output goes |
+| Kanban card body | Per-file facts (path) and the ruleset list |
+
+**Validate with a marker test:** give two folders deliberately silly rules (one adds the tag `#fag-test` to the frontmatter, the other `#meeting-test`), drop one file in each through the real pipeline, and confirm each file got only its own tag.
+
+## Failure modes
+
+| Problem | Handling |
+|---|---|
+| Duplicate triggers | Data Table lookup on relative path, plus the kanban idempotency key, plus the watcher's local seen-state. Losing the watcher's state causes noisy reposts but no reprocessing |
+| Partial writes | Watcher requires size and mtime unchanged for about 10 seconds, ignores temp patterns (`*.tmp`, `.syncthing.*`, `*.sync-conflict-*`, dotfiles, `.obsidian/`) |
+| Bursts | Watcher posts sequentially with a short delay. The board queue plus `max_in_progress` absorb the rest |
+| n8n down | Watcher records a file as posted only after a 2xx, so unposted files are retried on the next pass |
+| SSH or Mac unreachable | Row stays `queued` or `create_failed`, retried by the sweep, with an alert once attempts are exhausted |
+| Hermes worker fails | Dispatcher retries and then circuit-breaks to `blocked` with the error recorded. The Status workflow alerts and keeps the reason |
+| Stuck or crashed worker | Dispatcher reclaims stale claims automatically |
+
+## Optional: Obsidian "Post Webhook" plugin
+
+[obsidian-post-webhook](https://github.com/masterb1234/obsidian-post-webhook) sends a note to a webhook when you run a command. It is not a file watcher and is not the primary trigger. It needs Obsidian running with the note open, and it sends note content with base64 attachments rather than a path and hash.
+
+Use it only as a manual "process this note now" entry point into the same Intake workflow:
+
+- Accept both payload shapes (`{path, sha256}` and `{content, filename}`) and branch with an **If node**.
+- For plugin payloads, compute the hash with the **Crypto node** and set `source=obsidian-plugin`.
+- Enable "Exclude Attachments" and set response handling to **None**, so n8n output never writes back into the vault.
+
+## Security notes
+
+- The n8n SSH key can run only the gateway script, via `restrict,command=...`.
+- Do not run `hermes dashboard --host 0.0.0.0`. The kanban REST routes under `/api/plugins/` are unauthenticated.
+- Keep the watcher webhook behind header auth, and restrict it by network if possible.
+
+## Verify on first real run
+
+1. `hermes kanban create --json` output shape (the task id key name).
+2. Idempotency: run the same `create` twice and confirm the second returns the existing card.
+3. `--workspace dir:<vault>` resolves correctly in `hermes kanban show <id>`.
+4. `--skill` matches a skill installed on the librarian profile (`hermes skills list`).
+5. `hermes gateway start` is running and the dispatcher picks the card up.
+6. A deliberately failing card reaches `blocked` and triggers exactly one alert.
+
+## Open items
+
+- [x] Mac watcher script and launchd plist (poll loop, stability check, seen-state, one POST per file)
+- [ ] Build Intake, Status, and Retry sweep workflows, and create the Data Table
+- [ ] Choose the notification channel (Slack or email)
+- [ ] Confirm expected file volume (sets watcher delay and `max_in_progress`)
+- [ ] Decide whether to add a Hermes plugin hook (`kanban_task_blocked`) that POSTs to an n8n webhook for lower-latency alerts. Polling stays as the backstop either way
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `kanban-gateway.sh` | SSH forced-command wrapper (tested with a stub `hermes`) |
+| `wiki-llm-watcher.py` | Mac watcher (tested with a mock webhook) |
+| `wiki-llm-watcher.env.example` | Watcher configuration template |
+| `com.wiki-llm.watcher.plist` | launchd LaunchAgent for the watcher |
+| `README.md` | This document |

hermes/obsidian-file-watcher/com.wiki-llm.watcher.plist

@@ -0,0 +1,34 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+    <key>Label</key>
+    <string>com.wiki-llm.watcher</string>
+
+    <key>ProgramArguments</key>
+    <array>
+        <string>/usr/bin/python3</string>
+        <string>/Users/rblundon/bin/wiki-llm-watcher.py</string>
+    </array>
+
+    <key>EnvironmentVariables</key>
+    <dict>
+        <key>WIKI_ENV_FILE</key>
+        <string>/Users/rblundon/.config/wiki-llm-watcher/env</string>
+        <key>PYTHONUNBUFFERED</key>
+        <string>1</string>
+    </dict>
+
+    <key>RunAtLoad</key>
+    <true/>
+    <key>KeepAlive</key>
+    <true/>
+    <key>ThrottleInterval</key>
+    <integer>30</integer>
+
+    <key>StandardOutPath</key>
+    <string>/Users/rblundon/Library/Logs/wiki-llm-watcher.log</string>
+    <key>StandardErrorPath</key>
+    <string>/Users/rblundon/Library/Logs/wiki-llm-watcher.log</string>
+</dict>
+</plist>

hermes/obsidian-file-watcher/kanban-gateway.sh

@@ -0,0 +1,128 @@
+#!/bin/bash
+# kanban-gateway: SSH forced-command wrapper for n8n -> Hermes kanban.
+#
+# Install on the Mac as the user that owns ~/.hermes, e.g. ~/bin/kanban-gateway (chmod 755).
+# Wire it up in ~/.ssh/authorized_keys with a forced command (see the line in the chat).
+#
+# Verbs n8n may call (sent as the SSH command string):
+#   create <base64 of path relative to the vault, e.g. raw/clippings/foo.md>
+#   list [status]          status in: triage todo ready running blocked review done archived
+#   show <task id>         id like t_ab12cd34
+#
+# Nothing from the SSH command is ever passed through a shell or eval; the path
+# travels base64-encoded so spaces/quotes/$()/newlines in filenames are inert.
+
+set -euo pipefail
+
+# ---- EDIT THESE ------------------------------------------------------------
+HERMES_BIN="${HERMES_BIN:-$HOME/.local/bin/hermes}"   # absolute path; non-interactive SSH has a minimal PATH
+VAULT_ROOT="${VAULT_ROOT:-$HOME/Documents/wiki-llm}"  # absolute path to the vault root
+RAW_DIR="raw"                                         # inbox folder inside the vault
+LIBRARIAN_PROFILE="${LIBRARIAN_PROFILE:-librarian}"   # assignee profile name
+SKILL_NAME="${SKILL_NAME:-llm-wiki}"                  # skill installed on that profile ('' to skip pinning)
+# ----------------------------------------------------------------------------
+
+export PATH="/usr/bin:/bin:/usr/sbin:/sbin:/opt/homebrew/bin:/usr/local/bin:$HOME/.local/bin"
+
+fail() { echo "kanban-gateway: $1" >&2; exit "${2:-2}"; }
+
+cmd_line="${SSH_ORIGINAL_COMMAND:-}"
+[[ "$cmd_line" != *$'\n'* ]] || fail "newline in command"
+
+# n8n's SSH node prefixes the command with "cd <dir> ; " (or "&&") when its Working Directory
+# field is set. The directory is never used or executed here; only that exact prefix shape is
+# stripped, and whatever remains must still be one of the allowed verbs below.
+cd_re='^cd[[:space:]]+("[^"]*"|'"'"'[^'"'"']*'"'"'|[^[:space:];&|`$()<>\\]+)[[:space:]]*(;|&&)[[:space:]]*'
+if [[ "$cmd_line" =~ $cd_re ]]; then
+  cmd_line="${cmd_line#"${BASH_REMATCH[0]}"}"
+fi
+[[ -n "$cmd_line" ]] || fail "no command supplied"
+
+# Split only on spaces; the verb and a single argument.
+verb="${cmd_line%% *}"
+arg=""
+if [[ "$cmd_line" == *" "* ]]; then arg="${cmd_line#* }"; fi
+
+decode_b64() {
+  # macOS older = -D, newer/GNU = -d
+  printf '%s' "$1" | { base64 -d 2>/dev/null || base64 -D; }
+}
+
+case "$verb" in
+  create)
+    [[ "$arg" =~ ^[A-Za-z0-9+/=]+$ ]] || fail "create expects base64 argument"
+    rel="$(decode_b64 "$arg")" || fail "bad base64"
+    [[ -n "$rel" ]] || fail "empty path"
+    # Reject control characters (incl. newlines), absolute paths, and traversal.
+    if printf '%s' "$rel" | LC_ALL=C grep -q '[[:cntrl:]]'; then fail "control characters in path"; fi
+    [[ "$rel" != /* ]] || fail "absolute path rejected"
+    case "/$rel/" in */../*) fail "path traversal rejected" ;; esac
+    [[ "$rel" == "$RAW_DIR"/* ]] || fail "path must be under $RAW_DIR/"
+
+    vault_real="$(cd "$VAULT_ROOT" && pwd -P)" || fail "VAULT_ROOT not found"
+    abs="$vault_real/$rel"
+    [[ -f "$abs" ]] || fail "file not found: $rel" 3
+    # Resolve symlinks in the directory part and confirm it is still inside raw/.
+    dir_real="$(cd "$(dirname "$abs")" && pwd -P)" || fail "cannot resolve directory"
+    case "$dir_real/" in
+      "$vault_real/$RAW_DIR/"*) ;;
+      *) fail "resolved path escapes $RAW_DIR/" ;;
+    esac
+
+    # Folder-scoped rulesets live IN the vault as AGENTS.md files. Collect every AGENTS.md from the
+    # vault root down to the file's folder (shallow -> deep) and name them on the card, so the
+    # agent reads them deterministically instead of relying on lazy discovery.
+    rules=()
+    if [[ -f "$vault_real/AGENTS.md" && ! -L "$vault_real/AGENTS.md" ]]; then rules+=("AGENTS.md"); fi
+    cur=""
+    IFS='/' read -r -a parts <<< "$(dirname "$rel")"
+    for part in "${parts[@]}"; do
+      cur="${cur:+$cur/}$part"
+      if [[ -f "$vault_real/$cur/AGENTS.md" && ! -L "$vault_real/$cur/AGENTS.md" ]]; then
+        rules+=("$cur/AGENTS.md")
+      fi
+    done
+    if (( ${#rules[@]} > 0 )); then
+      rules_text="Rulesets for this file's folder. Read ALL of them before doing anything else, in this order (later ones are more specific and take precedence):"
+      for r in "${rules[@]}"; do rules_text+=$'\n'"- $r"; done
+    else
+      rules_text="No AGENTS.md rulesets found for this folder; follow the llm-wiki skill defaults."
+    fi
+
+    title="Catalog: $(basename "$rel")"
+    body="Catalog this new raw file per the llm-wiki skill.
+Relative path: $rel
+Absolute path: $abs
+
+$rules_text"
+
+    args=( kanban create "$title"
+           --body "$body"
+           --assignee "$LIBRARIAN_PROFILE"
+           --workspace "dir:$vault_real"
+           --idempotency-key "$rel"
+           --json )
+    if [[ -n "$SKILL_NAME" ]]; then args+=( --skill "$SKILL_NAME" ); fi
+    exec "$HERMES_BIN" "${args[@]}"
+    ;;
+
+  list)
+    if [[ -z "$arg" ]]; then
+      exec "$HERMES_BIN" kanban list --json
+    fi
+    case "$arg" in
+      triage|todo|ready|running|blocked|review|done|archived)
+        exec "$HERMES_BIN" kanban list --status "$arg" --json ;;
+      *) fail "invalid status" ;;
+    esac
+    ;;
+
+  show)
+    [[ "$arg" =~ ^t_[A-Za-z0-9]+$ ]] || fail "invalid task id"
+    exec "$HERMES_BIN" kanban show "$arg" --json
+    ;;
+
+  *)
+    fail "verb not allowed: $verb"
+    ;;
+esac

hermes/obsidian-file-watcher/wiki-llm-watcher.env.example

@@ -0,0 +1,23 @@
+# Copy to ~/.config/wiki-llm-watcher/env and run: chmod 600 ~/.config/wiki-llm-watcher/env
+# Lines are KEY=VALUE. No export, no shell expansion (use absolute paths).
+
+# Absolute path to the vault root, and the inbox folder inside it
+WIKI_VAULT_ROOT=/Users/YOURNAME/Documents/wiki-llm
+WIKI_RAW_DIR=raw
+
+# n8n Webhook node PRODUCTION URL (the workflow must be Active; the "test" URL only works
+# while you are listening in the editor)
+WIKI_WEBHOOK_URL=https://n8n.example.com/webhook/wiki-llm-intake
+
+# Header-auth credential configured on the Webhook node
+WIKI_AUTH_HEADER=X-Webhook-Secret
+WIKI_AUTH_SECRET=change-me
+
+# Optional tuning (defaults shown)
+# WIKI_POLL_SECONDS=20        # how often to scan raw/
+# WIKI_STABLE_SECONDS=10      # file must be unchanged this long before it is posted
+# WIKI_POST_DELAY=0.5         # pause between POSTs during a burst
+# WIKI_TIMEOUT=30             # HTTP timeout per POST
+# WIKI_IGNORE_NAMES=AGENTS.md,CLAUDE.md,.hermes.md,README.md,SCHEMA.md  # file names never posted, at any depth (case-insensitive)
+# WIKI_EXTENSIONS=.md         # comma-separated, or * for all (keep in sync with n8n's path allowlist)
+# WIKI_STATE_DB=~/.local/state/wiki-llm-watcher/seen.db

hermes/obsidian-file-watcher/wiki-llm-watcher.py

@@ -0,0 +1,363 @@
+#!/usr/bin/env python3
+"""wiki-llm-watcher: poll the vault's raw/ inbox and POST one webhook call per new file.
+
+Design (see README.md):
+  * Polling is the source of truth. No filesystem events are trusted.
+  * A file is posted only after (size, mtime) has been unchanged for WIKI_STABLE_SECONDS
+    and across at least one full scan, so half-written clips/syncs are skipped.
+  * Files are posted sequentially, one webhook call each, with a short delay between calls.
+  * A file is recorded as posted in the local SQLite seen-state ONLY after n8n returns 2xx.
+    On restart, anything not recorded is retried, so nothing is lost while n8n or the
+    watcher is down. (n8n dedupes against its Data Table, so reposts are harmless.)
+  * Detection is by NEW PATH only. Later edits (the librarian rewrites frontmatter) are ignored.
+  * Standard library only. Python 3.8+.
+
+Usage:
+  wiki-llm-watcher.py --baseline          mark everything currently in raw/ as seen (no POSTs)
+  wiki-llm-watcher.py --dry-run --once    show what WOULD be posted, post nothing
+  wiki-llm-watcher.py                     run forever (this is what launchd runs)
+  wiki-llm-watcher.py --once              one stabilised pass, then exit
+  wiki-llm-watcher.py --forget raw/x.md   drop a path from seen-state so it is posted again
+
+Configuration is by environment variables, optionally loaded from the KEY=VALUE file named by
+WIKI_ENV_FILE (keep that file chmod 600; it holds the webhook secret).
+"""
+
+import argparse
+import hashlib
+import json
+import logging
+import os
+import signal
+import socket
+import sqlite3
+import sys
+import time
+import unicodedata
+import urllib.error
+import urllib.request
+from datetime import datetime, timezone
+
+log = logging.getLogger("wiki-llm-watcher")
+STOP = False
+
+
+# ----------------------------------------------------------------------------- config
+def load_env_file(path):
+    with open(path, "r", encoding="utf-8") as fh:
+        for line in fh:
+            line = line.strip()
+            if not line or line.startswith("#") or "=" not in line:
+                continue
+            key, _, value = line.partition("=")
+            key, value = key.strip(), value.strip()
+            if len(value) >= 2 and value[0] == value[-1] and value[0] in "'\"":
+                value = value[1:-1]
+            os.environ.setdefault(key, value)
+
+
+def get_config():
+    env_file = os.environ.get("WIKI_ENV_FILE")
+    if env_file:
+        load_env_file(os.path.expanduser(env_file))
+    cfg = {
+        "vault_root": os.path.expanduser(os.environ.get("WIKI_VAULT_ROOT", "")),
+        "raw_dir": os.environ.get("WIKI_RAW_DIR", "raw"),
+        "url": os.environ.get("WIKI_WEBHOOK_URL", ""),
+        "auth_header": os.environ.get("WIKI_AUTH_HEADER", "X-Webhook-Secret"),
+        "auth_secret": os.environ.get("WIKI_AUTH_SECRET", ""),
+        "poll": float(os.environ.get("WIKI_POLL_SECONDS", "20")),
+        "stable": float(os.environ.get("WIKI_STABLE_SECONDS", "10")),
+        "post_delay": float(os.environ.get("WIKI_POST_DELAY", "0.5")),
+        "timeout": float(os.environ.get("WIKI_TIMEOUT", "30")),
+        "ignore_names": {n.strip().lower() for n in os.environ.get(
+            "WIKI_IGNORE_NAMES", "AGENTS.md,CLAUDE.md,.hermes.md,README.md,SCHEMA.md").split(",") if n.strip()},
+        "extensions": [e.strip().lower() for e in os.environ.get("WIKI_EXTENSIONS", ".md").split(",") if e.strip()],
+        "state_db": os.path.expanduser(
+            os.environ.get("WIKI_STATE_DB", "~/.local/state/wiki-llm-watcher/seen.db")
+        ),
+    }
+    return cfg
+
+
+def validate_config(cfg, need_url):
+    problems = []
+    if not cfg["vault_root"] or not os.path.isdir(cfg["vault_root"]):
+        problems.append("WIKI_VAULT_ROOT is not set or not a directory")
+    elif not os.path.isdir(os.path.join(cfg["vault_root"], cfg["raw_dir"])):
+        problems.append("raw dir not found: %s" % os.path.join(cfg["vault_root"], cfg["raw_dir"]))
+    if need_url:
+        if not cfg["url"].startswith(("http://", "https://")):
+            problems.append("WIKI_WEBHOOK_URL must be an http(s) URL")
+        if not cfg["auth_secret"]:
+            problems.append("WIKI_AUTH_SECRET is empty")
+    return problems
+
+
+# ----------------------------------------------------------------------------- state
+def open_state(path):
+    os.makedirs(os.path.dirname(path), exist_ok=True)
+    db = sqlite3.connect(path)
+    db.execute(
+        "CREATE TABLE IF NOT EXISTS files ("
+        " path TEXT PRIMARY KEY, sha256 TEXT, size INTEGER, mtime TEXT,"
+        " status TEXT NOT NULL, updated_at TEXT NOT NULL)"
+    )
+    db.commit()
+    return db
+
+
+def now_iso():
+    return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+
+
+def mark(db, rel, status, sha=None, size=None, mtime=None):
+    db.execute(
+        "INSERT INTO files(path, sha256, size, mtime, status, updated_at) VALUES(?,?,?,?,?,?)"
+        " ON CONFLICT(path) DO UPDATE SET sha256=excluded.sha256, size=excluded.size,"
+        " mtime=excluded.mtime, status=excluded.status, updated_at=excluded.updated_at",
+        (rel, sha, size, mtime, status, now_iso()),
+    )
+    db.commit()
+
+
+def known_paths(db):
+    return {row[0] for row in db.execute("SELECT path FROM files")}
+
+
+# ----------------------------------------------------------------------------- scanning
+IGNORED_SUFFIXES = (".tmp", ".part", ".partial", ".crdownload", ".download", ".swp", ".swx", "~")
+
+
+def is_ignored(name, extensions, ignore_names=frozenset()):
+    low = name.lower()
+    if low in ignore_names:  # instruction files that live in raw/ (AGENTS.md etc.) are never ingested
+        return True
+    if name.startswith("."):  # dotfiles, .syncthing.*, .DS_Store, iCloud placeholders (.foo.icloud)
+        return True
+    if low.endswith(IGNORED_SUFFIXES) or low.endswith(".icloud"):
+        return True
+    if ".sync-conflict-" in low:
+        return True
+    if "*" in extensions:
+        return False
+    return not any(low.endswith(ext) for ext in extensions)
+
+
+def scan_files(cfg):
+    """Yield (relative_path_nfc, absolute_path) for every candidate file under raw/."""
+    root = os.path.realpath(cfg["vault_root"])
+    raw = os.path.join(root, cfg["raw_dir"])
+    for dirpath, dirnames, filenames in os.walk(raw, followlinks=False):
+        dirnames[:] = sorted(d for d in dirnames if not d.startswith("."))
+        for name in sorted(filenames):
+            if is_ignored(name, cfg["extensions"], cfg["ignore_names"]):
+                continue
+            full = os.path.join(dirpath, name)
+            if os.path.islink(full):
+                continue
+            rel = os.path.relpath(full, root).replace(os.sep, "/")
+            yield unicodedata.normalize("NFC", rel), full
+
+
+def sha256_of(path):
+    h = hashlib.sha256()
+    with open(path, "rb") as fh:
+        for chunk in iter(lambda: fh.read(1024 * 1024), b""):
+            h.update(chunk)
+    return h.hexdigest()
+
+
+# ----------------------------------------------------------------------------- HTTP
+def post_json(cfg, payload):
+    """Return (status_code, body_snippet). Raises URLError/OSError when unreachable."""
+    req = urllib.request.Request(
+        cfg["url"],
+        data=json.dumps(payload).encode("utf-8"),
+        method="POST",
+        headers={"Content-Type": "application/json", cfg["auth_header"]: cfg["auth_secret"]},
+    )
+    try:
+        with urllib.request.urlopen(req, timeout=cfg["timeout"]) as resp:
+            return resp.status, resp.read(500).decode("utf-8", "replace")
+    except urllib.error.HTTPError as err:
+        return err.code, err.read(500).decode("utf-8", "replace")
+
+
+# ----------------------------------------------------------------------------- core pass
+class Watcher:
+    def __init__(self, cfg, db, dry_run=False):
+        self.cfg = cfg
+        self.db = db
+        self.dry_run = dry_run
+        self.observed = {}  # rel -> [(size, mtime_ns), first_seen_monotonic, seen_in_scans]
+        self.retry = {}  # rel -> (fail_count, not_before_monotonic)
+        self.cooldown_until = 0.0  # set when n8n is unreachable or config is wrong
+
+    def observe(self):
+        """One scan: update stability tracking, return list of (rel, full, stat) eligible now."""
+        now = time.monotonic()
+        seen = known_paths(self.db)
+        present = set()
+        eligible = []
+        for rel, full in scan_files(self.cfg):
+            if rel in seen:
+                continue
+            try:
+                st = os.stat(full)
+            except OSError:
+                continue  # vanished between listing and stat
+            present.add(rel)
+            sig = (st.st_size, st.st_mtime_ns)
+            rec = self.observed.get(rel)
+            if rec is None or rec[0] != sig:
+                self.observed[rel] = [sig, now, 1]
+                continue
+            rec[2] += 1
+            if rec[2] >= 2 and (now - rec[1]) >= self.cfg["stable"]:
+                nb = self.retry.get(rel)
+                if nb and now < nb[1]:
+                    continue
+                eligible.append((rel, full, st))
+        for gone in set(self.observed) - present:
+            del self.observed[gone]
+        eligible.sort(key=lambda item: item[2].st_mtime_ns)
+        return eligible
+
+    def post_pass(self):
+        eligible = self.observe()
+        if eligible and time.monotonic() < self.cooldown_until:
+            return  # n8n was unreachable / misconfigured a moment ago; don't spam the log
+        if eligible:
+            log.info("%d stable new file(s) to post", len(eligible))
+        for rel, full, st in eligible:
+            if STOP:
+                return
+            try:
+                digest = sha256_of(full)
+                st2 = os.stat(full)
+            except OSError as exc:
+                log.warning("skip %s (unreadable: %s)", rel, exc)
+                continue
+            if (st2.st_size, st2.st_mtime_ns) != (st.st_size, st.st_mtime_ns):
+                log.info("changed while hashing, will re-check: %s", rel)
+                self.observed.pop(rel, None)
+                continue
+            mtime_iso = datetime.fromtimestamp(st.st_mtime, timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+            payload = {
+                "path": rel,
+                "sha256": digest,
+                "size": st.st_size,
+                "mtime": mtime_iso,
+                "source": "watcher",
+                "host": socket.gethostname(),
+            }
+            if self.dry_run:
+                log.info("[dry-run] would POST %s", json.dumps(payload, ensure_ascii=False))
+                continue
+            try:
+                code, body = post_json(self.cfg, payload)
+            except (urllib.error.URLError, OSError) as exc:
+                log.error("n8n unreachable (%s); pausing posts for 60s", exc)
+                self.cooldown_until = time.monotonic() + 60
+                return  # stop this pass; do not hammer a down endpoint
+            if 200 <= code < 300:
+                mark(self.db, rel, "posted", digest, st.st_size, mtime_iso)
+                self.retry.pop(rel, None)
+                self.observed.pop(rel, None)
+                log.info("posted %s (HTTP %d)", rel, code)
+            elif code in (400, 409, 413, 422):
+                mark(self.db, rel, "rejected", digest, st.st_size, mtime_iso)
+                self.observed.pop(rel, None)
+                log.warning("rejected by n8n, not retrying: %s (HTTP %d: %s). Use --forget to retry.",
+                            rel, code, body[:200].replace("\n", " "))
+            elif code in (401, 403, 404):
+                log.error("config problem (HTTP %d) - check URL (production, workflow active?) and secret. "
+                          "Pausing posts for 60s.", code)
+                self.cooldown_until = time.monotonic() + 60
+                return
+            else:
+                fails = self.retry.get(rel, (0, 0))[0] + 1
+                delay = min(30 * (2 ** (fails - 1)), 900)
+                self.retry[rel] = (fails, time.monotonic() + delay)
+                log.warning("HTTP %d for %s, retry in %ds (attempt %d)", code, rel, delay, fails)
+            time.sleep(self.cfg["post_delay"])
+
+
+def do_baseline(cfg, db):
+    count = 0
+    for rel, full in scan_files(cfg):
+        try:
+            st = os.stat(full)
+        except OSError:
+            continue
+        mark(db, rel, "baseline", None, st.st_size,
+             datetime.fromtimestamp(st.st_mtime, timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"))
+        count += 1
+    log.info("baseline: marked %d existing file(s) as seen; nothing was posted", count)
+
+
+def sleep_interruptible(seconds):
+    end = time.monotonic() + seconds
+    while not STOP and time.monotonic() < end:
+        time.sleep(min(1.0, end - time.monotonic()))
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument("--baseline", action="store_true", help="mark existing raw/ files as seen without posting")
+    parser.add_argument("--once", action="store_true", help="run one stabilised pass and exit")
+    parser.add_argument("--dry-run", action="store_true", help="log what would be posted; post nothing, record nothing")
+    parser.add_argument("--forget", metavar="REL_PATH", help="remove a path from seen-state so it is posted again")
+    args = parser.parse_args()
+
+    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s", stream=sys.stdout)
+    cfg = get_config()
+    problems = validate_config(cfg, need_url=not (args.baseline or args.forget or args.dry_run))
+    if problems:
+        for p in problems:
+            log.error(p)
+        return 1
+    db = open_state(cfg["state_db"])
+
+    if args.forget:
+        rel = unicodedata.normalize("NFC", args.forget)
+        cur = db.execute("DELETE FROM files WHERE path = ?", (rel,))
+        db.commit()
+        log.info("forgot %d row(s) for %s", cur.rowcount, rel)
+        return 0
+    if args.baseline:
+        do_baseline(cfg, db)
+        return 0
+
+    def handle_signal(signum, _frame):
+        global STOP
+        STOP = True
+        log.info("signal %d received, stopping", signum)
+
+    signal.signal(signal.SIGTERM, handle_signal)
+    signal.signal(signal.SIGINT, handle_signal)
+
+    watcher = Watcher(cfg, db, dry_run=args.dry_run)
+    log.info("watching %s (poll %.0fs, stable %.0fs, ext %s)",
+             os.path.join(cfg["vault_root"], cfg["raw_dir"]), cfg["poll"], cfg["stable"],
+             ",".join(cfg["extensions"]))
+
+    if args.once:
+        watcher.observe()  # prime stability tracking
+        sleep_interruptible(max(cfg["stable"], 1.0))
+        watcher.observe()  # second observation (counts as the 'unchanged across scans' check)
+        watcher.post_pass()
+        return 0
+
+    while not STOP:
+        try:
+            watcher.post_pass()
+        except Exception:  # keep the daemon alive; launchd would restart us anyway
+            log.exception("unexpected error in pass")
+        sleep_interruptible(cfg["poll"])
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())