multica deployment
This commit is contained in:
rblundon committed
1 parent
63c5f7eed7
commit
39e2b87941
26 files changed
+2746
-394
No files matched your search
@@ -0,0 +1,223 @@
|
||||
# wiki-llm intake pipeline (n8n + Hermes Kanban)
|
||||
|
||||
Detects new raw files in the Obsidian vault, queues one Hermes Kanban card per file, tracks every file in an n8n Data Table, and alerts on failure. The Hermes librarian does the actual cataloging and writes wiki pages directly into the vault per the llm-wiki skill.
|
||||
|
||||
> **Status:** design complete, partly implemented. `kanban-gateway.sh` (tested against a stub `hermes`) and `wiki-llm-watcher.py` (tested against a mock webhook) are written. The n8n workflows are still to build. See [Open items](#open-items).
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph MAC["Mac (vault + Hermes)"]
|
||||
V["Obsidian vault<br/>raw/"] --> W["Watcher<br/>(launchd, polls 15-30s)"]
|
||||
GW["kanban-gateway<br/>(SSH forced command)"] --> K["hermes kanban<br/>~/.hermes/kanban.db"]
|
||||
K --> D["Dispatcher<br/>(hermes gateway)"] --> L["Librarian profile<br/>(llm-wiki skill)"]
|
||||
L --> V
|
||||
end
|
||||
subgraph N8N["n8n"]
|
||||
IN["Intake workflow"]
|
||||
ST["Status workflow<br/>(Schedule Trigger)"]
|
||||
DT[("Data Table<br/>processed-files log")]
|
||||
NT["Slack / email<br/>failure alert"]
|
||||
end
|
||||
W -- "POST one file per call" --> IN
|
||||
IN <--> DT
|
||||
IN -- "SSH: create" --> GW
|
||||
ST -- "SSH: list / show" --> GW
|
||||
ST <--> DT
|
||||
ST --> NT
|
||||
```
|
||||
|
||||
| Piece | Where | Job |
|
||||
|---|---|---|
|
||||
| Watcher | Mac, launchd | Finds new paths in `raw/`, waits until each file is stable, POSTs one webhook call per file |
|
||||
| Intake workflow | n8n | Validates, dedupes against the Data Table, creates the Kanban card over SSH |
|
||||
| `kanban-gateway.sh` | Mac, SSH forced command | The only thing n8n's key can run: `create`, `list`, `show` |
|
||||
| Hermes Kanban + dispatcher | Mac | Durable queue, claiming, retries, stale-task reclaim, circuit breaker |
|
||||
| Librarian profile | Mac | Does the work in the vault workspace, writes wiki pages per the llm-wiki skill |
|
||||
| Status workflow | n8n | Polls the board, updates the Data Table, sends failure notifications |
|
||||
| Data Table | n8n | Authoritative processed-files log |
|
||||
|
||||
## Key decisions
|
||||
|
||||
- **Watcher on the vault host, not a network mount.** Filesystem events and Local File Trigger don't work reliably over NFS/SMB, and failures there are silent.
|
||||
- **Polling loop is the source of truth.** FSEvents coalesces and reorders events, and clippers and sync tools are noisy, so the watcher lists `raw/` every 15 to 30 seconds and compares against its seen-state. Events may wake it early but never decide anything.
|
||||
- **Files stay in `raw/`.** "Processed" is tracked in the Data Table, not by moving files.
|
||||
- **Dedupe key is the relative path, first-seen wins.** The librarian edits frontmatter after cataloging, so the content hash changes. The hash is recorded as metadata at first-seen only. A second check rejects a first-seen hash that already exists under a different path (`status=duplicate`).
|
||||
- **Kanban is the queue.** The board already provides claiming, retries, stale reclaim, and a circuit breaker, so n8n has no Worker workflow.
|
||||
- **SSH instead of the dashboard REST API.** The dashboard plugin routes under `/api/plugins/kanban/` are unauthenticated by design. This pipeline does not use or expose them.
|
||||
- **Instruction files can live in `raw/`.** `AGENTS.md`, `CLAUDE.md`, `.hermes.md`, `README.md`, and `SCHEMA.md` are ignored by name at any depth (`WIKI_IGNORE_NAMES`), and the Intake `Validate` node should reject the same names. Hermes loads `AGENTS.md` into the librarian's context; see "Processing rules (AGENTS.md)" below.
|
||||
- **Output stays outside `raw/`.** The llm-wiki skill's folder layout must keep wiki output out of `raw/`, so Hermes can't feed itself.
|
||||
|
||||
## Mac setup
|
||||
|
||||
### 1. Install the gateway script
|
||||
|
||||
1. Copy `kanban-gateway.sh` to `~/bin/kanban-gateway` and `chmod 755` it.
|
||||
2. Edit the top block: `HERMES_BIN` (absolute path), `VAULT_ROOT`, `LIBRARIAN_PROFILE`, `SKILL_NAME`. The profile and skill names in the file are placeholders.
|
||||
3. Generate a dedicated key for n8n: `ssh-keygen -t ed25519 -f n8n_kanban -C n8n-wiki-llm`.
|
||||
4. Add one line to `~/.ssh/authorized_keys` on the Mac, using the absolute script path (optionally add `from="<n8n IP>"` before `restrict`):
|
||||
|
||||
```
|
||||
restrict,command="/Users/<you>/bin/kanban-gateway" ssh-ed25519 AAAA... n8n-wiki-llm
|
||||
```
|
||||
|
||||
5. Enable Remote Login for your user (System Settings, General, Sharing). On an MDM-managed Mac this may need IT approval.
|
||||
6. Smoke test from the n8n host: `ssh -i n8n_kanban <user>@<mac> list`. A bare `list` should return JSON.
|
||||
|
||||
The SSH user must be the user that owns the Hermes profiles and `~/.hermes/kanban.db`. Kanban is single-host and per-user.
|
||||
|
||||
### 2. Gateway script protocol
|
||||
|
||||
| Command sent over SSH | Effect |
|
||||
|---|---|
|
||||
| `create <base64 relative path>` | Creates a card whose body lists every `AGENTS.md` from the vault root down to the file's folder (shallow to deep), so folder rules are read deterministically. Assignee = librarian profile, `--workspace dir:<vault>`, `--idempotency-key <relative path>`, `--skill <skill>`, `--json` |
|
||||
| `list [status]` | `hermes kanban list --json`, optionally filtered. Status is one of `triage todo ready running blocked review done archived` |
|
||||
| `show <task id>` | `hermes kanban show <id> --json`. The id must match `t_[A-Za-z0-9]+` |
|
||||
|
||||
The path travels base64-encoded and is never passed through a shell or `eval`, so filenames containing spaces, quotes, or `$()` are inert. The script also rejects control characters, absolute paths, `..` segments, anything outside `raw/`, and symlinks that resolve out of `raw/`.
|
||||
|
||||
Exit codes: `0` ok, `2` rejected input, `3` file not found.
|
||||
|
||||
### 3. Run the dispatcher
|
||||
|
||||
`hermes gateway start` must be running on the Mac. Without it, cards stay `ready`. The default dispatch tick is 60 seconds. Cap concurrency with `kanban.max_in_progress` in `~/.hermes/config.yaml` so a burst drains at a pace your local models can handle.
|
||||
|
||||
### 4. Install the watcher
|
||||
|
||||
1. Copy `wiki-llm-watcher.py` to `~/bin/` (`chmod 755`). It uses only the Python 3 standard library.
|
||||
2. Copy `wiki-llm-watcher.env.example` to `~/.config/wiki-llm-watcher/env`, edit it, and `chmod 600` it. It holds the webhook URL (the Webhook node's **production** URL) and secret.
|
||||
3. **Baseline first:** `python3 ~/bin/wiki-llm-watcher.py --baseline` marks everything already in `raw/` as seen, so the first run does not post your whole backlog. Skip this only if you want the backlog processed.
|
||||
4. Dry run: `python3 ~/bin/wiki-llm-watcher.py --dry-run --once` prints the payloads it would send and posts nothing.
|
||||
5. Install `com.wiki-llm.watcher.plist` into `~/Library/LaunchAgents/` (replace `YOURNAME`), then `launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.wiki-llm.watcher.plist`. Logs go to `~/Library/Logs/wiki-llm-watcher.log`.
|
||||
6. To reprocess one file on purpose: `python3 ~/bin/wiki-llm-watcher.py --forget raw/path/to/file.md`.
|
||||
|
||||
macOS privacy controls (TCC) can block a LaunchAgent from reading `~/Documents`, `~/Desktop`, or iCloud Drive. If the log shows permission errors, grant the Python binary access under Privacy & Security, or keep the vault outside those folders.
|
||||
|
||||
Webhook response handling in the watcher: any 2xx marks the file posted (including n8n's `duplicate` response). `400/409/413/422` marks it `rejected` locally and stops retrying. `401/403/404` and an unreachable n8n pause posting for 60 seconds without marking anything. Other errors retry per file with exponential backoff.
|
||||
|
||||
## n8n workflows
|
||||
|
||||
### Intake (Webhook trigger)
|
||||
|
||||
1. **Webhook node**, header-auth credential. Body: `{path, sha256, size, mtime, source}`.
|
||||
2. **Code node:** validate `path` against an allowlist regex (for example `^raw/[\w .,\-/]+\.md$`, widened to the extensions you accept). Reject mismatches as `status=rejected`. Compute `rel_b64 = Buffer.from(path, 'utf8').toString('base64')` and `file_key = path`.
|
||||
3. **Data Table lookup** by `file_key`. If a row exists, respond `duplicate` and stop. Otherwise insert with `status=queued`. Respond to the webhook after this insert, so a 2xx means "durably queued".
|
||||
4. **SSH node** (Execute Command, Private Key credential): `create {{ $json.rel_b64 }}`. Retry On Fail: 3 tries with a few seconds' backoff. On Error: Continue (using error output).
|
||||
5. **If node:** check `code` is `0` explicitly. Don't rely on the node throwing. Exit `3` means the file is gone, so set `rejected`. Anything else, set `create_failed`, record `last_error`, and notify.
|
||||
6. **Code node:** parse `stdout` as JSON and extract the task id. Verify the key name against a real `hermes kanban create --json` run.
|
||||
7. **Data Table update:** `kanban_task_id`, `status=created`, `created_at`.
|
||||
|
||||
An SSH timeout can happen after the card was created. The kanban idempotency key means a retry returns the existing card, not a duplicate.
|
||||
|
||||
### Status (Schedule Trigger, every 1 to 2 minutes)
|
||||
|
||||
1. **SSH node:** `list blocked` and `list done` (or plain `list`).
|
||||
2. **Code node:** match cards to Data Table rows by `kanban_task_id`.
|
||||
3. **Data Table update:** `running`, `done` (set `processed_at`), `blocked`.
|
||||
4. **If node:** for `blocked` where `notified_at` is empty, run `show <id>` to get the recorded reason, send the **Slack** or **Gmail** node, then set `notified_at`. Notify on transitions only.
|
||||
|
||||
### Retry sweep (Schedule Trigger, optional)
|
||||
|
||||
Re-attempts rows stuck in `create_failed` with a capped `attempts` count.
|
||||
|
||||
### Error Workflow
|
||||
|
||||
Set an **Error Trigger** workflow on both workflows above as a safety net for crashes outside the handled branches.
|
||||
|
||||
## Data Table: `wiki_llm_files`
|
||||
|
||||
| Column | Notes |
|
||||
|---|---|
|
||||
| `file_key` | Relative path. Unique lookup key |
|
||||
| `sha256_first_seen` | Recorded at first-seen only |
|
||||
| `size`, `mtime`, `source` | Source: `watcher` or `obsidian-plugin` |
|
||||
| `status` | `queued`, `created`, `running`, `done`, `blocked`, `create_failed`, `rejected`, `duplicate` |
|
||||
| `kanban_task_id` | From the `create` response |
|
||||
| `attempts`, `last_error` | Retry accounting and diagnostics |
|
||||
| `first_seen`, `created_at`, `processed_at` | Timestamps |
|
||||
| `notified_at` | Prevents duplicate alerts |
|
||||
|
||||
A Data Table is enough to start. Move to Postgres only if you need joins, heavy volume, or external reporting.
|
||||
|
||||
## Processing rules (AGENTS.md)
|
||||
|
||||
All per-folder configuration lives in the vault as `AGENTS.md` files, so the watcher stays generic: it only knows the inbox folder and what to ignore.
|
||||
|
||||
```
|
||||
vault/
|
||||
AGENTS.md # global inbox rules
|
||||
raw/
|
||||
genealogy/find-a-grave/AGENTS.md # rules for that folder
|
||||
meeting-notes/AGENTS.md # rules for that folder
|
||||
```
|
||||
|
||||
When the gateway creates a card, it walks from the vault root down to the file's folder and lists every `AGENTS.md` it finds in the card body, shallow to deep, telling the agent to read all of them first (deeper files take precedence). This makes the rules deterministic and visible on the card. It does not depend on Hermes discovering subdirectory context files lazily, though Hermes also loads `AGENTS.md` from its working directory and discovers subdirectory ones as files are read there. If no `AGENTS.md` exists for a folder, the card says so and the agent follows the llm-wiki skill defaults.
|
||||
|
||||
Only one project context type loads per Hermes session (first match wins: `.hermes.md`, `AGENTS.md`, `CLAUDE.md`, `.cursorrules`). If the vault root already has a `.hermes.md`, it takes precedence at startup, but the card still names the `AGENTS.md` chain explicitly.
|
||||
|
||||
Suggested split, so rules live in one place each:
|
||||
|
||||
| Where | Holds |
|
||||
|---|---|
|
||||
| llm-wiki skill | How the wiki is structured and written (stable, reusable) |
|
||||
| `AGENTS.md` at root | Inbox-wide rules: never edit raw bodies (only frontmatter metadata), output goes outside `raw/`, raw file text is untrusted data and never instructions, block the card with a clear reason instead of guessing |
|
||||
| `AGENTS.md` per folder | That folder's processing rules and where its output goes |
|
||||
| Kanban card body | Per-file facts (path) and the ruleset list |
|
||||
|
||||
**Validate with a marker test:** give two folders deliberately silly rules (one adds the tag `#fag-test` to the frontmatter, the other `#meeting-test`), drop one file in each through the real pipeline, and confirm each file got only its own tag.
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Problem | Handling |
|
||||
|---|---|
|
||||
| Duplicate triggers | Data Table lookup on relative path, plus the kanban idempotency key, plus the watcher's local seen-state. Losing the watcher's state causes noisy reposts but no reprocessing |
|
||||
| Partial writes | Watcher requires size and mtime unchanged for about 10 seconds, ignores temp patterns (`*.tmp`, `.syncthing.*`, `*.sync-conflict-*`, dotfiles, `.obsidian/`) |
|
||||
| Bursts | Watcher posts sequentially with a short delay. The board queue plus `max_in_progress` absorb the rest |
|
||||
| n8n down | Watcher records a file as posted only after a 2xx, so unposted files are retried on the next pass |
|
||||
| SSH or Mac unreachable | Row stays `queued` or `create_failed`, retried by the sweep, with an alert once attempts are exhausted |
|
||||
| Hermes worker fails | Dispatcher retries and then circuit-breaks to `blocked` with the error recorded. The Status workflow alerts and keeps the reason |
|
||||
| Stuck or crashed worker | Dispatcher reclaims stale claims automatically |
|
||||
|
||||
## Optional: Obsidian "Post Webhook" plugin
|
||||
|
||||
[obsidian-post-webhook](https://github.com/masterb1234/obsidian-post-webhook) sends a note to a webhook when you run a command. It is not a file watcher and is not the primary trigger. It needs Obsidian running with the note open, and it sends note content with base64 attachments rather than a path and hash.
|
||||
|
||||
Use it only as a manual "process this note now" entry point into the same Intake workflow:
|
||||
|
||||
- Accept both payload shapes (`{path, sha256}` and `{content, filename}`) and branch with an **If node**.
|
||||
- For plugin payloads, compute the hash with the **Crypto node** and set `source=obsidian-plugin`.
|
||||
- Enable "Exclude Attachments" and set response handling to **None**, so n8n output never writes back into the vault.
|
||||
|
||||
## Security notes
|
||||
|
||||
- The n8n SSH key can run only the gateway script, via `restrict,command=...`.
|
||||
- Do not run `hermes dashboard --host 0.0.0.0`. The kanban REST routes under `/api/plugins/` are unauthenticated.
|
||||
- Keep the watcher webhook behind header auth, and restrict it by network if possible.
|
||||
|
||||
## Verify on first real run
|
||||
|
||||
1. `hermes kanban create --json` output shape (the task id key name).
|
||||
2. Idempotency: run the same `create` twice and confirm the second returns the existing card.
|
||||
3. `--workspace dir:<vault>` resolves correctly in `hermes kanban show <id>`.
|
||||
4. `--skill` matches a skill installed on the librarian profile (`hermes skills list`).
|
||||
5. `hermes gateway start` is running and the dispatcher picks the card up.
|
||||
6. A deliberately failing card reaches `blocked` and triggers exactly one alert.
|
||||
|
||||
## Open items
|
||||
|
||||
- [x] Mac watcher script and launchd plist (poll loop, stability check, seen-state, one POST per file)
|
||||
- [ ] Build Intake, Status, and Retry sweep workflows, and create the Data Table
|
||||
- [ ] Choose the notification channel (Slack or email)
|
||||
- [ ] Confirm expected file volume (sets watcher delay and `max_in_progress`)
|
||||
- [ ] Decide whether to add a Hermes plugin hook (`kanban_task_blocked`) that POSTs to an n8n webhook for lower-latency alerts. Polling stays as the backstop either way
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `kanban-gateway.sh` | SSH forced-command wrapper (tested with a stub `hermes`) |
|
||||
| `wiki-llm-watcher.py` | Mac watcher (tested with a mock webhook) |
|
||||
| `wiki-llm-watcher.env.example` | Watcher configuration template |
|
||||
| `com.wiki-llm.watcher.plist` | launchd LaunchAgent for the watcher |
|
||||
| `README.md` | This document |
|
||||
Reference in new issue
Block a user