feat(jmri): headless JMRI server with Leviton layout power monitor and X11 GUI mode
- Stable udev device symlinks (/dev/jmri/nce, /dev/jmri/loconet, /dev/jmri/lcc) - jmri-monitor: polls Leviton Decora Smart switch to start/stop JMRI automatically - Quiet hours 1-10 AM (no polling) - 30s off-delay before shutdown - LCRR config cloned from Gitea (ssh://gitea.mk-labs.cloud:2221/rblundon/LCRR.git) - ~/.jmri symlinked to LCRR repo for GitOps config management - jmri-gui: X11 remote GUI access (PanelPro/DecoderPro) via ssh -X as jmri user - Stops daemon, launches GUI, restarts daemon on exit if layout still on - jmri user gets login shell + SSH key for GUI sessions - Full JRE installed (openjdk-21-jre) for AWT/X11 support
This commit is contained in:
Hermes Agent service account
337
cluster/applications/firecrawl/ARCHITECTURE.md
Normal file
337
cluster/applications/firecrawl/ARCHITECTURE.md
Normal file
@@ -0,0 +1,337 @@
|
||||
# Firecrawl Architecture Diagram
|
||||
|
||||
## High-Level Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ External Access │
|
||||
│ spaceship-earth.local.mk-labs.cloud / firecrawl.local.mk-labs.cloud │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────┐ │
|
||||
│ │ Gateway API │ │
|
||||
│ │ (HTTPRoute) │ │
|
||||
│ │ TLS Termination │ │
|
||||
│ └────────┬─────────┘ │
|
||||
└─────────────────────────────────┼──────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────────────┼──────────────────────────────────────┐
|
||||
│ Firecrawl Namespace │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────┐ │
|
||||
│ │ API Service │ │
|
||||
│ │ (ClusterIP) │ │
|
||||
│ │ Port 3002 │ │
|
||||
│ └────────┬─────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────────────┼──────────────────┐ │
|
||||
│ │ │ │ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ ┌─────────────────┐ ┌─────────────┐ ┌──────────────────┐ │
|
||||
│ │ API Deployment │ │ Worker │ │ NUQ Worker │ │
|
||||
│ │ (firecrawl-api)│ │ Deployment │ │ Deployment │ │
|
||||
│ │ │ │(firecrawl- │ │ (firecrawl-api) │ │
|
||||
│ │ Entrypoint: │ │ api) │ │ │ │
|
||||
│ │ dist/src/ │ │ │ │ Entrypoint: │ │
|
||||
│ │ index.js │ │ Entrypoint: │ │ dist/src/ │ │
|
||||
│ │ │ │ dist/src/ │ │ services/worker/│ │
|
||||
│ │ 4-6GB / 2 CPU │ │ services/ │ │ nuq-worker.js │ │
|
||||
│ │ │ │ queue- │ │ │ │
|
||||
│ │ Replicas: 1 │ │ worker.js │ │ 3-4GB / 1 CPU │ │
|
||||
│ │ │ │ │ │ │ │
|
||||
│ │ Health: │ │ 3-4GB/1 CPU │ │ Replicas: 1 │ │
|
||||
│ │ /v0/health/* │ │ │ │ │ │
|
||||
│ │ │ │ Replicas: 1 │ │ │ │
|
||||
│ └────────┬────────┘ └──────┬──────┘ └────────┬─────────┘ │
|
||||
│ │ │ │ │
|
||||
│ └─────────┬────────┴──────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌─────────┼──────────────┬──────────────┐ │
|
||||
│ │ │ │ │ │
|
||||
│ ▼ ▼ ▼ ▼ │
|
||||
│ ┌────────────┐ ┌──────────┐ ┌──────────┐ ┌───────────────┐ │
|
||||
│ │ Playwright │ │ Redis │ │PostgreSQL│ │ RabbitMQ │ │
|
||||
│ │ Service │ │ │ │ │ │ │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ Deployment │ │Deployment│ │StatefulSet│ │ Deployment │ │
|
||||
│ │ (Harbor) │ │(Upstream)│ │ (Harbor) │ │ (Upstream) │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ Service: │ │ Service: │ │ Service: │ │ Service: │ │
|
||||
│ │ 3000 │ │ 6379 │ │ 5432 │ │ 5672, 15672 │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ 4GB/2 CPU │ │ 1GB/0.5 │ │ 2GB/1 CPU│ │ 1GB/0.5 CPU │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ tmpfs: │ │ │ │ PVC: │ │ Healthcheck: │ │
|
||||
│ │ 1GB │ │ │ │ 10GB │ │ Required │ │
|
||||
│ └────────────┘ └──────────┘ └──────────┘ └───────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────┐ │
|
||||
│ │ NFS PVC │ │
|
||||
│ │ 10GB │ │
|
||||
│ │(nfs- │ │
|
||||
│ │emporium) │ │
|
||||
│ └──────────┘ │
|
||||
│ │
|
||||
│ Configuration: │
|
||||
│ ┌──────────────┐ ┌────────────────┐ │
|
||||
│ │ ConfigMap │ │ ExternalSecret │ │
|
||||
│ │ (firecrawl- │ │ (firecrawl- │ │
|
||||
│ │ config) │ │ secrets) │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ - URLs │ │ ┌──────────┐ │ │
|
||||
│ │ - Ports │ │ │1Password │ │ │
|
||||
│ │ - Tuning │ │ │ Vault │ │ │
|
||||
│ │ │ │ └────┬─────┘ │ │
|
||||
│ └──────────────┘ │ │ │ │
|
||||
│ │ ▼ │ │
|
||||
│ │ - postgres- │ │
|
||||
│ │ password │ │
|
||||
│ │ - bull-auth- │ │
|
||||
│ │ key │ │
|
||||
│ └────────────────┘ │
|
||||
└────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Service Communication Flow
|
||||
|
||||
### API Request Flow
|
||||
```
|
||||
External User
|
||||
│
|
||||
▼
|
||||
Gateway API (TLS termination)
|
||||
│
|
||||
▼
|
||||
API Service (port 3002)
|
||||
│
|
||||
├─────► Playwright Service (browser automation)
|
||||
│ └─► Returns rendered HTML/Markdown
|
||||
│
|
||||
├─────► Redis (queue jobs, cache results)
|
||||
│
|
||||
├─────► PostgreSQL (store job metadata)
|
||||
│
|
||||
└─────► RabbitMQ (publish job events)
|
||||
```
|
||||
|
||||
### Background Job Processing Flow
|
||||
```
|
||||
API receives request
|
||||
│
|
||||
▼
|
||||
Job queued in Redis
|
||||
│
|
||||
▼
|
||||
RabbitMQ notifies workers
|
||||
│
|
||||
├─────► Worker picks up job
|
||||
│ └─► Processes scraping tasks
|
||||
│
|
||||
└─────► NUQ Worker picks up database jobs
|
||||
└─► Processes queue from PostgreSQL
|
||||
```
|
||||
|
||||
### Database Queue Flow (NUQ)
|
||||
```
|
||||
Job created in PostgreSQL (nuq.queue_scrape table)
|
||||
│
|
||||
▼
|
||||
NUQ Worker polls for jobs (prefetch)
|
||||
│
|
||||
├─► Status: queued → active
|
||||
│
|
||||
├─► Worker processes job
|
||||
│ └─► Calls Playwright or direct fetch
|
||||
│
|
||||
└─► Status: active → completed/failed
|
||||
└─► Results stored in returnvalue column
|
||||
```
|
||||
|
||||
## Build Pipeline Flow
|
||||
|
||||
```
|
||||
GitHub: mendableai/firecrawl
|
||||
│
|
||||
▼
|
||||
Tekton Pipeline (innoventions namespace)
|
||||
│
|
||||
├─────► firecrawl-api-build
|
||||
│ │
|
||||
│ ├─► Git Clone Task
|
||||
│ │
|
||||
│ ├─► Kaniko Build Task
|
||||
│ │ └─► Multi-stage: Go → Node → Runtime
|
||||
│ │
|
||||
│ └─► Push to Harbor
|
||||
│ └─► the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest
|
||||
│
|
||||
├─────► firecrawl-playwright-build
|
||||
│ │
|
||||
│ ├─► Git Clone Task
|
||||
│ │
|
||||
│ ├─► Kaniko Build Task
|
||||
│ │ └─► Node.js + Chromium install
|
||||
│ │
|
||||
│ └─► Push to Harbor
|
||||
│ └─► .../firecrawl-playwright:latest
|
||||
│
|
||||
└─────► firecrawl-postgres-build
|
||||
│
|
||||
├─► Git Clone Task
|
||||
│
|
||||
├─► Kaniko Build Task
|
||||
│ └─► postgres:16 + pg_cron + nuq.sql
|
||||
│
|
||||
└─► Push to Harbor
|
||||
└─► .../firecrawl-postgres:latest
|
||||
```
|
||||
|
||||
## Deployment Flow (ArgoCD)
|
||||
|
||||
```
|
||||
Gitea Repository (homelab)
|
||||
│
|
||||
└─► cluster/applications/firecrawl/
|
||||
│
|
||||
▼
|
||||
ArgoCD Application (sync)
|
||||
│
|
||||
├─► Wave 0: Namespace
|
||||
│
|
||||
├─► Wave 1: ConfigMap, ExternalSecret
|
||||
│
|
||||
├─► Wave 2: PostgreSQL StatefulSet + PVC
|
||||
│ Redis Deployment
|
||||
│ RabbitMQ Deployment
|
||||
│
|
||||
├─► Wave 3: Playwright Deployment
|
||||
│ (waits for infrastructure)
|
||||
│
|
||||
├─► Wave 4: API Deployment
|
||||
│ Worker Deployments
|
||||
│ (waits for all dependencies)
|
||||
│
|
||||
└─► Wave 5: Services, HTTPRoute
|
||||
```
|
||||
|
||||
## Resource Distribution
|
||||
|
||||
```
|
||||
Total Cluster Capacity: ~48 CPU / ~96GB RAM (6 nodes)
|
||||
|
||||
Firecrawl Allocation:
|
||||
┌────────────────────────────────────┐
|
||||
│ API: 2 CPU / 4-6GB │ ████████████
|
||||
│ Worker: 1 CPU / 3-4GB │ ██████
|
||||
│ NUQ Worker: 1 CPU / 3-4GB │ ██████
|
||||
│ Playwright: 2 CPU / 4GB │ ████████████
|
||||
│ PostgreSQL: 1 CPU / 2GB │ ██████
|
||||
│ Redis: 0.5 CPU / 1GB │ ███
|
||||
│ RabbitMQ: 0.5 CPU / 1GB │ ███
|
||||
├────────────────────────────────────┤
|
||||
│ TOTAL: 8 CPU / 22GB RAM │
|
||||
└────────────────────────────────────┘
|
||||
|
||||
Percentage of cluster: ~17% CPU, ~23% RAM
|
||||
Headroom available: ✅ Excellent
|
||||
```
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Scrape Request Example
|
||||
```
|
||||
1. User → POST /v1/scrape {"url": "https://example.com"}
|
||||
│
|
||||
2. API validates request
|
||||
│
|
||||
3. API creates job in PostgreSQL (nuq.queue_scrape)
|
||||
│
|
||||
4. API queues job in Redis
|
||||
│
|
||||
5. RabbitMQ notifies workers
|
||||
│
|
||||
6. Worker picks up job
|
||||
│
|
||||
7. Worker calls Playwright service
|
||||
│ └─► Playwright launches Chromium
|
||||
│ └─► Renders page (handles JS)
|
||||
│ └─► Returns HTML
|
||||
│
|
||||
8. Worker converts HTML → Markdown (Go library)
|
||||
│
|
||||
9. Worker stores result in PostgreSQL (returnvalue column)
|
||||
│
|
||||
10. Worker updates job status: completed
|
||||
│
|
||||
11. API returns result to user
|
||||
└─► {"markdown": "...", "html": "...", "metadata": {...}}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Network Policies (Future Enhancement)
|
||||
|
||||
```
|
||||
firecrawl namespace:
|
||||
│
|
||||
├─► Ingress Rules:
|
||||
│ ├─ Allow: Gateway API → API Service (port 3002)
|
||||
│ └─ Deny: All other external traffic
|
||||
│
|
||||
├─► Egress Rules:
|
||||
│ ├─ Allow: API → Playwright (port 3000)
|
||||
│ ├─ Allow: API → Redis (port 6379)
|
||||
│ ├─ Allow: API → PostgreSQL (port 5432)
|
||||
│ ├─ Allow: API → RabbitMQ (port 5672)
|
||||
│ ├─ Allow: All → Internet (for web scraping)
|
||||
│ └─ Deny: All other cluster traffic
|
||||
│
|
||||
└─► Inter-Pod Rules:
|
||||
├─ Allow: API → All infrastructure services
|
||||
├─ Allow: Workers → All infrastructure services
|
||||
├─ Deny: PostgreSQL → Internet (security)
|
||||
└─ Deny: Redis → Internet (security)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Observability (Future Enhancement)
|
||||
|
||||
```
|
||||
Prometheus Metrics:
|
||||
│
|
||||
├─► API Metrics (port 3002/metrics)
|
||||
│ ├─ Request rate
|
||||
│ ├─ Response times
|
||||
│ ├─ Job queue depth
|
||||
│ └─ Error rates
|
||||
│
|
||||
├─► Worker Metrics (port 3005/metrics)
|
||||
│ ├─ Jobs processed
|
||||
│ ├─ Processing times
|
||||
│ └─ Success/failure rates
|
||||
│
|
||||
├─► PostgreSQL Metrics
|
||||
│ ├─ Connection pool usage
|
||||
│ ├─ Query performance
|
||||
│ └─ Table sizes
|
||||
│
|
||||
└─► Playwright Metrics
|
||||
├─ Browser pool usage
|
||||
├─ Page load times
|
||||
└─ Chromium memory usage
|
||||
|
||||
Grafana Dashboards:
|
||||
├─ Firecrawl Overview
|
||||
├─ Job Processing Metrics
|
||||
├─ Service Health
|
||||
└─ Resource Utilization
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Diagram Version:** 1.0
|
||||
**Last Updated:** June 6, 2026
|
||||
**Created By:** Rocket Raccoon (CI/CD Specialist)
|
||||
Reference in New Issue
Block a user