d974c75d7c
- Stable udev device symlinks (/dev/jmri/nce, /dev/jmri/loconet, /dev/jmri/lcc) - jmri-monitor: polls Leviton Decora Smart switch to start/stop JMRI automatically - Quiet hours 1-10 AM (no polling) - 30s off-delay before shutdown - LCRR config cloned from Gitea (ssh://gitea.mk-labs.cloud:2221/rblundon/LCRR.git) - ~/.jmri symlinked to LCRR repo for GitOps config management - jmri-gui: X11 remote GUI access (PanelPro/DecoderPro) via ssh -X as jmri user - Stops daemon, launches GUI, restarts daemon on exit if layout still on - jmri user gets login shell + SSH key for GUI sessions - Full JRE installed (openjdk-21-jre) for AWT/X11 support
15 KiB
15 KiB
Firecrawl Architecture Diagram
High-Level Overview
┌─────────────────────────────────────────────────────────────────────────┐
│ External Access │
│ spaceship-earth.local.mk-labs.cloud / firecrawl.local.mk-labs.cloud │
│ │ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ Gateway API │ │
│ │ (HTTPRoute) │ │
│ │ TLS Termination │ │
│ └────────┬─────────┘ │
└─────────────────────────────────┼──────────────────────────────────────┘
│
┌─────────────────────────────────┼──────────────────────────────────────┐
│ Firecrawl Namespace │
│ │ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ API Service │ │
│ │ (ClusterIP) │ │
│ │ Port 3002 │ │
│ └────────┬─────────┘ │
│ │ │
│ ┌──────────────────┼──────────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────────┐ ┌─────────────┐ ┌──────────────────┐ │
│ │ API Deployment │ │ Worker │ │ NUQ Worker │ │
│ │ (firecrawl-api)│ │ Deployment │ │ Deployment │ │
│ │ │ │(firecrawl- │ │ (firecrawl-api) │ │
│ │ Entrypoint: │ │ api) │ │ │ │
│ │ dist/src/ │ │ │ │ Entrypoint: │ │
│ │ index.js │ │ Entrypoint: │ │ dist/src/ │ │
│ │ │ │ dist/src/ │ │ services/worker/│ │
│ │ 4-6GB / 2 CPU │ │ services/ │ │ nuq-worker.js │ │
│ │ │ │ queue- │ │ │ │
│ │ Replicas: 1 │ │ worker.js │ │ 3-4GB / 1 CPU │ │
│ │ │ │ │ │ │ │
│ │ Health: │ │ 3-4GB/1 CPU │ │ Replicas: 1 │ │
│ │ /v0/health/* │ │ │ │ │ │
│ │ │ │ Replicas: 1 │ │ │ │
│ └────────┬────────┘ └──────┬──────┘ └────────┬─────────┘ │
│ │ │ │ │
│ └─────────┬────────┴──────────────────┘ │
│ │ │
│ ┌─────────┼──────────────┬──────────────┐ │
│ │ │ │ │ │
│ ▼ ▼ ▼ ▼ │
│ ┌────────────┐ ┌──────────┐ ┌──────────┐ ┌───────────────┐ │
│ │ Playwright │ │ Redis │ │PostgreSQL│ │ RabbitMQ │ │
│ │ Service │ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ Deployment │ │Deployment│ │StatefulSet│ │ Deployment │ │
│ │ (Harbor) │ │(Upstream)│ │ (Harbor) │ │ (Upstream) │ │
│ │ │ │ │ │ │ │ │ │
│ │ Service: │ │ Service: │ │ Service: │ │ Service: │ │
│ │ 3000 │ │ 6379 │ │ 5432 │ │ 5672, 15672 │ │
│ │ │ │ │ │ │ │ │ │
│ │ 4GB/2 CPU │ │ 1GB/0.5 │ │ 2GB/1 CPU│ │ 1GB/0.5 CPU │ │
│ │ │ │ │ │ │ │ │ │
│ │ tmpfs: │ │ │ │ PVC: │ │ Healthcheck: │ │
│ │ 1GB │ │ │ │ 10GB │ │ Required │ │
│ └────────────┘ └──────────┘ └──────────┘ └───────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────┐ │
│ │ NFS PVC │ │
│ │ 10GB │ │
│ │(nfs- │ │
│ │emporium) │ │
│ └──────────┘ │
│ │
│ Configuration: │
│ ┌──────────────┐ ┌────────────────┐ │
│ │ ConfigMap │ │ ExternalSecret │ │
│ │ (firecrawl- │ │ (firecrawl- │ │
│ │ config) │ │ secrets) │ │
│ │ │ │ │ │
│ │ - URLs │ │ ┌──────────┐ │ │
│ │ - Ports │ │ │1Password │ │ │
│ │ - Tuning │ │ │ Vault │ │ │
│ │ │ │ └────┬─────┘ │ │
│ └──────────────┘ │ │ │ │
│ │ ▼ │ │
│ │ - postgres- │ │
│ │ password │ │
│ │ - bull-auth- │ │
│ │ key │ │
│ └────────────────┘ │
└────────────────────────────────────────────────────────────────────────┘
Service Communication Flow
API Request Flow
External User
│
▼
Gateway API (TLS termination)
│
▼
API Service (port 3002)
│
├─────► Playwright Service (browser automation)
│ └─► Returns rendered HTML/Markdown
│
├─────► Redis (queue jobs, cache results)
│
├─────► PostgreSQL (store job metadata)
│
└─────► RabbitMQ (publish job events)
Background Job Processing Flow
API receives request
│
▼
Job queued in Redis
│
▼
RabbitMQ notifies workers
│
├─────► Worker picks up job
│ └─► Processes scraping tasks
│
└─────► NUQ Worker picks up database jobs
└─► Processes queue from PostgreSQL
Database Queue Flow (NUQ)
Job created in PostgreSQL (nuq.queue_scrape table)
│
▼
NUQ Worker polls for jobs (prefetch)
│
├─► Status: queued → active
│
├─► Worker processes job
│ └─► Calls Playwright or direct fetch
│
└─► Status: active → completed/failed
└─► Results stored in returnvalue column
Build Pipeline Flow
GitHub: mendableai/firecrawl
│
▼
Tekton Pipeline (innoventions namespace)
│
├─────► firecrawl-api-build
│ │
│ ├─► Git Clone Task
│ │
│ ├─► Kaniko Build Task
│ │ └─► Multi-stage: Go → Node → Runtime
│ │
│ └─► Push to Harbor
│ └─► the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest
│
├─────► firecrawl-playwright-build
│ │
│ ├─► Git Clone Task
│ │
│ ├─► Kaniko Build Task
│ │ └─► Node.js + Chromium install
│ │
│ └─► Push to Harbor
│ └─► .../firecrawl-playwright:latest
│
└─────► firecrawl-postgres-build
│
├─► Git Clone Task
│
├─► Kaniko Build Task
│ └─► postgres:16 + pg_cron + nuq.sql
│
└─► Push to Harbor
└─► .../firecrawl-postgres:latest
Deployment Flow (ArgoCD)
Gitea Repository (homelab)
│
└─► cluster/applications/firecrawl/
│
▼
ArgoCD Application (sync)
│
├─► Wave 0: Namespace
│
├─► Wave 1: ConfigMap, ExternalSecret
│
├─► Wave 2: PostgreSQL StatefulSet + PVC
│ Redis Deployment
│ RabbitMQ Deployment
│
├─► Wave 3: Playwright Deployment
│ (waits for infrastructure)
│
├─► Wave 4: API Deployment
│ Worker Deployments
│ (waits for all dependencies)
│
└─► Wave 5: Services, HTTPRoute
Resource Distribution
Total Cluster Capacity: ~48 CPU / ~96GB RAM (6 nodes)
Firecrawl Allocation:
┌────────────────────────────────────┐
│ API: 2 CPU / 4-6GB │ ████████████
│ Worker: 1 CPU / 3-4GB │ ██████
│ NUQ Worker: 1 CPU / 3-4GB │ ██████
│ Playwright: 2 CPU / 4GB │ ████████████
│ PostgreSQL: 1 CPU / 2GB │ ██████
│ Redis: 0.5 CPU / 1GB │ ███
│ RabbitMQ: 0.5 CPU / 1GB │ ███
├────────────────────────────────────┤
│ TOTAL: 8 CPU / 22GB RAM │
└────────────────────────────────────┘
Percentage of cluster: ~17% CPU, ~23% RAM
Headroom available: ✅ Excellent
Data Flow
Scrape Request Example
1. User → POST /v1/scrape {"url": "https://example.com"}
│
2. API validates request
│
3. API creates job in PostgreSQL (nuq.queue_scrape)
│
4. API queues job in Redis
│
5. RabbitMQ notifies workers
│
6. Worker picks up job
│
7. Worker calls Playwright service
│ └─► Playwright launches Chromium
│ └─► Renders page (handles JS)
│ └─► Returns HTML
│
8. Worker converts HTML → Markdown (Go library)
│
9. Worker stores result in PostgreSQL (returnvalue column)
│
10. Worker updates job status: completed
│
11. API returns result to user
└─► {"markdown": "...", "html": "...", "metadata": {...}}
Network Policies (Future Enhancement)
firecrawl namespace:
│
├─► Ingress Rules:
│ ├─ Allow: Gateway API → API Service (port 3002)
│ └─ Deny: All other external traffic
│
├─► Egress Rules:
│ ├─ Allow: API → Playwright (port 3000)
│ ├─ Allow: API → Redis (port 6379)
│ ├─ Allow: API → PostgreSQL (port 5432)
│ ├─ Allow: API → RabbitMQ (port 5672)
│ ├─ Allow: All → Internet (for web scraping)
│ └─ Deny: All other cluster traffic
│
└─► Inter-Pod Rules:
├─ Allow: API → All infrastructure services
├─ Allow: Workers → All infrastructure services
├─ Deny: PostgreSQL → Internet (security)
└─ Deny: Redis → Internet (security)
Monitoring & Observability (Future Enhancement)
Prometheus Metrics:
│
├─► API Metrics (port 3002/metrics)
│ ├─ Request rate
│ ├─ Response times
│ ├─ Job queue depth
│ └─ Error rates
│
├─► Worker Metrics (port 3005/metrics)
│ ├─ Jobs processed
│ ├─ Processing times
│ └─ Success/failure rates
│
├─► PostgreSQL Metrics
│ ├─ Connection pool usage
│ ├─ Query performance
│ └─ Table sizes
│
└─► Playwright Metrics
├─ Browser pool usage
├─ Page load times
└─ Chromium memory usage
Grafana Dashboards:
├─ Firecrawl Overview
├─ Job Processing Metrics
├─ Service Health
└─ Resource Utilization
Diagram Version: 1.0
Last Updated: June 6, 2026
Created By: Rocket Raccoon (CI/CD Specialist)