d974c75d7c
- Stable udev device symlinks (/dev/jmri/nce, /dev/jmri/loconet, /dev/jmri/lcc) - jmri-monitor: polls Leviton Decora Smart switch to start/stop JMRI automatically - Quiet hours 1-10 AM (no polling) - 30s off-delay before shutdown - LCRR config cloned from Gitea (ssh://gitea.mk-labs.cloud:2221/rblundon/LCRR.git) - ~/.jmri symlinked to LCRR repo for GitOps config management - jmri-gui: X11 remote GUI access (PanelPro/DecoderPro) via ssh -X as jmri user - Stops daemon, launches GUI, restarts daemon on exit if layout still on - jmri user gets login shell + SSH key for GUI sessions - Full JRE installed (openjdk-21-jre) for AWT/X11 support
338 lines
15 KiB
Markdown
338 lines
15 KiB
Markdown
# Firecrawl Architecture Diagram
|
|
|
|
## High-Level Overview
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ External Access │
|
|
│ spaceship-earth.local.mk-labs.cloud / firecrawl.local.mk-labs.cloud │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌──────────────────┐ │
|
|
│ │ Gateway API │ │
|
|
│ │ (HTTPRoute) │ │
|
|
│ │ TLS Termination │ │
|
|
│ └────────┬─────────┘ │
|
|
└─────────────────────────────────┼──────────────────────────────────────┘
|
|
│
|
|
┌─────────────────────────────────┼──────────────────────────────────────┐
|
|
│ Firecrawl Namespace │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌──────────────────┐ │
|
|
│ │ API Service │ │
|
|
│ │ (ClusterIP) │ │
|
|
│ │ Port 3002 │ │
|
|
│ └────────┬─────────┘ │
|
|
│ │ │
|
|
│ ┌──────────────────┼──────────────────┐ │
|
|
│ │ │ │ │
|
|
│ ▼ ▼ ▼ │
|
|
│ ┌─────────────────┐ ┌─────────────┐ ┌──────────────────┐ │
|
|
│ │ API Deployment │ │ Worker │ │ NUQ Worker │ │
|
|
│ │ (firecrawl-api)│ │ Deployment │ │ Deployment │ │
|
|
│ │ │ │(firecrawl- │ │ (firecrawl-api) │ │
|
|
│ │ Entrypoint: │ │ api) │ │ │ │
|
|
│ │ dist/src/ │ │ │ │ Entrypoint: │ │
|
|
│ │ index.js │ │ Entrypoint: │ │ dist/src/ │ │
|
|
│ │ │ │ dist/src/ │ │ services/worker/│ │
|
|
│ │ 4-6GB / 2 CPU │ │ services/ │ │ nuq-worker.js │ │
|
|
│ │ │ │ queue- │ │ │ │
|
|
│ │ Replicas: 1 │ │ worker.js │ │ 3-4GB / 1 CPU │ │
|
|
│ │ │ │ │ │ │ │
|
|
│ │ Health: │ │ 3-4GB/1 CPU │ │ Replicas: 1 │ │
|
|
│ │ /v0/health/* │ │ │ │ │ │
|
|
│ │ │ │ Replicas: 1 │ │ │ │
|
|
│ └────────┬────────┘ └──────┬──────┘ └────────┬─────────┘ │
|
|
│ │ │ │ │
|
|
│ └─────────┬────────┴──────────────────┘ │
|
|
│ │ │
|
|
│ ┌─────────┼──────────────┬──────────────┐ │
|
|
│ │ │ │ │ │
|
|
│ ▼ ▼ ▼ ▼ │
|
|
│ ┌────────────┐ ┌──────────┐ ┌──────────┐ ┌───────────────┐ │
|
|
│ │ Playwright │ │ Redis │ │PostgreSQL│ │ RabbitMQ │ │
|
|
│ │ Service │ │ │ │ │ │ │ │
|
|
│ │ │ │ │ │ │ │ │ │
|
|
│ │ Deployment │ │Deployment│ │StatefulSet│ │ Deployment │ │
|
|
│ │ (Harbor) │ │(Upstream)│ │ (Harbor) │ │ (Upstream) │ │
|
|
│ │ │ │ │ │ │ │ │ │
|
|
│ │ Service: │ │ Service: │ │ Service: │ │ Service: │ │
|
|
│ │ 3000 │ │ 6379 │ │ 5432 │ │ 5672, 15672 │ │
|
|
│ │ │ │ │ │ │ │ │ │
|
|
│ │ 4GB/2 CPU │ │ 1GB/0.5 │ │ 2GB/1 CPU│ │ 1GB/0.5 CPU │ │
|
|
│ │ │ │ │ │ │ │ │ │
|
|
│ │ tmpfs: │ │ │ │ PVC: │ │ Healthcheck: │ │
|
|
│ │ 1GB │ │ │ │ 10GB │ │ Required │ │
|
|
│ └────────────┘ └──────────┘ └──────────┘ └───────────────┘ │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌──────────┐ │
|
|
│ │ NFS PVC │ │
|
|
│ │ 10GB │ │
|
|
│ │(nfs- │ │
|
|
│ │emporium) │ │
|
|
│ └──────────┘ │
|
|
│ │
|
|
│ Configuration: │
|
|
│ ┌──────────────┐ ┌────────────────┐ │
|
|
│ │ ConfigMap │ │ ExternalSecret │ │
|
|
│ │ (firecrawl- │ │ (firecrawl- │ │
|
|
│ │ config) │ │ secrets) │ │
|
|
│ │ │ │ │ │
|
|
│ │ - URLs │ │ ┌──────────┐ │ │
|
|
│ │ - Ports │ │ │1Password │ │ │
|
|
│ │ - Tuning │ │ │ Vault │ │ │
|
|
│ │ │ │ └────┬─────┘ │ │
|
|
│ └──────────────┘ │ │ │ │
|
|
│ │ ▼ │ │
|
|
│ │ - postgres- │ │
|
|
│ │ password │ │
|
|
│ │ - bull-auth- │ │
|
|
│ │ key │ │
|
|
│ └────────────────┘ │
|
|
└────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Service Communication Flow
|
|
|
|
### API Request Flow
|
|
```
|
|
External User
|
|
│
|
|
▼
|
|
Gateway API (TLS termination)
|
|
│
|
|
▼
|
|
API Service (port 3002)
|
|
│
|
|
├─────► Playwright Service (browser automation)
|
|
│ └─► Returns rendered HTML/Markdown
|
|
│
|
|
├─────► Redis (queue jobs, cache results)
|
|
│
|
|
├─────► PostgreSQL (store job metadata)
|
|
│
|
|
└─────► RabbitMQ (publish job events)
|
|
```
|
|
|
|
### Background Job Processing Flow
|
|
```
|
|
API receives request
|
|
│
|
|
▼
|
|
Job queued in Redis
|
|
│
|
|
▼
|
|
RabbitMQ notifies workers
|
|
│
|
|
├─────► Worker picks up job
|
|
│ └─► Processes scraping tasks
|
|
│
|
|
└─────► NUQ Worker picks up database jobs
|
|
└─► Processes queue from PostgreSQL
|
|
```
|
|
|
|
### Database Queue Flow (NUQ)
|
|
```
|
|
Job created in PostgreSQL (nuq.queue_scrape table)
|
|
│
|
|
▼
|
|
NUQ Worker polls for jobs (prefetch)
|
|
│
|
|
├─► Status: queued → active
|
|
│
|
|
├─► Worker processes job
|
|
│ └─► Calls Playwright or direct fetch
|
|
│
|
|
└─► Status: active → completed/failed
|
|
└─► Results stored in returnvalue column
|
|
```
|
|
|
|
## Build Pipeline Flow
|
|
|
|
```
|
|
GitHub: mendableai/firecrawl
|
|
│
|
|
▼
|
|
Tekton Pipeline (innoventions namespace)
|
|
│
|
|
├─────► firecrawl-api-build
|
|
│ │
|
|
│ ├─► Git Clone Task
|
|
│ │
|
|
│ ├─► Kaniko Build Task
|
|
│ │ └─► Multi-stage: Go → Node → Runtime
|
|
│ │
|
|
│ └─► Push to Harbor
|
|
│ └─► the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest
|
|
│
|
|
├─────► firecrawl-playwright-build
|
|
│ │
|
|
│ ├─► Git Clone Task
|
|
│ │
|
|
│ ├─► Kaniko Build Task
|
|
│ │ └─► Node.js + Chromium install
|
|
│ │
|
|
│ └─► Push to Harbor
|
|
│ └─► .../firecrawl-playwright:latest
|
|
│
|
|
└─────► firecrawl-postgres-build
|
|
│
|
|
├─► Git Clone Task
|
|
│
|
|
├─► Kaniko Build Task
|
|
│ └─► postgres:16 + pg_cron + nuq.sql
|
|
│
|
|
└─► Push to Harbor
|
|
└─► .../firecrawl-postgres:latest
|
|
```
|
|
|
|
## Deployment Flow (ArgoCD)
|
|
|
|
```
|
|
Gitea Repository (homelab)
|
|
│
|
|
└─► cluster/applications/firecrawl/
|
|
│
|
|
▼
|
|
ArgoCD Application (sync)
|
|
│
|
|
├─► Wave 0: Namespace
|
|
│
|
|
├─► Wave 1: ConfigMap, ExternalSecret
|
|
│
|
|
├─► Wave 2: PostgreSQL StatefulSet + PVC
|
|
│ Redis Deployment
|
|
│ RabbitMQ Deployment
|
|
│
|
|
├─► Wave 3: Playwright Deployment
|
|
│ (waits for infrastructure)
|
|
│
|
|
├─► Wave 4: API Deployment
|
|
│ Worker Deployments
|
|
│ (waits for all dependencies)
|
|
│
|
|
└─► Wave 5: Services, HTTPRoute
|
|
```
|
|
|
|
## Resource Distribution
|
|
|
|
```
|
|
Total Cluster Capacity: ~48 CPU / ~96GB RAM (6 nodes)
|
|
|
|
Firecrawl Allocation:
|
|
┌────────────────────────────────────┐
|
|
│ API: 2 CPU / 4-6GB │ ████████████
|
|
│ Worker: 1 CPU / 3-4GB │ ██████
|
|
│ NUQ Worker: 1 CPU / 3-4GB │ ██████
|
|
│ Playwright: 2 CPU / 4GB │ ████████████
|
|
│ PostgreSQL: 1 CPU / 2GB │ ██████
|
|
│ Redis: 0.5 CPU / 1GB │ ███
|
|
│ RabbitMQ: 0.5 CPU / 1GB │ ███
|
|
├────────────────────────────────────┤
|
|
│ TOTAL: 8 CPU / 22GB RAM │
|
|
└────────────────────────────────────┘
|
|
|
|
Percentage of cluster: ~17% CPU, ~23% RAM
|
|
Headroom available: ✅ Excellent
|
|
```
|
|
|
|
## Data Flow
|
|
|
|
### Scrape Request Example
|
|
```
|
|
1. User → POST /v1/scrape {"url": "https://example.com"}
|
|
│
|
|
2. API validates request
|
|
│
|
|
3. API creates job in PostgreSQL (nuq.queue_scrape)
|
|
│
|
|
4. API queues job in Redis
|
|
│
|
|
5. RabbitMQ notifies workers
|
|
│
|
|
6. Worker picks up job
|
|
│
|
|
7. Worker calls Playwright service
|
|
│ └─► Playwright launches Chromium
|
|
│ └─► Renders page (handles JS)
|
|
│ └─► Returns HTML
|
|
│
|
|
8. Worker converts HTML → Markdown (Go library)
|
|
│
|
|
9. Worker stores result in PostgreSQL (returnvalue column)
|
|
│
|
|
10. Worker updates job status: completed
|
|
│
|
|
11. API returns result to user
|
|
└─► {"markdown": "...", "html": "...", "metadata": {...}}
|
|
```
|
|
|
|
---
|
|
|
|
## Network Policies (Future Enhancement)
|
|
|
|
```
|
|
firecrawl namespace:
|
|
│
|
|
├─► Ingress Rules:
|
|
│ ├─ Allow: Gateway API → API Service (port 3002)
|
|
│ └─ Deny: All other external traffic
|
|
│
|
|
├─► Egress Rules:
|
|
│ ├─ Allow: API → Playwright (port 3000)
|
|
│ ├─ Allow: API → Redis (port 6379)
|
|
│ ├─ Allow: API → PostgreSQL (port 5432)
|
|
│ ├─ Allow: API → RabbitMQ (port 5672)
|
|
│ ├─ Allow: All → Internet (for web scraping)
|
|
│ └─ Deny: All other cluster traffic
|
|
│
|
|
└─► Inter-Pod Rules:
|
|
├─ Allow: API → All infrastructure services
|
|
├─ Allow: Workers → All infrastructure services
|
|
├─ Deny: PostgreSQL → Internet (security)
|
|
└─ Deny: Redis → Internet (security)
|
|
```
|
|
|
|
---
|
|
|
|
## Monitoring & Observability (Future Enhancement)
|
|
|
|
```
|
|
Prometheus Metrics:
|
|
│
|
|
├─► API Metrics (port 3002/metrics)
|
|
│ ├─ Request rate
|
|
│ ├─ Response times
|
|
│ ├─ Job queue depth
|
|
│ └─ Error rates
|
|
│
|
|
├─► Worker Metrics (port 3005/metrics)
|
|
│ ├─ Jobs processed
|
|
│ ├─ Processing times
|
|
│ └─ Success/failure rates
|
|
│
|
|
├─► PostgreSQL Metrics
|
|
│ ├─ Connection pool usage
|
|
│ ├─ Query performance
|
|
│ └─ Table sizes
|
|
│
|
|
└─► Playwright Metrics
|
|
├─ Browser pool usage
|
|
├─ Page load times
|
|
└─ Chromium memory usage
|
|
|
|
Grafana Dashboards:
|
|
├─ Firecrawl Overview
|
|
├─ Job Processing Metrics
|
|
├─ Service Health
|
|
└─ Resource Utilization
|
|
```
|
|
|
|
---
|
|
|
|
**Diagram Version:** 1.0
|
|
**Last Updated:** June 6, 2026
|
|
**Created By:** Rocket Raccoon (CI/CD Specialist)
|