# Firecrawl Architecture Diagram ## High-Level Overview ``` ┌─────────────────────────────────────────────────────────────────────────┐ │ External Access │ │ spaceship-earth.local.mk-labs.cloud / firecrawl.local.mk-labs.cloud │ │ │ │ │ ▼ │ │ ┌──────────────────┐ │ │ │ Gateway API │ │ │ │ (HTTPRoute) │ │ │ │ TLS Termination │ │ │ └────────┬─────────┘ │ └─────────────────────────────────┼──────────────────────────────────────┘ │ ┌─────────────────────────────────┼──────────────────────────────────────┐ │ Firecrawl Namespace │ │ │ │ │ ▼ │ │ ┌──────────────────┐ │ │ │ API Service │ │ │ │ (ClusterIP) │ │ │ │ Port 3002 │ │ │ └────────┬─────────┘ │ │ │ │ │ ┌──────────────────┼──────────────────┐ │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ ┌─────────────────┐ ┌─────────────┐ ┌──────────────────┐ │ │ │ API Deployment │ │ Worker │ │ NUQ Worker │ │ │ │ (firecrawl-api)│ │ Deployment │ │ Deployment │ │ │ │ │ │(firecrawl- │ │ (firecrawl-api) │ │ │ │ Entrypoint: │ │ api) │ │ │ │ │ │ dist/src/ │ │ │ │ Entrypoint: │ │ │ │ index.js │ │ Entrypoint: │ │ dist/src/ │ │ │ │ │ │ dist/src/ │ │ services/worker/│ │ │ │ 4-6GB / 2 CPU │ │ services/ │ │ nuq-worker.js │ │ │ │ │ │ queue- │ │ │ │ │ │ Replicas: 1 │ │ worker.js │ │ 3-4GB / 1 CPU │ │ │ │ │ │ │ │ │ │ │ │ Health: │ │ 3-4GB/1 CPU │ │ Replicas: 1 │ │ │ │ /v0/health/* │ │ │ │ │ │ │ │ │ │ Replicas: 1 │ │ │ │ │ └────────┬────────┘ └──────┬──────┘ └────────┬─────────┘ │ │ │ │ │ │ │ └─────────┬────────┴──────────────────┘ │ │ │ │ │ ┌─────────┼──────────────┬──────────────┐ │ │ │ │ │ │ │ │ ▼ ▼ ▼ ▼ │ │ ┌────────────┐ ┌──────────┐ ┌──────────┐ ┌───────────────┐ │ │ │ Playwright │ │ Redis │ │PostgreSQL│ │ RabbitMQ │ │ │ │ Service │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ Deployment │ │Deployment│ │StatefulSet│ │ Deployment │ │ │ │ (Harbor) │ │(Upstream)│ │ (Harbor) │ │ (Upstream) │ │ │ │ │ │ │ │ │ │ │ │ │ │ Service: │ │ Service: │ │ Service: │ │ Service: │ │ │ │ 3000 │ │ 6379 │ │ 5432 │ │ 5672, 15672 │ │ │ │ │ │ │ │ │ │ │ │ │ │ 4GB/2 CPU │ │ 1GB/0.5 │ │ 2GB/1 CPU│ │ 1GB/0.5 CPU │ │ │ │ │ │ │ │ │ │ │ │ │ │ tmpfs: │ │ │ │ PVC: │ │ Healthcheck: │ │ │ │ 1GB │ │ │ │ 10GB │ │ Required │ │ │ └────────────┘ └──────────┘ └──────────┘ └───────────────┘ │ │ │ │ │ ▼ │ │ ┌──────────┐ │ │ │ NFS PVC │ │ │ │ 10GB │ │ │ │(nfs- │ │ │ │emporium) │ │ │ └──────────┘ │ │ │ │ Configuration: │ │ ┌──────────────┐ ┌────────────────┐ │ │ │ ConfigMap │ │ ExternalSecret │ │ │ │ (firecrawl- │ │ (firecrawl- │ │ │ │ config) │ │ secrets) │ │ │ │ │ │ │ │ │ │ - URLs │ │ ┌──────────┐ │ │ │ │ - Ports │ │ │1Password │ │ │ │ │ - Tuning │ │ │ Vault │ │ │ │ │ │ │ └────┬─────┘ │ │ │ └──────────────┘ │ │ │ │ │ │ ▼ │ │ │ │ - postgres- │ │ │ │ password │ │ │ │ - bull-auth- │ │ │ │ key │ │ │ └────────────────┘ │ └────────────────────────────────────────────────────────────────────────┘ ``` ## Service Communication Flow ### API Request Flow ``` External User │ ▼ Gateway API (TLS termination) │ ▼ API Service (port 3002) │ ├─────► Playwright Service (browser automation) │ └─► Returns rendered HTML/Markdown │ ├─────► Redis (queue jobs, cache results) │ ├─────► PostgreSQL (store job metadata) │ └─────► RabbitMQ (publish job events) ``` ### Background Job Processing Flow ``` API receives request │ ▼ Job queued in Redis │ ▼ RabbitMQ notifies workers │ ├─────► Worker picks up job │ └─► Processes scraping tasks │ └─────► NUQ Worker picks up database jobs └─► Processes queue from PostgreSQL ``` ### Database Queue Flow (NUQ) ``` Job created in PostgreSQL (nuq.queue_scrape table) │ ▼ NUQ Worker polls for jobs (prefetch) │ ├─► Status: queued → active │ ├─► Worker processes job │ └─► Calls Playwright or direct fetch │ └─► Status: active → completed/failed └─► Results stored in returnvalue column ``` ## Build Pipeline Flow ``` GitHub: mendableai/firecrawl │ ▼ Tekton Pipeline (innoventions namespace) │ ├─────► firecrawl-api-build │ │ │ ├─► Git Clone Task │ │ │ ├─► Kaniko Build Task │ │ └─► Multi-stage: Go → Node → Runtime │ │ │ └─► Push to Harbor │ └─► the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest │ ├─────► firecrawl-playwright-build │ │ │ ├─► Git Clone Task │ │ │ ├─► Kaniko Build Task │ │ └─► Node.js + Chromium install │ │ │ └─► Push to Harbor │ └─► .../firecrawl-playwright:latest │ └─────► firecrawl-postgres-build │ ├─► Git Clone Task │ ├─► Kaniko Build Task │ └─► postgres:16 + pg_cron + nuq.sql │ └─► Push to Harbor └─► .../firecrawl-postgres:latest ``` ## Deployment Flow (ArgoCD) ``` Gitea Repository (homelab) │ └─► cluster/applications/firecrawl/ │ ▼ ArgoCD Application (sync) │ ├─► Wave 0: Namespace │ ├─► Wave 1: ConfigMap, ExternalSecret │ ├─► Wave 2: PostgreSQL StatefulSet + PVC │ Redis Deployment │ RabbitMQ Deployment │ ├─► Wave 3: Playwright Deployment │ (waits for infrastructure) │ ├─► Wave 4: API Deployment │ Worker Deployments │ (waits for all dependencies) │ └─► Wave 5: Services, HTTPRoute ``` ## Resource Distribution ``` Total Cluster Capacity: ~48 CPU / ~96GB RAM (6 nodes) Firecrawl Allocation: ┌────────────────────────────────────┐ │ API: 2 CPU / 4-6GB │ ████████████ │ Worker: 1 CPU / 3-4GB │ ██████ │ NUQ Worker: 1 CPU / 3-4GB │ ██████ │ Playwright: 2 CPU / 4GB │ ████████████ │ PostgreSQL: 1 CPU / 2GB │ ██████ │ Redis: 0.5 CPU / 1GB │ ███ │ RabbitMQ: 0.5 CPU / 1GB │ ███ ├────────────────────────────────────┤ │ TOTAL: 8 CPU / 22GB RAM │ └────────────────────────────────────┘ Percentage of cluster: ~17% CPU, ~23% RAM Headroom available: ✅ Excellent ``` ## Data Flow ### Scrape Request Example ``` 1. User → POST /v1/scrape {"url": "https://example.com"} │ 2. API validates request │ 3. API creates job in PostgreSQL (nuq.queue_scrape) │ 4. API queues job in Redis │ 5. RabbitMQ notifies workers │ 6. Worker picks up job │ 7. Worker calls Playwright service │ └─► Playwright launches Chromium │ └─► Renders page (handles JS) │ └─► Returns HTML │ 8. Worker converts HTML → Markdown (Go library) │ 9. Worker stores result in PostgreSQL (returnvalue column) │ 10. Worker updates job status: completed │ 11. API returns result to user └─► {"markdown": "...", "html": "...", "metadata": {...}} ``` --- ## Network Policies (Future Enhancement) ``` firecrawl namespace: │ ├─► Ingress Rules: │ ├─ Allow: Gateway API → API Service (port 3002) │ └─ Deny: All other external traffic │ ├─► Egress Rules: │ ├─ Allow: API → Playwright (port 3000) │ ├─ Allow: API → Redis (port 6379) │ ├─ Allow: API → PostgreSQL (port 5432) │ ├─ Allow: API → RabbitMQ (port 5672) │ ├─ Allow: All → Internet (for web scraping) │ └─ Deny: All other cluster traffic │ └─► Inter-Pod Rules: ├─ Allow: API → All infrastructure services ├─ Allow: Workers → All infrastructure services ├─ Deny: PostgreSQL → Internet (security) └─ Deny: Redis → Internet (security) ``` --- ## Monitoring & Observability (Future Enhancement) ``` Prometheus Metrics: │ ├─► API Metrics (port 3002/metrics) │ ├─ Request rate │ ├─ Response times │ ├─ Job queue depth │ └─ Error rates │ ├─► Worker Metrics (port 3005/metrics) │ ├─ Jobs processed │ ├─ Processing times │ └─ Success/failure rates │ ├─► PostgreSQL Metrics │ ├─ Connection pool usage │ ├─ Query performance │ └─ Table sizes │ └─► Playwright Metrics ├─ Browser pool usage ├─ Page load times └─ Chromium memory usage Grafana Dashboards: ├─ Firecrawl Overview ├─ Job Processing Metrics ├─ Service Health └─ Resource Utilization ``` --- **Diagram Version:** 1.0 **Last Updated:** June 6, 2026 **Created By:** Rocket Raccoon (CI/CD Specialist)