homelab/cluster/applications/firecrawl

Firecrawl - Web Scraping & Crawling Service

Project: Platform Buildout - Firecrawl Deployment
Service Name: Spaceship Earth (EPCOT themed)
DNS: spaceship-earth.local.mk-labs.cloud (primary), firecrawl.local.mk-labs.cloud (alias)
Namespace: firecrawl
Owner: Rocket Raccoon (CI/CD Specialist)
Status: 🚧 IN PROGRESS - Day 1 Complete

---

Overview

Firecrawl is a self-hosted web scraping and crawling API that converts URLs to LLM-ready content (Markdown, JSON, HTML). This deployment enables JARVIS web search capability.

Upstream: https://github.com/mendableai/firecrawl
License: AGPLv3 (open source)

Capabilities

• Search: Search the web and get full page content
• Scrape: Convert URLs to markdown, HTML, screenshots, or structured JSON
• Crawl: Scrape all URLs of a website with a single request
• Interact: Click, scroll, write, wait before extracting (JS-heavy sites)
• Map: Discover all URLs on a website

---

Architecture

Services (5 components)

Service	Image	CPU	Memory	Storage	Purpose
API	Harbor: firecrawl-api:latest	2.0	4-6GB	-	Main REST API
Worker	Harbor: firecrawl-api:latest	1.0	3-4GB	-	Background job processor
NUQ Worker	Harbor: firecrawl-api:latest	1.0	3-4GB	-	Database queue worker
Playwright	Harbor: firecrawl-playwright:latest	2.0	4GB	1GB tmpfs	Browser automation
PostgreSQL	Harbor: firecrawl-postgres:latest	1.0	2GB	10GB PVC	Data storage
Redis	Upstream: redis:alpine	0.5	1GB	-	Queue & cache
RabbitMQ	Upstream: rabbitmq:3-management	0.5	1GB	-	Message broker

Total Resources: ~8 CPU, ~22GB RAM, 11GB storage

Service Dependencies

API Service ─┬─► Redis (queue/cache)
             ├─► PostgreSQL (data storage)
             ├─► RabbitMQ (message broker) ⚠️ HEALTH CHECK REQUIRED
             └─► Playwright Service (browser automation)

Worker ──────┬─► Redis
             ├─► PostgreSQL
             └─► RabbitMQ

NUQ Worker ──┴─► PostgreSQL

Startup Order:

1. Redis, PostgreSQL, RabbitMQ (infrastructure)
2. Playwright Service
3. API, Workers (after all dependencies ready)

---

Build Strategy

Hybrid Approach: Build custom images via Tekton, use upstream for infrastructure.

Custom Builds (Tekton → Harbor)

1. firecrawl-api (Multi-stage: Go + Node.js + Rust)

   • Source: apps/api/Dockerfile
   • Registry: the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest
   • Build time: ~15 minutes (first), ~5 minutes (cached)
   • Used by: API, Worker, NUQ Worker (different entrypoints)

2. firecrawl-playwright (Node.js + Chromium)

   • Source: apps/playwright-service-ts/Dockerfile
   • Registry: the-seas.local.mk-labs.cloud/applications/firecrawl-playwright:latest
   • Build time: ~10 minutes

3. firecrawl-postgres (PostgreSQL + pg_cron + init script)

   • Source: apps/nuq-postgres/Dockerfile
   • Registry: the-seas.local.mk-labs.cloud/applications/firecrawl-postgres:latest
   • Build time: ~3 minutes
   • Note: Custom build required for pg_cron extension and nuq.sql schema

Upstream Images

• Redis: redis:alpine
• RabbitMQ: rabbitmq:3-management

---

Configuration

Required Secrets (1Password)

Managed via ExternalSecret → 1Password vault item: firecrawl

• POSTGRES_PASSWORD - Database password (CRITICAL)
• BULL_AUTH_KEY - Queue admin UI authentication (CRITICAL)
• OPENAI_API_KEY - Optional, for AI features
• TEST_API_KEY - Optional, for testing

ConfigMap (Non-sensitive)

• Service URLs (Redis, PostgreSQL, RabbitMQ, Playwright)
• Port configuration (3002 API, 3005 Worker)
• Performance tuning (worker pools, concurrency limits)
• Logging level

Environment Variables Reference

See ENVIRONMENT_VARIABLES.md for complete list.

---

Deployment Timeline

Day 1 (June 6) - Investigation & Planning ✅

• Repository analysis complete
• Architecture decisions finalized
• Environment configuration researched
• Day 2 plan created

Day 2 (June 7) - Tekton Pipelines 🚧

• Create 3 build pipelines (API, Playwright, PostgreSQL)
• Test builds and push to Harbor
• Validate image integrity

Day 3 (June 8) - Kubernetes Manifests

• Create Deployments, Services, StatefulSets
• Configure ConfigMaps and ExternalSecrets
• Set up HTTPRoute for ingress

Day 4 (June 9) - Secrets & Configuration

• Create 1Password vault item
• Configure ExternalSecret sync
• Validate configuration

Day 5 (June 10) - Deployment & Testing

• ArgoCD Application creation
• Deploy to cluster
• Service health validation
• API functionality testing

Day 6 (June 11) - JARVIS Integration

• Configure JARVIS environment variables
• Test web search functionality
• End-to-end validation
• Documentation delivery

Target Completion: June 12, 2026

---

Access & URLs

Primary Access:

• API: https://spaceship-earth.local.mk-labs.cloud
• Alias: https://firecrawl.local.mk-labs.cloud
• Queue UI: https://spaceship-earth.local.mk-labs.cloud/admin/[BULL_AUTH_KEY]/queues

Health Endpoints:

• Liveness: https://spaceship-earth.local.mk-labs.cloud/v0/health/liveness
• Readiness: https://spaceship-earth.local.mk-labs.cloud/v0/health/readiness

Internal Services (cluster-only):

• Playwright: http://playwright-service.firecrawl.svc:3000
• PostgreSQL: postgresql://nuq-postgres.firecrawl.svc:5432
• Redis: redis://redis.firecrawl.svc:6379
• RabbitMQ: amqp://rabbitmq.firecrawl.svc:5672
• RabbitMQ Mgmt: http://rabbitmq.firecrawl.svc:15672

---

API Usage Examples

Scrape a URL

curl -X POST https://spaceship-earth.local.mk-labs.cloud/v1/scrape \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com"
  }'

Search the Web

curl -X POST https://spaceship-earth.local.mk-labs.cloud/v1/search \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "kubernetes best practices"
  }'

Crawl a Website

curl -X POST https://spaceship-earth.local.mk-labs.cloud/v1/crawl \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://docs.example.com"
  }'

---

Directory Structure

cluster/applications/firecrawl/
├── README.md                      # This file
├── ENVIRONMENT_VARIABLES.md       # Complete env var reference
├── namespace.yaml                 # Namespace definition
├── configmap.yaml                 # Non-sensitive configuration
├── externalsecret.yaml            # 1Password secret sync
├── postgresql/
│   ├── statefulset.yaml          # PostgreSQL StatefulSet
│   ├── service.yaml              # PostgreSQL Service
│   └── pvc.yaml                  # Persistent Volume Claim
├── redis/
│   ├── deployment.yaml           # Redis Deployment
│   └── service.yaml              # Redis Service
├── rabbitmq/
│   ├── deployment.yaml           # RabbitMQ Deployment
│   └── service.yaml              # RabbitMQ Service
├── playwright/
│   ├── deployment.yaml           # Playwright Deployment
│   └── service.yaml              # Playwright Service
├── api/
│   ├── deployment.yaml           # API Deployment
│   └── service.yaml              # API Service
├── workers/
│   ├── worker-deployment.yaml    # Queue Worker Deployment
│   └── nuq-worker-deployment.yaml # NUQ Worker Deployment
├── ingress/
│   └── httproute.yaml            # Gateway API HTTPRoute
└── argocd/
    └── application.yaml          # ArgoCD Application manifest

---

Tekton Pipelines

Build pipelines located in cluster/tekton/pipelines/:

• firecrawl-api-build.yaml - API service build
• firecrawl-playwright-build.yaml - Playwright service build
• firecrawl-postgres-build.yaml - PostgreSQL build

Trigger Builds:

# API build
kubectl create -f cluster/tekton/pipelines/firecrawl-api-build.yaml

# Playwright build
kubectl create -f cluster/tekton/pipelines/firecrawl-playwright-build.yaml

# PostgreSQL build
kubectl create -f cluster/tekton/pipelines/firecrawl-postgres-build.yaml

Monitor Builds:

# List pipeline runs
tkn pipelinerun list -n innoventions

# Watch logs
tkn pipelinerun logs -f <pipelinerun-name> -n innoventions

---

Troubleshooting

API Pod Not Starting

Check:

1. RabbitMQ health status (API depends on healthy RabbitMQ)
2. Environment variables (ConfigMap and Secret)
3. Database connectivity (PostgreSQL)

Commands:

kubectl logs -n firecrawl deployment/api
kubectl describe pod -n firecrawl -l app=api
kubectl get externalsecret -n firecrawl

Build Failures

Check:

1. Harbor connectivity
2. Harbor credentials secret
3. Build resource limits (increase if OOM)

Commands:

tkn pipelinerun describe <name> -n innoventions
kubectl logs -n innoventions <kaniko-pod>

Database Connection Errors

Check:

1. PostgreSQL pod status
2. PVC binding
3. Init script execution

Commands:

kubectl logs -n firecrawl statefulset/nuq-postgres
kubectl exec -it -n firecrawl nuq-postgres-0 -- psql -U postgres -d postgres -c '\d nuq.queue_scrape'

---

Monitoring

Resource Usage:

kubectl top pods -n firecrawl

Service Health:

# API liveness
curl -k https://spaceship-earth.local.mk-labs.cloud/v0/health/liveness

# API readiness
curl -k https://spaceship-earth.local.mk-labs.cloud/v0/health/readiness

# Playwright health
kubectl exec -n firecrawl deployment/playwright-service -- curl localhost:3000/health

Queue Status: Navigate to: https://spaceship-earth.local.mk-labs.cloud/admin/[BULL_AUTH_KEY]/queues

---

Security Notes

1. PostgreSQL Credentials: Stored in 1Password, synced via ExternalSecret
2. Admin UI: Protected by BULL_AUTH_KEY (in URL path)
3. Database Port: NOT exposed outside cluster (ClusterIP only)
4. TLS: All external traffic encrypted via cert-manager certificates
5. RBAC: Service accounts scoped to firecrawl namespace

---

References

• Upstream Docs: https://docs.firecrawl.dev
• GitHub: https://github.com/mendableai/firecrawl
• Self-Hosting Guide: https://github.com/mendableai/firecrawl/blob/main/SELF_HOST.md
• K8s Examples: /tmp/firecrawl/examples/kubernetes/
• Mission Brief: ~/friday/inbox/agents/rocket/FIRECRAWL-DEPLOYMENT-MISSION-BRIEF.md

---

Changelog

2026-06-06 - Day 1 Complete

• Initial repository structure created
• Architecture decisions finalized
• Build strategy documented
• Environment variables researched
• Ready for Day 2 (pipeline creation)

---

Contact: Rocket Raccoon (CI/CD Specialist)
Project Manager: Pepper Potts
Cluster: fastpass (Talos Kubernetes)
Last Updated: June 6, 2026