homelab/cluster/applications/firecrawl

Firecrawl (Spaceship Earth)

Web scraping and search service for JARVIS's web search capabilities.

Service Names

• Primary (EPCOT): spaceship-earth.local.mk-labs.cloud
• Secondary: firecrawl.local.mk-labs.cloud

Architecture

Components

1. Firecrawl API - Main REST API service for web scraping

   • Port: 3002
   • Health checks: /v0/health/liveness, /v0/health/readiness
   • Resources: 4Gi memory, 2 CPU cores (6Gi limit)

2. Firecrawl Worker - Background job processor

   • Processes crawl jobs from queue
   • Resources: 3Gi memory, 1 CPU core (4Gi limit)

3. Playwright Service - Browser automation

   • Port: 3000
   • Headless Chrome/Firefox for JavaScript rendering
   • Resources: 2Gi memory, 1 CPU core (4Gi limit)

4. Redis - Cache and job queue

   • Port: 6379
   • Resources: 256Mi memory, 100m CPU (512Mi limit)

5. PostgreSQL (nuq-postgres) - State management

   • Port: 5432
   • Custom image with queue extensions
   • Resources: 512Mi memory, 250m CPU (1Gi limit)
   • Note: Currently using emptyDir (ephemeral). Add PVC for persistence if needed.

6. RabbitMQ - Message queue for distributed processing

   • AMQP Port: 5672
   • Management UI: 15672
   • Resources: 512Mi memory, 250m CPU (1Gi limit)

Configuration

Key environment variables in configmap.yaml:

• USE_DB_AUTHENTICATION=false - Simplified auth for internal use
• NUM_WORKERS_PER_QUEUE=8 - Worker concurrency
• CRAWL_CONCURRENT_REQUESTS=10 - Parallel crawl requests
• MAX_CONCURRENT_JOBS=5 - Maximum simultaneous jobs

Access

• HTTPS (Gateway API): https://spaceship-earth.local.mk-labs.cloud
• HTTPS (Gateway API): https://firecrawl.local.mk-labs.cloud
• Internal: http://firecrawl-api.firecrawl.svc.cluster.local:3002

DNS

ExternalDNS automatically configures DNS records pointing to the Gateway load balancer (10.1.71.90).

TLS

Certificates are automatically provisioned via cert-manager using the letsencrypt-prod ClusterIssuer.

Deployment

Managed by ArgoCD (wave 20):

# Check application status
kubectl -n argocd get application firecrawl

# View pods
kubectl -n firecrawl get pods

# Check API logs
kubectl -n firecrawl logs -l app=firecrawl-api -f

# Check worker logs
kubectl -n firecrawl logs -l app=firecrawl-worker -f

API Usage

Example: Scrape a webpage

curl -X POST https://spaceship-earth.local.mk-labs.cloud/v0/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Troubleshooting

Pods not starting

Check resource availability:

kubectl -n firecrawl describe pod <pod-name>

PostgreSQL data persistence

Currently using emptyDir for simplicity. To add persistence:

1. Create a PVC in postgres-pvc.yaml
2. Update postgres-deployment.yaml to use the PVC
3. Commit and push

Gateway not routing

Verify ReferenceGrant allows firecrawl namespace:

kubectl -n gateway get referencegrant allow-httproutes -o yaml

Future Enhancements

• Add persistent volume for PostgreSQL
• Add extract worker deployment (data extraction)
• Add nuq worker deployment (advanced queue processing)
• Configure monitoring/metrics
• Add resource quotas and limits
• Integrate with JARVIS via API key authentication