homelab/cluster/applications/firecrawl/chart/README.md

Firecrawl Helm Chart

Production-ready Helm chart for deploying Firecrawl web scraping service.

Overview

Firecrawl is a web scraping and search service that provides JARVIS with web context capabilities. This deployment includes all necessary components with proper persistent storage and pinned image versions.

Components

Application Services

• Firecrawl API - Main REST API service

  • Port: 3002
  • Health checks: /v0/health/liveness, /v0/health/readiness
  • Resources: 4Gi memory, 2 CPU cores

• Firecrawl Worker - Background job processor

  • Processes crawl jobs from queue
  • Resources: 3Gi memory, 1 CPU core

• Playwright Service - Browser automation

  • Port: 3000
  • Headless Chrome/Firefox for JavaScript rendering
  • Resources: 2Gi memory, 1 CPU core

Infrastructure Services

• PostgreSQL (nuq-postgres) - State management

  • Port: 5432
  • Custom image with queue extensions
  • Persistent Storage: 20Gi PVC on nfs-emporium

• Redis - Cache and job queue

  • Port: 6379
  • AOF persistence enabled
  • Persistent Storage: 10Gi PVC on nfs-emporium

• RabbitMQ - Message queue

  • AMQP Port: 5672
  • Management UI: 15672
  • Persistent Storage: 5Gi PVC on nfs-emporium

Production Features

✅ Persistent Storage

All stateful services use PersistentVolumeClaims backed by NFS storage:

• PostgreSQL: 20Gi (database state)
• Redis: 10Gi (cache and job queue)
• RabbitMQ: 5Gi (message queue data)

✅ Pinned Image Versions

All container images use specific version tags (no latest):

• Firecrawl API/Worker: ghcr.io/mendableai/firecrawl:v1.0.0
• Playwright Service: ghcr.io/mendableai/firecrawl/playwright-service:v1.0.0
• PostgreSQL: ghcr.io/mendableai/firecrawl/nuq-postgres:v1.0.0
• Redis: redis:7.4.1-alpine
• RabbitMQ: rabbitmq:3.13.7-management-alpine

✅ Helm Configuration Management

Proper Helm chart structure with:

• Configurable values via values.yaml
• Template helpers for consistent naming
• Resource requests and limits
• Health checks and probes
• Service discovery

Configuration

Key Values

# Image versions (pinned)
image:
  repository: ghcr.io/mendableai/firecrawl
  tag: "v1.0.0"

# Persistent storage
postgres.persistence:
  enabled: true
  storageClass: "nfs-emporium"
  size: 20Gi

redis.persistence:
  enabled: true
  storageClass: "nfs-emporium"
  size: 10Gi

rabbitmq.persistence:
  enabled: true
  storageClass: "nfs-emporium"
  size: 5Gi

# Performance tuning
config:
  NUM_WORKERS_PER_QUEUE: "8"
  CRAWL_CONCURRENT_REQUESTS: "10"
  MAX_CONCURRENT_JOBS: "5"

See values.yaml for all configuration options.

Deployment

ArgoCD (Recommended)

This application is managed by ArgoCD in wave 20 (applications tier):

# Check application status
kubectl -n argocd get application firecrawl

# Sync manually if needed
argocd app sync firecrawl

# View application details
argocd app get firecrawl

Manual Helm Deployment

# From repository root
cd cluster/applications/firecrawl

# Install
helm install firecrawl ./chart -n firecrawl --create-namespace

# Upgrade
helm upgrade firecrawl ./chart -n firecrawl

# Uninstall
helm uninstall firecrawl -n firecrawl

Access

External Access (via Gateway API)

• Primary: https://spaceship-earth.local.mk-labs.cloud
• Secondary: https://firecrawl.local.mk-labs.cloud

DNS automatically configured via ExternalDNS pointing to Gateway LB (10.1.71.90).

Internal Access

http://firecrawl-api.firecrawl.svc.cluster.local:3002

API Usage Example

curl -X POST https://spaceship-earth.local.mk-labs.cloud/v0/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Monitoring

View Pods

kubectl -n firecrawl get pods

Check Logs

# API logs
kubectl -n firecrawl logs -l app.kubernetes.io/component=api -f

# Worker logs
kubectl -n firecrawl logs -l app.kubernetes.io/component=worker -f

# PostgreSQL logs
kubectl -n firecrawl logs -l app.kubernetes.io/component=postgres -f

# Redis logs
kubectl -n firecrawl logs -l app.kubernetes.io/component=redis -f

Check Storage

# View PVCs
kubectl -n firecrawl get pvc

# Check PVC details
kubectl -n firecrawl describe pvc firecrawl-postgres-pvc
kubectl -n firecrawl describe pvc firecrawl-redis-pvc
kubectl -n firecrawl describe pvc firecrawl-rabbitmq-pvc

Troubleshooting

Pods Not Starting

kubectl -n firecrawl describe pod <pod-name>
kubectl -n firecrawl get events --sort-by='.lastTimestamp'

Storage Issues

# Check PVC binding
kubectl -n firecrawl get pvc

# Verify NFS storage class
kubectl get storageclass nfs-emporium -o yaml

# Check NFS CSI driver
kubectl -n kube-system get pods -l app=csi-nfs-controller

Database Connection Issues

# Test PostgreSQL connectivity from API pod
kubectl -n firecrawl exec -it deployment/firecrawl-api -- \
  sh -c 'apt-get update && apt-get install -y postgresql-client && \
  psql -h firecrawl-postgres -U postgres -d postgres -c "SELECT 1"'

# Test Redis connectivity
kubectl -n firecrawl exec -it deployment/firecrawl-api -- \
  sh -c 'apt-get update && apt-get install -y redis-tools && \
  redis-cli -h firecrawl-redis PING'

Gateway Routing Issues

# Verify ReferenceGrant
kubectl -n gateway get referencegrant allow-httproutes -o yaml

# Check HTTPRoutes
kubectl -n firecrawl get httproute

# View Gateway status
kubectl -n gateway get gateway fastpass-gateway -o yaml

Upgrading

Update Image Versions

1. Edit values.yaml and update image tags
2. Commit and push changes
3. ArgoCD will automatically sync
4. Or manually: helm upgrade firecrawl ./chart -n firecrawl

Migrate from Old Deployment

The old raw manifest deployment will be automatically replaced by ArgoCD:

1. Commit this Helm chart
2. Push to repository
3. ArgoCD detects the change and applies Helm chart
4. Old resources are pruned
5. New resources with PVCs are created

Note: Old PostgreSQL/Redis data was ephemeral and will be lost. Fresh start with persistent storage.

Storage Architecture

NFS Provisioner

• Storage Class: nfs-emporium
• Provisioner: nfs.csi.k8s.io
• NFS Server: emporium (Synology DS1621+) - 10.1.71.9
• NFS Export: /volume1/fastpass
• Reclaim Policy: Delete
• Binding Mode: Immediate

PVC Structure

Each PVC gets a unique subdirectory:

/volume1/fastpass/firecrawl/firecrawl-postgres-pvc/
/volume1/fastpass/firecrawl/firecrawl-redis-pvc/
/volume1/fastpass/firecrawl/firecrawl-rabbitmq-pvc/

Theme

Part of the EPCOT-themed namespace with primary hostname spaceship-earth representing Firecrawl's role as the information and knowledge hub for JARVIS.

Maintenance

Backup Recommendations

# PostgreSQL backup (manual)
kubectl -n firecrawl exec deployment/firecrawl-postgres -- \
  pg_dump -U postgres postgres > firecrawl-backup-$(date +%Y%m%d).sql

# NFS snapshots (via Synology)
# Use Synology Snapshot Replication for point-in-time backups

Scaling

# Scale API replicas
helm upgrade firecrawl ./chart -n firecrawl \
  --set api.replicaCount=2

# Scale workers
helm upgrade firecrawl ./chart -n firecrawl \
  --set worker.replicaCount=3

Future Enhancements

• Add Prometheus metrics and ServiceMonitor
• Implement resource quotas and LimitRanges
• Add PodDisruptionBudgets for HA
• Configure NetworkPolicies for security
• Add external secrets management (HashiCorp Vault or Sealed Secrets)
• Implement horizontal pod autoscaling
• Add extract worker deployment
• Add nuq worker deployment

Support

For issues or questions:

• Check ArgoCD application status
• Review pod logs
• Verify storage bindings
• Consult Firecrawl documentation: https://docs.firecrawl.dev