9.0 KiB
Firecrawl Production Refactoring - Summary
Mission Status: ✅ COMPLETE
All critical issues identified by Ryan have been resolved. The Firecrawl deployment is now production-ready.
Critical Fixes Implemented
1. ✅ Persistent Storage Added
Problem: Redis and PostgreSQL used emptyDir volumes - all data lost on pod restart.
Solution: PersistentVolumeClaims for all stateful services
-
PostgreSQL: 20Gi PVC on
nfs-emporiumstorage class- Location:
/volume1/fastpass/firecrawl/firecrawl-postgres-pvc/ - Stores: Database state and queue management
- Location:
-
Redis: 10Gi PVC on
nfs-emporiumstorage class- Location:
/volume1/fastpass/firecrawl/firecrawl-redis-pvc/ - Stores: Cache and job queue data
- Config: AOF persistence enabled with snapshots
- Location:
-
RabbitMQ: 5Gi PVC on
nfs-emporiumstorage class- Location:
/volume1/fastpass/firecrawl/firecrawl-rabbitmq-pvc/ - Stores: Message queue state
- Location:
2. ✅ Image Versions Pinned
Problem: All images used latest tags - unpredictable updates, hard to rollback.
Solution: Explicit version tags for all images
| Component | Repository | Tag |
|---|---|---|
| Firecrawl API | ghcr.io/mendableai/firecrawl |
v1.0.0 |
| Firecrawl Worker | ghcr.io/mendableai/firecrawl |
v1.0.0 |
| Playwright Service | ghcr.io/mendableai/firecrawl/playwright-service |
v1.0.0 |
| PostgreSQL | ghcr.io/mendableai/firecrawl/nuq-postgres |
v1.0.0 |
| Redis | redis |
7.4.1-alpine |
| RabbitMQ | rabbitmq |
3.13.7-management-alpine |
3. ✅ Converted to Helm Chart
Problem: Raw YAML manifests - hard to maintain, no templating.
Solution: Production-ready Helm chart structure
chart/
├── Chart.yaml # Chart metadata
├── values.yaml # Configuration values (200+ lines)
├── README.md # Complete documentation
└── templates/
├── _helpers.tpl # Template helpers
├── namespace.yaml # Namespace definition
├── configmap.yaml # Configuration
├── pvc.yaml # PersistentVolumeClaims
├── postgres.yaml # PostgreSQL deployment + service
├── redis.yaml # Redis deployment + service
├── rabbitmq.yaml # RabbitMQ deployment + service
├── playwright.yaml # Playwright deployment + service
├── api.yaml # API deployment + service
├── worker.yaml # Worker deployment
└── httproute.yaml # Gateway/HTTPRoute configs
Research Findings
Firecrawl Official Sources
- GitHub: https://github.com/mendableai/firecrawl
- Current Version: 1.0.0 (from package.json)
- Official Helm Chart: Found at
/examples/kubernetes/firecrawl-helm/- Used as reference for structure
- Adapted for mk-labs infrastructure
Storage Configuration
- Storage Class:
nfs-emporium(existing in cluster) - Provisioner:
nfs.csi.k8s.io - Backend: Synology DS1621+ (emporium) - 10.1.71.9
- NFS Export:
/volume1/fastpass - Reclaim Policy: Delete
- Binding Mode: Immediate
Image Version Research
- Checked Firecrawl repository for version tags
- Used official docker-compose.yaml as reference
- Selected stable Alpine-based images for Redis/RabbitMQ
- Pinned to Firecrawl v1.0.0 based on package.json
Patterns from Other Applications
Reviewed monitoring stack for PVC patterns:
- Prometheus uses volumeClaimTemplate in StatefulSet
- Alertmanager uses similar pattern
- Applied same storage class (
nfs-emporium) - Used comparable sizing for workload types
Files Created
Helm Chart Structure
- chart/Chart.yaml - Chart metadata (v1.0.0)
- chart/values.yaml - Complete configuration (6KB)
- chart/README.md - Comprehensive documentation (8KB)
- chart/templates/_helpers.tpl - Template functions
- chart/templates/namespace.yaml - Namespace with labels
- chart/templates/configmap.yaml - App and Playwright config
- chart/templates/pvc.yaml - All three PVCs
- chart/templates/postgres.yaml - PostgreSQL deployment + service
- chart/templates/redis.yaml - Redis deployment + service
- chart/templates/rabbitmq.yaml - RabbitMQ deployment + service
- chart/templates/playwright.yaml - Playwright deployment + service
- chart/templates/api.yaml - API deployment + service
- chart/templates/worker.yaml - Worker deployment
- chart/templates/httproute.yaml - Gateway routing + TLS certs
Documentation
- MIGRATION.md - Migration guide with rollback procedures
- README.md - Updated with Helm chart references
Modified
- application.yaml - Updated ArgoCD app to use Helm chart source
Archived
- old-manifests/ - All 15 original YAML files preserved
Deployment Architecture
Persistent Storage Breakdown
Total Storage Allocated: 35Gi
PostgreSQL: 20Gi (57%) ━━━━━━━━━━━━━━━━━━━━━
Redis: 10Gi (29%) ━━━━━━━━━━━
RabbitMQ: 5Gi (14%) ━━━━━━
Resource Allocation
Component CPU Request Memory Request
------------------------------------------------
API 2000m 4Gi
Worker 1000m 3Gi
Playwright 1000m 2Gi
PostgreSQL 250m 512Mi
Redis 100m 256Mi
RabbitMQ 250m 512Mi
------------------------------------------------
TOTAL 4600m 10.25Gi
Verification Commands
# Check ArgoCD sync status
kubectl -n argocd get application firecrawl
# Verify all PVCs are bound
kubectl -n firecrawl get pvc
# Check all pods are running
kubectl -n firecrawl get pods
# Verify image versions (no 'latest' tags)
kubectl -n firecrawl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}'
# Test API health
curl https://spaceship-earth.local.mk-labs.cloud/v0/health/liveness
# Check storage usage
kubectl -n firecrawl exec deployment/firecrawl-postgres -- df -h /var/lib/postgresql/data
kubectl -n firecrawl exec deployment/firecrawl-redis -- df -h /data
Git Commit Details
Commit: 6bcb6fa
Branch: main
Status: ✅ Pushed to origin
Changes:
- 31 files changed
- 1354 insertions (+)
- 7 deletions (-)
- 15 files moved to old-manifests/
- 16 new files created
ArgoCD Sync Behavior
When ArgoCD detects these changes:
-
Prune Phase: Old resources deleted
- Old deployments (without PVCs)
- Old services
- Old configmaps
-
Create Phase: New resources created
- Namespace (with updated labels)
- PVCs provisioned via NFS CSI
- New deployments (with PVC mounts)
- New services (with Helm naming)
-
Sync Phase: HTTPRoutes updated
- TLS certificates renewed if needed
- Gateway routes point to new service names
Data Impact: ⚠️ Old PostgreSQL/Redis data will be lost (was ephemeral anyway). Fresh start with persistent storage.
Production Readiness Checklist
✅ Persistent Storage: All stateful services have PVCs
✅ Version Pinning: No latest tags anywhere
✅ Configuration Management: Helm chart with proper values
✅ Resource Limits: All pods have requests and limits
✅ Health Checks: Liveness and readiness probes configured
✅ Documentation: Complete README and migration guide
✅ Backup Strategy: NFS backend supports Synology snapshots
✅ Monitoring Ready: Labels for Prometheus ServiceMonitor
✅ Network Policy Ready: Proper component labels
✅ GitOps: Fully managed by ArgoCD
Next Steps (Optional Enhancements)
- Monitoring: Add Prometheus ServiceMonitor and dashboards
- Alerting: Configure alerts for storage, memory, and API errors
- Secrets Management: Integrate with Vault or Sealed Secrets
- High Availability: Scale API replicas, add PodDisruptionBudgets
- Network Policies: Restrict inter-pod communication
- Resource Quotas: Add namespace quotas and limit ranges
- Autoscaling: Implement HPA for API and Worker
- Additional Workers: Deploy extract and nuq workers
References
- Firecrawl Docs: https://docs.firecrawl.dev
- Firecrawl GitHub: https://github.com/mendableai/firecrawl
- Official Helm Chart: /examples/kubernetes/firecrawl-helm/
- Monitoring Stack: cluster/applications/monitoring/ (for PVC patterns)
- Storage Class: cluster/platform/nfs-csi/storageclass.yaml
- Gateway Config: cluster/platform/gateway/
Contact
For issues or questions about this deployment:
- Check ArgoCD application status
- Review pod logs:
kubectl -n firecrawl logs -l app.kubernetes.io/name=firecrawl - Verify storage:
kubectl -n firecrawl get pvc - Consult chart README:
cluster/applications/firecrawl/chart/README.md
Mission Accomplished: Firecrawl is now production-ready with persistent storage, pinned versions, and proper Helm chart management. Ryan's concerns have been fully addressed. 🚀