diff --git a/cluster/applications/firecrawl/REFACTOR-SUMMARY.md b/cluster/applications/firecrawl/REFACTOR-SUMMARY.md new file mode 100644 index 0000000..3416294 --- /dev/null +++ b/cluster/applications/firecrawl/REFACTOR-SUMMARY.md @@ -0,0 +1,267 @@ +# Firecrawl Production Refactoring - Summary + +## Mission Status: ✅ COMPLETE + +All critical issues identified by Ryan have been resolved. The Firecrawl deployment is now production-ready. + +--- + +## Critical Fixes Implemented + +### 1. ✅ Persistent Storage Added +**Problem**: Redis and PostgreSQL used `emptyDir` volumes - all data lost on pod restart. + +**Solution**: PersistentVolumeClaims for all stateful services +- **PostgreSQL**: 20Gi PVC on `nfs-emporium` storage class + - Location: `/volume1/fastpass/firecrawl/firecrawl-postgres-pvc/` + - Stores: Database state and queue management + +- **Redis**: 10Gi PVC on `nfs-emporium` storage class + - Location: `/volume1/fastpass/firecrawl/firecrawl-redis-pvc/` + - Stores: Cache and job queue data + - Config: AOF persistence enabled with snapshots + +- **RabbitMQ**: 5Gi PVC on `nfs-emporium` storage class + - Location: `/volume1/fastpass/firecrawl/firecrawl-rabbitmq-pvc/` + - Stores: Message queue state + +### 2. ✅ Image Versions Pinned +**Problem**: All images used `latest` tags - unpredictable updates, hard to rollback. + +**Solution**: Explicit version tags for all images +| Component | Repository | Tag | +|-----------|-----------|-----| +| Firecrawl API | `ghcr.io/mendableai/firecrawl` | `v1.0.0` | +| Firecrawl Worker | `ghcr.io/mendableai/firecrawl` | `v1.0.0` | +| Playwright Service | `ghcr.io/mendableai/firecrawl/playwright-service` | `v1.0.0` | +| PostgreSQL | `ghcr.io/mendableai/firecrawl/nuq-postgres` | `v1.0.0` | +| Redis | `redis` | `7.4.1-alpine` | +| RabbitMQ | `rabbitmq` | `3.13.7-management-alpine` | + +### 3. ✅ Converted to Helm Chart +**Problem**: Raw YAML manifests - hard to maintain, no templating. + +**Solution**: Production-ready Helm chart structure +``` +chart/ +├── Chart.yaml # Chart metadata +├── values.yaml # Configuration values (200+ lines) +├── README.md # Complete documentation +└── templates/ + ├── _helpers.tpl # Template helpers + ├── namespace.yaml # Namespace definition + ├── configmap.yaml # Configuration + ├── pvc.yaml # PersistentVolumeClaims + ├── postgres.yaml # PostgreSQL deployment + service + ├── redis.yaml # Redis deployment + service + ├── rabbitmq.yaml # RabbitMQ deployment + service + ├── playwright.yaml # Playwright deployment + service + ├── api.yaml # API deployment + service + ├── worker.yaml # Worker deployment + └── httproute.yaml # Gateway/HTTPRoute configs +``` + +--- + +## Research Findings + +### Firecrawl Official Sources +- **GitHub**: https://github.com/mendableai/firecrawl +- **Current Version**: 1.0.0 (from package.json) +- **Official Helm Chart**: Found at `/examples/kubernetes/firecrawl-helm/` + - Used as reference for structure + - Adapted for mk-labs infrastructure + +### Storage Configuration +- **Storage Class**: `nfs-emporium` (existing in cluster) +- **Provisioner**: `nfs.csi.k8s.io` +- **Backend**: Synology DS1621+ (emporium) - 10.1.71.9 +- **NFS Export**: `/volume1/fastpass` +- **Reclaim Policy**: Delete +- **Binding Mode**: Immediate + +### Image Version Research +- Checked Firecrawl repository for version tags +- Used official docker-compose.yaml as reference +- Selected stable Alpine-based images for Redis/RabbitMQ +- Pinned to Firecrawl v1.0.0 based on package.json + +### Patterns from Other Applications +Reviewed monitoring stack for PVC patterns: +- Prometheus uses volumeClaimTemplate in StatefulSet +- Alertmanager uses similar pattern +- Applied same storage class (`nfs-emporium`) +- Used comparable sizing for workload types + +--- + +## Files Created + +### Helm Chart Structure +1. **chart/Chart.yaml** - Chart metadata (v1.0.0) +2. **chart/values.yaml** - Complete configuration (6KB) +3. **chart/README.md** - Comprehensive documentation (8KB) +4. **chart/templates/_helpers.tpl** - Template functions +5. **chart/templates/namespace.yaml** - Namespace with labels +6. **chart/templates/configmap.yaml** - App and Playwright config +7. **chart/templates/pvc.yaml** - All three PVCs +8. **chart/templates/postgres.yaml** - PostgreSQL deployment + service +9. **chart/templates/redis.yaml** - Redis deployment + service +10. **chart/templates/rabbitmq.yaml** - RabbitMQ deployment + service +11. **chart/templates/playwright.yaml** - Playwright deployment + service +12. **chart/templates/api.yaml** - API deployment + service +13. **chart/templates/worker.yaml** - Worker deployment +14. **chart/templates/httproute.yaml** - Gateway routing + TLS certs + +### Documentation +15. **MIGRATION.md** - Migration guide with rollback procedures +16. **README.md** - Updated with Helm chart references + +### Modified +17. **application.yaml** - Updated ArgoCD app to use Helm chart source + +### Archived +18. **old-manifests/** - All 15 original YAML files preserved + +--- + +## Deployment Architecture + +### Persistent Storage Breakdown +``` +Total Storage Allocated: 35Gi + +PostgreSQL: 20Gi (57%) ━━━━━━━━━━━━━━━━━━━━━ +Redis: 10Gi (29%) ━━━━━━━━━━━ +RabbitMQ: 5Gi (14%) ━━━━━━ +``` + +### Resource Allocation +``` +Component CPU Request Memory Request +------------------------------------------------ +API 2000m 4Gi +Worker 1000m 3Gi +Playwright 1000m 2Gi +PostgreSQL 250m 512Mi +Redis 100m 256Mi +RabbitMQ 250m 512Mi +------------------------------------------------ +TOTAL 4600m 10.25Gi +``` + +--- + +## Verification Commands + +```bash +# Check ArgoCD sync status +kubectl -n argocd get application firecrawl + +# Verify all PVCs are bound +kubectl -n firecrawl get pvc + +# Check all pods are running +kubectl -n firecrawl get pods + +# Verify image versions (no 'latest' tags) +kubectl -n firecrawl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}' + +# Test API health +curl https://spaceship-earth.local.mk-labs.cloud/v0/health/liveness + +# Check storage usage +kubectl -n firecrawl exec deployment/firecrawl-postgres -- df -h /var/lib/postgresql/data +kubectl -n firecrawl exec deployment/firecrawl-redis -- df -h /data +``` + +--- + +## Git Commit Details + +**Commit**: 6bcb6fa +**Branch**: main +**Status**: ✅ Pushed to origin + +**Changes**: +- 31 files changed +- 1354 insertions (+) +- 7 deletions (-) +- 15 files moved to old-manifests/ +- 16 new files created + +--- + +## ArgoCD Sync Behavior + +When ArgoCD detects these changes: + +1. **Prune Phase**: Old resources deleted + - Old deployments (without PVCs) + - Old services + - Old configmaps + +2. **Create Phase**: New resources created + - Namespace (with updated labels) + - PVCs provisioned via NFS CSI + - New deployments (with PVC mounts) + - New services (with Helm naming) + +3. **Sync Phase**: HTTPRoutes updated + - TLS certificates renewed if needed + - Gateway routes point to new service names + +**Data Impact**: ⚠️ Old PostgreSQL/Redis data will be lost (was ephemeral anyway). Fresh start with persistent storage. + +--- + +## Production Readiness Checklist + +✅ **Persistent Storage**: All stateful services have PVCs +✅ **Version Pinning**: No `latest` tags anywhere +✅ **Configuration Management**: Helm chart with proper values +✅ **Resource Limits**: All pods have requests and limits +✅ **Health Checks**: Liveness and readiness probes configured +✅ **Documentation**: Complete README and migration guide +✅ **Backup Strategy**: NFS backend supports Synology snapshots +✅ **Monitoring Ready**: Labels for Prometheus ServiceMonitor +✅ **Network Policy Ready**: Proper component labels +✅ **GitOps**: Fully managed by ArgoCD + +--- + +## Next Steps (Optional Enhancements) + +1. **Monitoring**: Add Prometheus ServiceMonitor and dashboards +2. **Alerting**: Configure alerts for storage, memory, and API errors +3. **Secrets Management**: Integrate with Vault or Sealed Secrets +4. **High Availability**: Scale API replicas, add PodDisruptionBudgets +5. **Network Policies**: Restrict inter-pod communication +6. **Resource Quotas**: Add namespace quotas and limit ranges +7. **Autoscaling**: Implement HPA for API and Worker +8. **Additional Workers**: Deploy extract and nuq workers + +--- + +## References + +- **Firecrawl Docs**: https://docs.firecrawl.dev +- **Firecrawl GitHub**: https://github.com/mendableai/firecrawl +- **Official Helm Chart**: /examples/kubernetes/firecrawl-helm/ +- **Monitoring Stack**: cluster/applications/monitoring/ (for PVC patterns) +- **Storage Class**: cluster/platform/nfs-csi/storageclass.yaml +- **Gateway Config**: cluster/platform/gateway/ + +--- + +## Contact + +For issues or questions about this deployment: +- Check ArgoCD application status +- Review pod logs: `kubectl -n firecrawl logs -l app.kubernetes.io/name=firecrawl` +- Verify storage: `kubectl -n firecrawl get pvc` +- Consult chart README: `cluster/applications/firecrawl/chart/README.md` + +--- + +**Mission Accomplished**: Firecrawl is now production-ready with persistent storage, pinned versions, and proper Helm chart management. Ryan's concerns have been fully addressed. 🚀