268 lines
9.0 KiB
Markdown
268 lines
9.0 KiB
Markdown
# Firecrawl Production Refactoring - Summary
|
|
|
|
## Mission Status: ✅ COMPLETE
|
|
|
|
All critical issues identified by Ryan have been resolved. The Firecrawl deployment is now production-ready.
|
|
|
|
---
|
|
|
|
## Critical Fixes Implemented
|
|
|
|
### 1. ✅ Persistent Storage Added
|
|
**Problem**: Redis and PostgreSQL used `emptyDir` volumes - all data lost on pod restart.
|
|
|
|
**Solution**: PersistentVolumeClaims for all stateful services
|
|
- **PostgreSQL**: 20Gi PVC on `nfs-emporium` storage class
|
|
- Location: `/volume1/fastpass/firecrawl/firecrawl-postgres-pvc/`
|
|
- Stores: Database state and queue management
|
|
|
|
- **Redis**: 10Gi PVC on `nfs-emporium` storage class
|
|
- Location: `/volume1/fastpass/firecrawl/firecrawl-redis-pvc/`
|
|
- Stores: Cache and job queue data
|
|
- Config: AOF persistence enabled with snapshots
|
|
|
|
- **RabbitMQ**: 5Gi PVC on `nfs-emporium` storage class
|
|
- Location: `/volume1/fastpass/firecrawl/firecrawl-rabbitmq-pvc/`
|
|
- Stores: Message queue state
|
|
|
|
### 2. ✅ Image Versions Pinned
|
|
**Problem**: All images used `latest` tags - unpredictable updates, hard to rollback.
|
|
|
|
**Solution**: Explicit version tags for all images
|
|
| Component | Repository | Tag |
|
|
|-----------|-----------|-----|
|
|
| Firecrawl API | `ghcr.io/mendableai/firecrawl` | `v1.0.0` |
|
|
| Firecrawl Worker | `ghcr.io/mendableai/firecrawl` | `v1.0.0` |
|
|
| Playwright Service | `ghcr.io/mendableai/firecrawl/playwright-service` | `v1.0.0` |
|
|
| PostgreSQL | `ghcr.io/mendableai/firecrawl/nuq-postgres` | `v1.0.0` |
|
|
| Redis | `redis` | `7.4.1-alpine` |
|
|
| RabbitMQ | `rabbitmq` | `3.13.7-management-alpine` |
|
|
|
|
### 3. ✅ Converted to Helm Chart
|
|
**Problem**: Raw YAML manifests - hard to maintain, no templating.
|
|
|
|
**Solution**: Production-ready Helm chart structure
|
|
```
|
|
chart/
|
|
├── Chart.yaml # Chart metadata
|
|
├── values.yaml # Configuration values (200+ lines)
|
|
├── README.md # Complete documentation
|
|
└── templates/
|
|
├── _helpers.tpl # Template helpers
|
|
├── namespace.yaml # Namespace definition
|
|
├── configmap.yaml # Configuration
|
|
├── pvc.yaml # PersistentVolumeClaims
|
|
├── postgres.yaml # PostgreSQL deployment + service
|
|
├── redis.yaml # Redis deployment + service
|
|
├── rabbitmq.yaml # RabbitMQ deployment + service
|
|
├── playwright.yaml # Playwright deployment + service
|
|
├── api.yaml # API deployment + service
|
|
├── worker.yaml # Worker deployment
|
|
└── httproute.yaml # Gateway/HTTPRoute configs
|
|
```
|
|
|
|
---
|
|
|
|
## Research Findings
|
|
|
|
### Firecrawl Official Sources
|
|
- **GitHub**: https://github.com/mendableai/firecrawl
|
|
- **Current Version**: 1.0.0 (from package.json)
|
|
- **Official Helm Chart**: Found at `/examples/kubernetes/firecrawl-helm/`
|
|
- Used as reference for structure
|
|
- Adapted for mk-labs infrastructure
|
|
|
|
### Storage Configuration
|
|
- **Storage Class**: `nfs-emporium` (existing in cluster)
|
|
- **Provisioner**: `nfs.csi.k8s.io`
|
|
- **Backend**: Synology DS1621+ (emporium) - 10.1.71.9
|
|
- **NFS Export**: `/volume1/fastpass`
|
|
- **Reclaim Policy**: Delete
|
|
- **Binding Mode**: Immediate
|
|
|
|
### Image Version Research
|
|
- Checked Firecrawl repository for version tags
|
|
- Used official docker-compose.yaml as reference
|
|
- Selected stable Alpine-based images for Redis/RabbitMQ
|
|
- Pinned to Firecrawl v1.0.0 based on package.json
|
|
|
|
### Patterns from Other Applications
|
|
Reviewed monitoring stack for PVC patterns:
|
|
- Prometheus uses volumeClaimTemplate in StatefulSet
|
|
- Alertmanager uses similar pattern
|
|
- Applied same storage class (`nfs-emporium`)
|
|
- Used comparable sizing for workload types
|
|
|
|
---
|
|
|
|
## Files Created
|
|
|
|
### Helm Chart Structure
|
|
1. **chart/Chart.yaml** - Chart metadata (v1.0.0)
|
|
2. **chart/values.yaml** - Complete configuration (6KB)
|
|
3. **chart/README.md** - Comprehensive documentation (8KB)
|
|
4. **chart/templates/_helpers.tpl** - Template functions
|
|
5. **chart/templates/namespace.yaml** - Namespace with labels
|
|
6. **chart/templates/configmap.yaml** - App and Playwright config
|
|
7. **chart/templates/pvc.yaml** - All three PVCs
|
|
8. **chart/templates/postgres.yaml** - PostgreSQL deployment + service
|
|
9. **chart/templates/redis.yaml** - Redis deployment + service
|
|
10. **chart/templates/rabbitmq.yaml** - RabbitMQ deployment + service
|
|
11. **chart/templates/playwright.yaml** - Playwright deployment + service
|
|
12. **chart/templates/api.yaml** - API deployment + service
|
|
13. **chart/templates/worker.yaml** - Worker deployment
|
|
14. **chart/templates/httproute.yaml** - Gateway routing + TLS certs
|
|
|
|
### Documentation
|
|
15. **MIGRATION.md** - Migration guide with rollback procedures
|
|
16. **README.md** - Updated with Helm chart references
|
|
|
|
### Modified
|
|
17. **application.yaml** - Updated ArgoCD app to use Helm chart source
|
|
|
|
### Archived
|
|
18. **old-manifests/** - All 15 original YAML files preserved
|
|
|
|
---
|
|
|
|
## Deployment Architecture
|
|
|
|
### Persistent Storage Breakdown
|
|
```
|
|
Total Storage Allocated: 35Gi
|
|
|
|
PostgreSQL: 20Gi (57%) ━━━━━━━━━━━━━━━━━━━━━
|
|
Redis: 10Gi (29%) ━━━━━━━━━━━
|
|
RabbitMQ: 5Gi (14%) ━━━━━━
|
|
```
|
|
|
|
### Resource Allocation
|
|
```
|
|
Component CPU Request Memory Request
|
|
------------------------------------------------
|
|
API 2000m 4Gi
|
|
Worker 1000m 3Gi
|
|
Playwright 1000m 2Gi
|
|
PostgreSQL 250m 512Mi
|
|
Redis 100m 256Mi
|
|
RabbitMQ 250m 512Mi
|
|
------------------------------------------------
|
|
TOTAL 4600m 10.25Gi
|
|
```
|
|
|
|
---
|
|
|
|
## Verification Commands
|
|
|
|
```bash
|
|
# Check ArgoCD sync status
|
|
kubectl -n argocd get application firecrawl
|
|
|
|
# Verify all PVCs are bound
|
|
kubectl -n firecrawl get pvc
|
|
|
|
# Check all pods are running
|
|
kubectl -n firecrawl get pods
|
|
|
|
# Verify image versions (no 'latest' tags)
|
|
kubectl -n firecrawl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}'
|
|
|
|
# Test API health
|
|
curl https://spaceship-earth.local.mk-labs.cloud/v0/health/liveness
|
|
|
|
# Check storage usage
|
|
kubectl -n firecrawl exec deployment/firecrawl-postgres -- df -h /var/lib/postgresql/data
|
|
kubectl -n firecrawl exec deployment/firecrawl-redis -- df -h /data
|
|
```
|
|
|
|
---
|
|
|
|
## Git Commit Details
|
|
|
|
**Commit**: 6bcb6fa
|
|
**Branch**: main
|
|
**Status**: ✅ Pushed to origin
|
|
|
|
**Changes**:
|
|
- 31 files changed
|
|
- 1354 insertions (+)
|
|
- 7 deletions (-)
|
|
- 15 files moved to old-manifests/
|
|
- 16 new files created
|
|
|
|
---
|
|
|
|
## ArgoCD Sync Behavior
|
|
|
|
When ArgoCD detects these changes:
|
|
|
|
1. **Prune Phase**: Old resources deleted
|
|
- Old deployments (without PVCs)
|
|
- Old services
|
|
- Old configmaps
|
|
|
|
2. **Create Phase**: New resources created
|
|
- Namespace (with updated labels)
|
|
- PVCs provisioned via NFS CSI
|
|
- New deployments (with PVC mounts)
|
|
- New services (with Helm naming)
|
|
|
|
3. **Sync Phase**: HTTPRoutes updated
|
|
- TLS certificates renewed if needed
|
|
- Gateway routes point to new service names
|
|
|
|
**Data Impact**: ⚠️ Old PostgreSQL/Redis data will be lost (was ephemeral anyway). Fresh start with persistent storage.
|
|
|
|
---
|
|
|
|
## Production Readiness Checklist
|
|
|
|
✅ **Persistent Storage**: All stateful services have PVCs
|
|
✅ **Version Pinning**: No `latest` tags anywhere
|
|
✅ **Configuration Management**: Helm chart with proper values
|
|
✅ **Resource Limits**: All pods have requests and limits
|
|
✅ **Health Checks**: Liveness and readiness probes configured
|
|
✅ **Documentation**: Complete README and migration guide
|
|
✅ **Backup Strategy**: NFS backend supports Synology snapshots
|
|
✅ **Monitoring Ready**: Labels for Prometheus ServiceMonitor
|
|
✅ **Network Policy Ready**: Proper component labels
|
|
✅ **GitOps**: Fully managed by ArgoCD
|
|
|
|
---
|
|
|
|
## Next Steps (Optional Enhancements)
|
|
|
|
1. **Monitoring**: Add Prometheus ServiceMonitor and dashboards
|
|
2. **Alerting**: Configure alerts for storage, memory, and API errors
|
|
3. **Secrets Management**: Integrate with Vault or Sealed Secrets
|
|
4. **High Availability**: Scale API replicas, add PodDisruptionBudgets
|
|
5. **Network Policies**: Restrict inter-pod communication
|
|
6. **Resource Quotas**: Add namespace quotas and limit ranges
|
|
7. **Autoscaling**: Implement HPA for API and Worker
|
|
8. **Additional Workers**: Deploy extract and nuq workers
|
|
|
|
---
|
|
|
|
## References
|
|
|
|
- **Firecrawl Docs**: https://docs.firecrawl.dev
|
|
- **Firecrawl GitHub**: https://github.com/mendableai/firecrawl
|
|
- **Official Helm Chart**: /examples/kubernetes/firecrawl-helm/
|
|
- **Monitoring Stack**: cluster/applications/monitoring/ (for PVC patterns)
|
|
- **Storage Class**: cluster/platform/nfs-csi/storageclass.yaml
|
|
- **Gateway Config**: cluster/platform/gateway/
|
|
|
|
---
|
|
|
|
## Contact
|
|
|
|
For issues or questions about this deployment:
|
|
- Check ArgoCD application status
|
|
- Review pod logs: `kubectl -n firecrawl logs -l app.kubernetes.io/name=firecrawl`
|
|
- Verify storage: `kubectl -n firecrawl get pvc`
|
|
- Consult chart README: `cluster/applications/firecrawl/chart/README.md`
|
|
|
|
---
|
|
|
|
**Mission Accomplished**: Firecrawl is now production-ready with persistent storage, pinned versions, and proper Helm chart management. Ryan's concerns have been fully addressed. 🚀
|