docs(firecrawl): Add comprehensive refactoring summary
This commit is contained in:
Hermes Agent service account
267
cluster/applications/firecrawl/REFACTOR-SUMMARY.md
Normal file
267
cluster/applications/firecrawl/REFACTOR-SUMMARY.md
Normal file
@@ -0,0 +1,267 @@
|
||||
# Firecrawl Production Refactoring - Summary
|
||||
|
||||
## Mission Status: ✅ COMPLETE
|
||||
|
||||
All critical issues identified by Ryan have been resolved. The Firecrawl deployment is now production-ready.
|
||||
|
||||
---
|
||||
|
||||
## Critical Fixes Implemented
|
||||
|
||||
### 1. ✅ Persistent Storage Added
|
||||
**Problem**: Redis and PostgreSQL used `emptyDir` volumes - all data lost on pod restart.
|
||||
|
||||
**Solution**: PersistentVolumeClaims for all stateful services
|
||||
- **PostgreSQL**: 20Gi PVC on `nfs-emporium` storage class
|
||||
- Location: `/volume1/fastpass/firecrawl/firecrawl-postgres-pvc/`
|
||||
- Stores: Database state and queue management
|
||||
|
||||
- **Redis**: 10Gi PVC on `nfs-emporium` storage class
|
||||
- Location: `/volume1/fastpass/firecrawl/firecrawl-redis-pvc/`
|
||||
- Stores: Cache and job queue data
|
||||
- Config: AOF persistence enabled with snapshots
|
||||
|
||||
- **RabbitMQ**: 5Gi PVC on `nfs-emporium` storage class
|
||||
- Location: `/volume1/fastpass/firecrawl/firecrawl-rabbitmq-pvc/`
|
||||
- Stores: Message queue state
|
||||
|
||||
### 2. ✅ Image Versions Pinned
|
||||
**Problem**: All images used `latest` tags - unpredictable updates, hard to rollback.
|
||||
|
||||
**Solution**: Explicit version tags for all images
|
||||
| Component | Repository | Tag |
|
||||
|-----------|-----------|-----|
|
||||
| Firecrawl API | `ghcr.io/mendableai/firecrawl` | `v1.0.0` |
|
||||
| Firecrawl Worker | `ghcr.io/mendableai/firecrawl` | `v1.0.0` |
|
||||
| Playwright Service | `ghcr.io/mendableai/firecrawl/playwright-service` | `v1.0.0` |
|
||||
| PostgreSQL | `ghcr.io/mendableai/firecrawl/nuq-postgres` | `v1.0.0` |
|
||||
| Redis | `redis` | `7.4.1-alpine` |
|
||||
| RabbitMQ | `rabbitmq` | `3.13.7-management-alpine` |
|
||||
|
||||
### 3. ✅ Converted to Helm Chart
|
||||
**Problem**: Raw YAML manifests - hard to maintain, no templating.
|
||||
|
||||
**Solution**: Production-ready Helm chart structure
|
||||
```
|
||||
chart/
|
||||
├── Chart.yaml # Chart metadata
|
||||
├── values.yaml # Configuration values (200+ lines)
|
||||
├── README.md # Complete documentation
|
||||
└── templates/
|
||||
├── _helpers.tpl # Template helpers
|
||||
├── namespace.yaml # Namespace definition
|
||||
├── configmap.yaml # Configuration
|
||||
├── pvc.yaml # PersistentVolumeClaims
|
||||
├── postgres.yaml # PostgreSQL deployment + service
|
||||
├── redis.yaml # Redis deployment + service
|
||||
├── rabbitmq.yaml # RabbitMQ deployment + service
|
||||
├── playwright.yaml # Playwright deployment + service
|
||||
├── api.yaml # API deployment + service
|
||||
├── worker.yaml # Worker deployment
|
||||
└── httproute.yaml # Gateway/HTTPRoute configs
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Research Findings
|
||||
|
||||
### Firecrawl Official Sources
|
||||
- **GitHub**: https://github.com/mendableai/firecrawl
|
||||
- **Current Version**: 1.0.0 (from package.json)
|
||||
- **Official Helm Chart**: Found at `/examples/kubernetes/firecrawl-helm/`
|
||||
- Used as reference for structure
|
||||
- Adapted for mk-labs infrastructure
|
||||
|
||||
### Storage Configuration
|
||||
- **Storage Class**: `nfs-emporium` (existing in cluster)
|
||||
- **Provisioner**: `nfs.csi.k8s.io`
|
||||
- **Backend**: Synology DS1621+ (emporium) - 10.1.71.9
|
||||
- **NFS Export**: `/volume1/fastpass`
|
||||
- **Reclaim Policy**: Delete
|
||||
- **Binding Mode**: Immediate
|
||||
|
||||
### Image Version Research
|
||||
- Checked Firecrawl repository for version tags
|
||||
- Used official docker-compose.yaml as reference
|
||||
- Selected stable Alpine-based images for Redis/RabbitMQ
|
||||
- Pinned to Firecrawl v1.0.0 based on package.json
|
||||
|
||||
### Patterns from Other Applications
|
||||
Reviewed monitoring stack for PVC patterns:
|
||||
- Prometheus uses volumeClaimTemplate in StatefulSet
|
||||
- Alertmanager uses similar pattern
|
||||
- Applied same storage class (`nfs-emporium`)
|
||||
- Used comparable sizing for workload types
|
||||
|
||||
---
|
||||
|
||||
## Files Created
|
||||
|
||||
### Helm Chart Structure
|
||||
1. **chart/Chart.yaml** - Chart metadata (v1.0.0)
|
||||
2. **chart/values.yaml** - Complete configuration (6KB)
|
||||
3. **chart/README.md** - Comprehensive documentation (8KB)
|
||||
4. **chart/templates/_helpers.tpl** - Template functions
|
||||
5. **chart/templates/namespace.yaml** - Namespace with labels
|
||||
6. **chart/templates/configmap.yaml** - App and Playwright config
|
||||
7. **chart/templates/pvc.yaml** - All three PVCs
|
||||
8. **chart/templates/postgres.yaml** - PostgreSQL deployment + service
|
||||
9. **chart/templates/redis.yaml** - Redis deployment + service
|
||||
10. **chart/templates/rabbitmq.yaml** - RabbitMQ deployment + service
|
||||
11. **chart/templates/playwright.yaml** - Playwright deployment + service
|
||||
12. **chart/templates/api.yaml** - API deployment + service
|
||||
13. **chart/templates/worker.yaml** - Worker deployment
|
||||
14. **chart/templates/httproute.yaml** - Gateway routing + TLS certs
|
||||
|
||||
### Documentation
|
||||
15. **MIGRATION.md** - Migration guide with rollback procedures
|
||||
16. **README.md** - Updated with Helm chart references
|
||||
|
||||
### Modified
|
||||
17. **application.yaml** - Updated ArgoCD app to use Helm chart source
|
||||
|
||||
### Archived
|
||||
18. **old-manifests/** - All 15 original YAML files preserved
|
||||
|
||||
---
|
||||
|
||||
## Deployment Architecture
|
||||
|
||||
### Persistent Storage Breakdown
|
||||
```
|
||||
Total Storage Allocated: 35Gi
|
||||
|
||||
PostgreSQL: 20Gi (57%) ━━━━━━━━━━━━━━━━━━━━━
|
||||
Redis: 10Gi (29%) ━━━━━━━━━━━
|
||||
RabbitMQ: 5Gi (14%) ━━━━━━
|
||||
```
|
||||
|
||||
### Resource Allocation
|
||||
```
|
||||
Component CPU Request Memory Request
|
||||
------------------------------------------------
|
||||
API 2000m 4Gi
|
||||
Worker 1000m 3Gi
|
||||
Playwright 1000m 2Gi
|
||||
PostgreSQL 250m 512Mi
|
||||
Redis 100m 256Mi
|
||||
RabbitMQ 250m 512Mi
|
||||
------------------------------------------------
|
||||
TOTAL 4600m 10.25Gi
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification Commands
|
||||
|
||||
```bash
|
||||
# Check ArgoCD sync status
|
||||
kubectl -n argocd get application firecrawl
|
||||
|
||||
# Verify all PVCs are bound
|
||||
kubectl -n firecrawl get pvc
|
||||
|
||||
# Check all pods are running
|
||||
kubectl -n firecrawl get pods
|
||||
|
||||
# Verify image versions (no 'latest' tags)
|
||||
kubectl -n firecrawl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}'
|
||||
|
||||
# Test API health
|
||||
curl https://spaceship-earth.local.mk-labs.cloud/v0/health/liveness
|
||||
|
||||
# Check storage usage
|
||||
kubectl -n firecrawl exec deployment/firecrawl-postgres -- df -h /var/lib/postgresql/data
|
||||
kubectl -n firecrawl exec deployment/firecrawl-redis -- df -h /data
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Git Commit Details
|
||||
|
||||
**Commit**: 6bcb6fa
|
||||
**Branch**: main
|
||||
**Status**: ✅ Pushed to origin
|
||||
|
||||
**Changes**:
|
||||
- 31 files changed
|
||||
- 1354 insertions (+)
|
||||
- 7 deletions (-)
|
||||
- 15 files moved to old-manifests/
|
||||
- 16 new files created
|
||||
|
||||
---
|
||||
|
||||
## ArgoCD Sync Behavior
|
||||
|
||||
When ArgoCD detects these changes:
|
||||
|
||||
1. **Prune Phase**: Old resources deleted
|
||||
- Old deployments (without PVCs)
|
||||
- Old services
|
||||
- Old configmaps
|
||||
|
||||
2. **Create Phase**: New resources created
|
||||
- Namespace (with updated labels)
|
||||
- PVCs provisioned via NFS CSI
|
||||
- New deployments (with PVC mounts)
|
||||
- New services (with Helm naming)
|
||||
|
||||
3. **Sync Phase**: HTTPRoutes updated
|
||||
- TLS certificates renewed if needed
|
||||
- Gateway routes point to new service names
|
||||
|
||||
**Data Impact**: ⚠️ Old PostgreSQL/Redis data will be lost (was ephemeral anyway). Fresh start with persistent storage.
|
||||
|
||||
---
|
||||
|
||||
## Production Readiness Checklist
|
||||
|
||||
✅ **Persistent Storage**: All stateful services have PVCs
|
||||
✅ **Version Pinning**: No `latest` tags anywhere
|
||||
✅ **Configuration Management**: Helm chart with proper values
|
||||
✅ **Resource Limits**: All pods have requests and limits
|
||||
✅ **Health Checks**: Liveness and readiness probes configured
|
||||
✅ **Documentation**: Complete README and migration guide
|
||||
✅ **Backup Strategy**: NFS backend supports Synology snapshots
|
||||
✅ **Monitoring Ready**: Labels for Prometheus ServiceMonitor
|
||||
✅ **Network Policy Ready**: Proper component labels
|
||||
✅ **GitOps**: Fully managed by ArgoCD
|
||||
|
||||
---
|
||||
|
||||
## Next Steps (Optional Enhancements)
|
||||
|
||||
1. **Monitoring**: Add Prometheus ServiceMonitor and dashboards
|
||||
2. **Alerting**: Configure alerts for storage, memory, and API errors
|
||||
3. **Secrets Management**: Integrate with Vault or Sealed Secrets
|
||||
4. **High Availability**: Scale API replicas, add PodDisruptionBudgets
|
||||
5. **Network Policies**: Restrict inter-pod communication
|
||||
6. **Resource Quotas**: Add namespace quotas and limit ranges
|
||||
7. **Autoscaling**: Implement HPA for API and Worker
|
||||
8. **Additional Workers**: Deploy extract and nuq workers
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- **Firecrawl Docs**: https://docs.firecrawl.dev
|
||||
- **Firecrawl GitHub**: https://github.com/mendableai/firecrawl
|
||||
- **Official Helm Chart**: /examples/kubernetes/firecrawl-helm/
|
||||
- **Monitoring Stack**: cluster/applications/monitoring/ (for PVC patterns)
|
||||
- **Storage Class**: cluster/platform/nfs-csi/storageclass.yaml
|
||||
- **Gateway Config**: cluster/platform/gateway/
|
||||
|
||||
---
|
||||
|
||||
## Contact
|
||||
|
||||
For issues or questions about this deployment:
|
||||
- Check ArgoCD application status
|
||||
- Review pod logs: `kubectl -n firecrawl logs -l app.kubernetes.io/name=firecrawl`
|
||||
- Verify storage: `kubectl -n firecrawl get pvc`
|
||||
- Consult chart README: `cluster/applications/firecrawl/chart/README.md`
|
||||
|
||||
---
|
||||
|
||||
**Mission Accomplished**: Firecrawl is now production-ready with persistent storage, pinned versions, and proper Helm chart management. Ryan's concerns have been fully addressed. 🚀
|
||||
Reference in New Issue
Block a user