refactor(firecrawl): Convert to production-ready Helm chart with persistent storage
CRITICAL FIXES: ✅ Add PersistentVolumeClaims for all stateful services - PostgreSQL: 20Gi PVC on nfs-emporium (data persistence) - Redis: 10Gi PVC on nfs-emporium (cache and queue) - RabbitMQ: 5Gi PVC on nfs-emporium (message queue) ✅ Pin all image versions (no more 'latest' tags) - Firecrawl API/Worker: v1.0.0 - Playwright Service: v1.0.0 - PostgreSQL (nuq-postgres): v1.0.0 - Redis: 7.4.1-alpine - RabbitMQ: 3.13.7-management-alpine ✅ Convert raw manifests to proper Helm chart - Template-based configuration - Centralized values.yaml - Proper Helm helpers and labels - Easy configuration management WHAT CHANGED: - Created chart/ directory with full Helm chart structure - Moved old manifests to old-manifests/ for reference - Updated ArgoCD Application to use Helm chart source - Added comprehensive README and MIGRATION docs - All services now use nfs-emporium storage class - Redis configured with AOF persistence - Proper resource limits and health checks - Gateway/HTTPRoute configs integrated into chart DEPLOYMENT: ArgoCD will automatically sync and apply changes. Old ephemeral data will be lost (fresh start with persistence). Resolves data loss issues and brings deployment to production standards.
This commit is contained in:
Hermes Agent service account
309
cluster/applications/firecrawl/chart/README.md
Normal file
309
cluster/applications/firecrawl/chart/README.md
Normal file
@@ -0,0 +1,309 @@
|
||||
# Firecrawl Helm Chart
|
||||
|
||||
Production-ready Helm chart for deploying Firecrawl web scraping service.
|
||||
|
||||
## Overview
|
||||
|
||||
Firecrawl is a web scraping and search service that provides JARVIS with web context capabilities. This deployment includes all necessary components with proper persistent storage and pinned image versions.
|
||||
|
||||
## Components
|
||||
|
||||
### Application Services
|
||||
- **Firecrawl API** - Main REST API service
|
||||
- Port: 3002
|
||||
- Health checks: `/v0/health/liveness`, `/v0/health/readiness`
|
||||
- Resources: 4Gi memory, 2 CPU cores
|
||||
|
||||
- **Firecrawl Worker** - Background job processor
|
||||
- Processes crawl jobs from queue
|
||||
- Resources: 3Gi memory, 1 CPU core
|
||||
|
||||
- **Playwright Service** - Browser automation
|
||||
- Port: 3000
|
||||
- Headless Chrome/Firefox for JavaScript rendering
|
||||
- Resources: 2Gi memory, 1 CPU core
|
||||
|
||||
### Infrastructure Services
|
||||
- **PostgreSQL** (nuq-postgres) - State management
|
||||
- Port: 5432
|
||||
- Custom image with queue extensions
|
||||
- **Persistent Storage**: 20Gi PVC on nfs-emporium
|
||||
|
||||
- **Redis** - Cache and job queue
|
||||
- Port: 6379
|
||||
- AOF persistence enabled
|
||||
- **Persistent Storage**: 10Gi PVC on nfs-emporium
|
||||
|
||||
- **RabbitMQ** - Message queue
|
||||
- AMQP Port: 5672
|
||||
- Management UI: 15672
|
||||
- **Persistent Storage**: 5Gi PVC on nfs-emporium
|
||||
|
||||
## Production Features
|
||||
|
||||
### ✅ Persistent Storage
|
||||
All stateful services use PersistentVolumeClaims backed by NFS storage:
|
||||
- PostgreSQL: 20Gi (database state)
|
||||
- Redis: 10Gi (cache and job queue)
|
||||
- RabbitMQ: 5Gi (message queue data)
|
||||
|
||||
### ✅ Pinned Image Versions
|
||||
All container images use specific version tags (no `latest`):
|
||||
- Firecrawl API/Worker: `ghcr.io/mendableai/firecrawl:v1.0.0`
|
||||
- Playwright Service: `ghcr.io/mendableai/firecrawl/playwright-service:v1.0.0`
|
||||
- PostgreSQL: `ghcr.io/mendableai/firecrawl/nuq-postgres:v1.0.0`
|
||||
- Redis: `redis:7.4.1-alpine`
|
||||
- RabbitMQ: `rabbitmq:3.13.7-management-alpine`
|
||||
|
||||
### ✅ Helm Configuration Management
|
||||
Proper Helm chart structure with:
|
||||
- Configurable values via `values.yaml`
|
||||
- Template helpers for consistent naming
|
||||
- Resource requests and limits
|
||||
- Health checks and probes
|
||||
- Service discovery
|
||||
|
||||
## Configuration
|
||||
|
||||
### Key Values
|
||||
|
||||
```yaml
|
||||
# Image versions (pinned)
|
||||
image:
|
||||
repository: ghcr.io/mendableai/firecrawl
|
||||
tag: "v1.0.0"
|
||||
|
||||
# Persistent storage
|
||||
postgres.persistence:
|
||||
enabled: true
|
||||
storageClass: "nfs-emporium"
|
||||
size: 20Gi
|
||||
|
||||
redis.persistence:
|
||||
enabled: true
|
||||
storageClass: "nfs-emporium"
|
||||
size: 10Gi
|
||||
|
||||
rabbitmq.persistence:
|
||||
enabled: true
|
||||
storageClass: "nfs-emporium"
|
||||
size: 5Gi
|
||||
|
||||
# Performance tuning
|
||||
config:
|
||||
NUM_WORKERS_PER_QUEUE: "8"
|
||||
CRAWL_CONCURRENT_REQUESTS: "10"
|
||||
MAX_CONCURRENT_JOBS: "5"
|
||||
```
|
||||
|
||||
See `values.yaml` for all configuration options.
|
||||
|
||||
## Deployment
|
||||
|
||||
### ArgoCD (Recommended)
|
||||
This application is managed by ArgoCD in wave 20 (applications tier):
|
||||
|
||||
```bash
|
||||
# Check application status
|
||||
kubectl -n argocd get application firecrawl
|
||||
|
||||
# Sync manually if needed
|
||||
argocd app sync firecrawl
|
||||
|
||||
# View application details
|
||||
argocd app get firecrawl
|
||||
```
|
||||
|
||||
### Manual Helm Deployment
|
||||
```bash
|
||||
# From repository root
|
||||
cd cluster/applications/firecrawl
|
||||
|
||||
# Install
|
||||
helm install firecrawl ./chart -n firecrawl --create-namespace
|
||||
|
||||
# Upgrade
|
||||
helm upgrade firecrawl ./chart -n firecrawl
|
||||
|
||||
# Uninstall
|
||||
helm uninstall firecrawl -n firecrawl
|
||||
```
|
||||
|
||||
## Access
|
||||
|
||||
### External Access (via Gateway API)
|
||||
- **Primary**: https://spaceship-earth.local.mk-labs.cloud
|
||||
- **Secondary**: https://firecrawl.local.mk-labs.cloud
|
||||
|
||||
DNS automatically configured via ExternalDNS pointing to Gateway LB (10.1.71.90).
|
||||
|
||||
### Internal Access
|
||||
```
|
||||
http://firecrawl-api.firecrawl.svc.cluster.local:3002
|
||||
```
|
||||
|
||||
### API Usage Example
|
||||
```bash
|
||||
curl -X POST https://spaceship-earth.local.mk-labs.cloud/v0/scrape \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"url": "https://example.com"}'
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### View Pods
|
||||
```bash
|
||||
kubectl -n firecrawl get pods
|
||||
```
|
||||
|
||||
### Check Logs
|
||||
```bash
|
||||
# API logs
|
||||
kubectl -n firecrawl logs -l app.kubernetes.io/component=api -f
|
||||
|
||||
# Worker logs
|
||||
kubectl -n firecrawl logs -l app.kubernetes.io/component=worker -f
|
||||
|
||||
# PostgreSQL logs
|
||||
kubectl -n firecrawl logs -l app.kubernetes.io/component=postgres -f
|
||||
|
||||
# Redis logs
|
||||
kubectl -n firecrawl logs -l app.kubernetes.io/component=redis -f
|
||||
```
|
||||
|
||||
### Check Storage
|
||||
```bash
|
||||
# View PVCs
|
||||
kubectl -n firecrawl get pvc
|
||||
|
||||
# Check PVC details
|
||||
kubectl -n firecrawl describe pvc firecrawl-postgres-pvc
|
||||
kubectl -n firecrawl describe pvc firecrawl-redis-pvc
|
||||
kubectl -n firecrawl describe pvc firecrawl-rabbitmq-pvc
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Pods Not Starting
|
||||
```bash
|
||||
kubectl -n firecrawl describe pod <pod-name>
|
||||
kubectl -n firecrawl get events --sort-by='.lastTimestamp'
|
||||
```
|
||||
|
||||
### Storage Issues
|
||||
```bash
|
||||
# Check PVC binding
|
||||
kubectl -n firecrawl get pvc
|
||||
|
||||
# Verify NFS storage class
|
||||
kubectl get storageclass nfs-emporium -o yaml
|
||||
|
||||
# Check NFS CSI driver
|
||||
kubectl -n kube-system get pods -l app=csi-nfs-controller
|
||||
```
|
||||
|
||||
### Database Connection Issues
|
||||
```bash
|
||||
# Test PostgreSQL connectivity from API pod
|
||||
kubectl -n firecrawl exec -it deployment/firecrawl-api -- \
|
||||
sh -c 'apt-get update && apt-get install -y postgresql-client && \
|
||||
psql -h firecrawl-postgres -U postgres -d postgres -c "SELECT 1"'
|
||||
|
||||
# Test Redis connectivity
|
||||
kubectl -n firecrawl exec -it deployment/firecrawl-api -- \
|
||||
sh -c 'apt-get update && apt-get install -y redis-tools && \
|
||||
redis-cli -h firecrawl-redis PING'
|
||||
```
|
||||
|
||||
### Gateway Routing Issues
|
||||
```bash
|
||||
# Verify ReferenceGrant
|
||||
kubectl -n gateway get referencegrant allow-httproutes -o yaml
|
||||
|
||||
# Check HTTPRoutes
|
||||
kubectl -n firecrawl get httproute
|
||||
|
||||
# View Gateway status
|
||||
kubectl -n gateway get gateway fastpass-gateway -o yaml
|
||||
```
|
||||
|
||||
## Upgrading
|
||||
|
||||
### Update Image Versions
|
||||
1. Edit `values.yaml` and update image tags
|
||||
2. Commit and push changes
|
||||
3. ArgoCD will automatically sync
|
||||
4. Or manually: `helm upgrade firecrawl ./chart -n firecrawl`
|
||||
|
||||
### Migrate from Old Deployment
|
||||
The old raw manifest deployment will be automatically replaced by ArgoCD:
|
||||
1. Commit this Helm chart
|
||||
2. Push to repository
|
||||
3. ArgoCD detects the change and applies Helm chart
|
||||
4. Old resources are pruned
|
||||
5. New resources with PVCs are created
|
||||
|
||||
**Note**: Old PostgreSQL/Redis data was ephemeral and will be lost. Fresh start with persistent storage.
|
||||
|
||||
## Storage Architecture
|
||||
|
||||
### NFS Provisioner
|
||||
- **Storage Class**: `nfs-emporium`
|
||||
- **Provisioner**: `nfs.csi.k8s.io`
|
||||
- **NFS Server**: emporium (Synology DS1621+) - 10.1.71.9
|
||||
- **NFS Export**: `/volume1/fastpass`
|
||||
- **Reclaim Policy**: Delete
|
||||
- **Binding Mode**: Immediate
|
||||
|
||||
### PVC Structure
|
||||
Each PVC gets a unique subdirectory:
|
||||
```
|
||||
/volume1/fastpass/firecrawl/firecrawl-postgres-pvc/
|
||||
/volume1/fastpass/firecrawl/firecrawl-redis-pvc/
|
||||
/volume1/fastpass/firecrawl/firecrawl-rabbitmq-pvc/
|
||||
```
|
||||
|
||||
## Theme
|
||||
|
||||
Part of the EPCOT-themed namespace with primary hostname **spaceship-earth** representing Firecrawl's role as the information and knowledge hub for JARVIS.
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Backup Recommendations
|
||||
```bash
|
||||
# PostgreSQL backup (manual)
|
||||
kubectl -n firecrawl exec deployment/firecrawl-postgres -- \
|
||||
pg_dump -U postgres postgres > firecrawl-backup-$(date +%Y%m%d).sql
|
||||
|
||||
# NFS snapshots (via Synology)
|
||||
# Use Synology Snapshot Replication for point-in-time backups
|
||||
```
|
||||
|
||||
### Scaling
|
||||
```bash
|
||||
# Scale API replicas
|
||||
helm upgrade firecrawl ./chart -n firecrawl \
|
||||
--set api.replicaCount=2
|
||||
|
||||
# Scale workers
|
||||
helm upgrade firecrawl ./chart -n firecrawl \
|
||||
--set worker.replicaCount=3
|
||||
```
|
||||
|
||||
## Future Enhancements
|
||||
- [ ] Add Prometheus metrics and ServiceMonitor
|
||||
- [ ] Implement resource quotas and LimitRanges
|
||||
- [ ] Add PodDisruptionBudgets for HA
|
||||
- [ ] Configure NetworkPolicies for security
|
||||
- [ ] Add external secrets management (HashiCorp Vault or Sealed Secrets)
|
||||
- [ ] Implement horizontal pod autoscaling
|
||||
- [ ] Add extract worker deployment
|
||||
- [ ] Add nuq worker deployment
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
- Check ArgoCD application status
|
||||
- Review pod logs
|
||||
- Verify storage bindings
|
||||
- Consult Firecrawl documentation: https://docs.firecrawl.dev
|
||||
Reference in New Issue
Block a user