Files

Hermes Agent service account 6bcb6fa93f refactor(firecrawl): Convert to production-ready Helm chart with persistent storage

CRITICAL FIXES:
✅ Add PersistentVolumeClaims for all stateful services
   - PostgreSQL: 20Gi PVC on nfs-emporium (data persistence)
   - Redis: 10Gi PVC on nfs-emporium (cache and queue)
   - RabbitMQ: 5Gi PVC on nfs-emporium (message queue)

✅ Pin all image versions (no more 'latest' tags)
   - Firecrawl API/Worker: v1.0.0
   - Playwright Service: v1.0.0
   - PostgreSQL (nuq-postgres): v1.0.0
   - Redis: 7.4.1-alpine
   - RabbitMQ: 3.13.7-management-alpine

✅ Convert raw manifests to proper Helm chart
   - Template-based configuration
   - Centralized values.yaml
   - Proper Helm helpers and labels
   - Easy configuration management

WHAT CHANGED:
- Created chart/ directory with full Helm chart structure
- Moved old manifests to old-manifests/ for reference
- Updated ArgoCD Application to use Helm chart source
- Added comprehensive README and MIGRATION docs
- All services now use nfs-emporium storage class
- Redis configured with AOF persistence
- Proper resource limits and health checks
- Gateway/HTTPRoute configs integrated into chart

DEPLOYMENT:
ArgoCD will automatically sync and apply changes.
Old ephemeral data will be lost (fresh start with persistence).

Resolves data loss issues and brings deployment to production standards.

2026-06-04 16:56:29 -05:00

chart

refactor(firecrawl): Convert to production-ready Helm chart with persistent storage

2026-06-04 16:56:29 -05:00

old-manifests

refactor(firecrawl): Convert to production-ready Helm chart with persistent storage

2026-06-04 16:56:29 -05:00

application.yaml

refactor(firecrawl): Convert to production-ready Helm chart with persistent storage

2026-06-04 16:56:29 -05:00

MIGRATION.md

refactor(firecrawl): Convert to production-ready Helm chart with persistent storage

2026-06-04 16:56:29 -05:00

README.md

feat: Deploy Firecrawl (spaceship-earth) to fastpass cluster

2026-06-04 16:40:12 -05:00

README.md

Firecrawl (Spaceship Earth)

Web scraping and search service for JARVIS's web search capabilities.

Service Names

Primary (EPCOT): spaceship-earth.local.mk-labs.cloud
Secondary: firecrawl.local.mk-labs.cloud

Architecture

Components

Firecrawl API - Main REST API service for web scraping
- Port: 3002
- Health checks: /v0/health/liveness, /v0/health/readiness
- Resources: 4Gi memory, 2 CPU cores (6Gi limit)
Firecrawl Worker - Background job processor
- Processes crawl jobs from queue
- Resources: 3Gi memory, 1 CPU core (4Gi limit)
Playwright Service - Browser automation
- Port: 3000
- Headless Chrome/Firefox for JavaScript rendering
- Resources: 2Gi memory, 1 CPU core (4Gi limit)
Redis - Cache and job queue
- Port: 6379
- Resources: 256Mi memory, 100m CPU (512Mi limit)
PostgreSQL (nuq-postgres) - State management
- Port: 5432
- Custom image with queue extensions
- Resources: 512Mi memory, 250m CPU (1Gi limit)
- Note: Currently using emptyDir (ephemeral). Add PVC for persistence if needed.
RabbitMQ - Message queue for distributed processing
- AMQP Port: 5672
- Management UI: 15672
- Resources: 512Mi memory, 250m CPU (1Gi limit)

Configuration

Key environment variables in configmap.yaml:

USE_DB_AUTHENTICATION=false - Simplified auth for internal use
NUM_WORKERS_PER_QUEUE=8 - Worker concurrency
CRAWL_CONCURRENT_REQUESTS=10 - Parallel crawl requests
MAX_CONCURRENT_JOBS=5 - Maximum simultaneous jobs

Access

HTTPS (Gateway API): https://spaceship-earth.local.mk-labs.cloud
HTTPS (Gateway API): https://firecrawl.local.mk-labs.cloud
Internal: http://firecrawl-api.firecrawl.svc.cluster.local:3002

DNS

ExternalDNS automatically configures DNS records pointing to the Gateway load balancer (10.1.71.90).

TLS

Certificates are automatically provisioned via cert-manager using the letsencrypt-prod ClusterIssuer.

Deployment

Managed by ArgoCD (wave 20):

# Check application status
kubectl -n argocd get application firecrawl

# View pods
kubectl -n firecrawl get pods

# Check API logs
kubectl -n firecrawl logs -l app=firecrawl-api -f

# Check worker logs
kubectl -n firecrawl logs -l app=firecrawl-worker -f

API Usage

Example: Scrape a webpage

curl -X POST https://spaceship-earth.local.mk-labs.cloud/v0/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Troubleshooting

Pods not starting

Check resource availability:

kubectl -n firecrawl describe pod <pod-name>

PostgreSQL data persistence

Currently using emptyDir for simplicity. To add persistence:

Create a PVC in postgres-pvc.yaml
Update postgres-deployment.yaml to use the PVC
Commit and push

Gateway not routing

Verify ReferenceGrant allows firecrawl namespace:

kubectl -n gateway get referencegrant allow-httproutes -o yaml

Future Enhancements

Add persistent volume for PostgreSQL
Add extract worker deployment (data extraction)
Add nuq worker deployment (advanced queue processing)
Configure monitoring/metrics
Add resource quotas and limits
Integrate with JARVIS via API key authentication