Files

Hermes Agent service account f4181349f8 feat: Deploy Firecrawl (spaceship-earth) to fastpass cluster

- Add Firecrawl application with full stack:
  - Firecrawl API (main service)
  - Firecrawl Worker (background jobs)
  - Playwright Service (browser automation)
  - Redis (cache & job queue)
  - PostgreSQL (state management)
  - RabbitMQ (message queue)

- Configure dual DNS names:
  - Primary: spaceship-earth.local.mk-labs.cloud (EPCOT theme)
  - Secondary: firecrawl.local.mk-labs.cloud

- Add Gateway API HTTPRoutes with TLS certificates
- Update ReferenceGrant for firecrawl namespace
- Configure ArgoCD application (wave 20)
- Set USE_DB_AUTHENTICATION=false for internal deployment

This provides JARVIS with web scraping and search capabilities.

2026-06-04 16:40:12 -05:00

3.3 KiB

Raw Blame History

Firecrawl (Spaceship Earth)

Web scraping and search service for JARVIS's web search capabilities.

Service Names

Primary (EPCOT): spaceship-earth.local.mk-labs.cloud
Secondary: firecrawl.local.mk-labs.cloud

Architecture

Components

Firecrawl API - Main REST API service for web scraping
- Port: 3002
- Health checks: /v0/health/liveness, /v0/health/readiness
- Resources: 4Gi memory, 2 CPU cores (6Gi limit)
Firecrawl Worker - Background job processor
- Processes crawl jobs from queue
- Resources: 3Gi memory, 1 CPU core (4Gi limit)
Playwright Service - Browser automation
- Port: 3000
- Headless Chrome/Firefox for JavaScript rendering
- Resources: 2Gi memory, 1 CPU core (4Gi limit)
Redis - Cache and job queue
- Port: 6379
- Resources: 256Mi memory, 100m CPU (512Mi limit)
PostgreSQL (nuq-postgres) - State management
- Port: 5432
- Custom image with queue extensions
- Resources: 512Mi memory, 250m CPU (1Gi limit)
- Note: Currently using emptyDir (ephemeral). Add PVC for persistence if needed.
RabbitMQ - Message queue for distributed processing
- AMQP Port: 5672
- Management UI: 15672
- Resources: 512Mi memory, 250m CPU (1Gi limit)

Configuration

Key environment variables in configmap.yaml:

USE_DB_AUTHENTICATION=false - Simplified auth for internal use
NUM_WORKERS_PER_QUEUE=8 - Worker concurrency
CRAWL_CONCURRENT_REQUESTS=10 - Parallel crawl requests
MAX_CONCURRENT_JOBS=5 - Maximum simultaneous jobs

Access

HTTPS (Gateway API): https://spaceship-earth.local.mk-labs.cloud
HTTPS (Gateway API): https://firecrawl.local.mk-labs.cloud
Internal: http://firecrawl-api.firecrawl.svc.cluster.local:3002

DNS

ExternalDNS automatically configures DNS records pointing to the Gateway load balancer (10.1.71.90).

TLS

Certificates are automatically provisioned via cert-manager using the letsencrypt-prod ClusterIssuer.

Deployment

Managed by ArgoCD (wave 20):

# Check application status
kubectl -n argocd get application firecrawl

# View pods
kubectl -n firecrawl get pods

# Check API logs
kubectl -n firecrawl logs -l app=firecrawl-api -f

# Check worker logs
kubectl -n firecrawl logs -l app=firecrawl-worker -f

API Usage

Example: Scrape a webpage

curl -X POST https://spaceship-earth.local.mk-labs.cloud/v0/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Troubleshooting

Pods not starting

Check resource availability:

kubectl -n firecrawl describe pod <pod-name>

PostgreSQL data persistence

Currently using emptyDir for simplicity. To add persistence:

Create a PVC in postgres-pvc.yaml
Update postgres-deployment.yaml to use the PVC
Commit and push

Gateway not routing

Verify ReferenceGrant allows firecrawl namespace:

kubectl -n gateway get referencegrant allow-httproutes -o yaml

Future Enhancements

Add persistent volume for PostgreSQL
Add extract worker deployment (data extraction)
Add nuq worker deployment (advanced queue processing)
Configure monitoring/metrics
Add resource quotas and limits
Integrate with JARVIS via API key authentication

3.3 KiB Raw Blame History