feat: Deploy Firecrawl (spaceship-earth) to fastpass cluster

- Add Firecrawl application with full stack:
  - Firecrawl API (main service)
  - Firecrawl Worker (background jobs)
  - Playwright Service (browser automation)
  - Redis (cache & job queue)
  - PostgreSQL (state management)
  - RabbitMQ (message queue)

- Configure dual DNS names:
  - Primary: spaceship-earth.local.mk-labs.cloud (EPCOT theme)
  - Secondary: firecrawl.local.mk-labs.cloud

- Add Gateway API HTTPRoutes with TLS certificates
- Update ReferenceGrant for firecrawl namespace
- Configure ArgoCD application (wave 20)
- Set USE_DB_AUTHENTICATION=false for internal deployment

This provides JARVIS with web scraping and search capabilities.
This commit is contained in:
Hermes Agent service account
2026-06-04 16:40:12 -05:00
parent 9d860367cc
commit f4181349f8
18 changed files with 725 additions and 0 deletions

View File

@@ -0,0 +1,67 @@
# ------------------------------------------------------------------------------
# Deployment: Firecrawl API
# Main API service for web scraping and crawling
# ------------------------------------------------------------------------------
apiVersion: apps/v1
kind: Deployment
metadata:
name: firecrawl-api
namespace: firecrawl
labels:
app: firecrawl-api
epcot.theme/service: spaceship-earth
spec:
replicas: 1
selector:
matchLabels:
app: firecrawl-api
template:
metadata:
labels:
app: firecrawl-api
epcot.theme/service: spaceship-earth
spec:
terminationGracePeriodSeconds: 180
containers:
- name: api
image: ghcr.io/mendableai/firecrawl:latest
command: ["node"]
args: ["--max-old-space-size=6144", "dist/src/index.js"]
envFrom:
- configMapRef:
name: firecrawl-config
env:
- name: FLY_PROCESS_GROUP
value: "app"
- name: NUQ_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
ports:
- containerPort: 3002
name: http
resources:
requests:
memory: "4Gi"
cpu: "2000m"
limits:
memory: "6Gi"
cpu: "2000m"
livenessProbe:
httpGet:
path: /v0/health/liveness
port: 3002
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /v0/health/readiness
port: 3002
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 3