feat(firecrawl): Day 3 - Complete Kubernetes manifests for Firecrawl deployment

- Created PersistentVolumeClaim for PostgreSQL (10GB, nfs-emporium)
- Created ConfigMaps for API and Playwright service configuration
- Created 7 Deployment manifests:
  * firecrawl-api (2 CPU, 4-6GB RAM)
  * firecrawl-api-worker (1 CPU, 3-4GB RAM)
  * firecrawl-api-nuq-worker (1 CPU, 3-4GB RAM)
  * firecrawl-playwright (2 CPU, 4GB RAM, 1GB tmpfs)
  * nuq-postgres (1 CPU, 2GB RAM, 10GB PVC)
  * redis (0.5 CPU, 1GB RAM)
  * rabbitmq (0.5 CPU, 1GB RAM)
- Created 5 ClusterIP Services for inter-service communication
- Created HTTPRoute for external access via Gateway API
  * Primary hostname: spaceship-earth.local.mk-labs.cloud
  * Alias: firecrawl.local.mk-labs.cloud
- All manifests validated with kubectl dry-run=client

Next steps (Day 4): Configure ExternalSecrets for 1Password integration
Next steps (Day 5): Deploy to cluster and verify functionality

Total resources: 8 CPU, 22GB RAM, 10GB storage
This commit is contained in:
Hermes Agent service account
2026-06-06 17:43:22 -05:00
parent e5d24f557a
commit 6023ee25e1
17 changed files with 655 additions and 0 deletions

View File

@@ -0,0 +1,45 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: firecrawl-config
namespace: firecrawl
labels:
app.kubernetes.io/name: firecrawl
app.kubernetes.io/component: configuration
app.kubernetes.io/part-of: firecrawl
data:
# Service URLs (Kubernetes internal DNS)
REDIS_URL: "redis://redis:6379"
REDIS_RATE_LIMIT_URL: "redis://redis:6379"
POSTGRES_HOST: "nuq-postgres"
POSTGRES_PORT: "5432"
POSTGRES_USER: "postgres"
POSTGRES_DB: "postgres"
NUQ_RABBITMQ_URL: "amqp://rabbitmq:5672"
PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000/scrape"
# Server Configuration
HOST: "0.0.0.0"
PORT: "3002"
WORKER_PORT: "3005"
EXTRACT_WORKER_PORT: "3004"
USE_DB_AUTHENTICATION: "false"
ENV: "production"
# Performance Tuning
NUM_WORKERS_PER_QUEUE: "8"
CRAWL_CONCURRENT_REQUESTS: "10"
MAX_CONCURRENT_JOBS: "5"
BROWSER_POOL_SIZE: "5"
MAX_CONCURRENT_PAGES: "10"
HARNESS_STARTUP_TIMEOUT_MS: "60000"
# Resource Limits (Self-Protection)
MAX_CPU: "0.8"
MAX_RAM: "0.8"
# Logging
LOGGING_LEVEL: "info"
# Security
ALLOW_LOCAL_WEBHOOKS: "false"