feat(jmri): headless JMRI server with Leviton layout power monitor and X11 GUI mode
- Stable udev device symlinks (/dev/jmri/nce, /dev/jmri/loconet, /dev/jmri/lcc) - jmri-monitor: polls Leviton Decora Smart switch to start/stop JMRI automatically - Quiet hours 1-10 AM (no polling) - 30s off-delay before shutdown - LCRR config cloned from Gitea (ssh://gitea.mk-labs.cloud:2221/rblundon/LCRR.git) - ~/.jmri symlinked to LCRR repo for GitOps config management - jmri-gui: X11 remote GUI access (PanelPro/DecoderPro) via ssh -X as jmri user - Stops daemon, launches GUI, restarts daemon on exit if layout still on - jmri user gets login shell + SSH key for GUI sessions - Full JRE installed (openjdk-21-jre) for AWT/X11 support
This commit is contained in:
Hermes Agent service account
337
cluster/applications/firecrawl/ARCHITECTURE.md
Normal file
337
cluster/applications/firecrawl/ARCHITECTURE.md
Normal file
@@ -0,0 +1,337 @@
|
||||
# Firecrawl Architecture Diagram
|
||||
|
||||
## High-Level Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ External Access │
|
||||
│ spaceship-earth.local.mk-labs.cloud / firecrawl.local.mk-labs.cloud │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────┐ │
|
||||
│ │ Gateway API │ │
|
||||
│ │ (HTTPRoute) │ │
|
||||
│ │ TLS Termination │ │
|
||||
│ └────────┬─────────┘ │
|
||||
└─────────────────────────────────┼──────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────────────┼──────────────────────────────────────┐
|
||||
│ Firecrawl Namespace │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────┐ │
|
||||
│ │ API Service │ │
|
||||
│ │ (ClusterIP) │ │
|
||||
│ │ Port 3002 │ │
|
||||
│ └────────┬─────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────────────┼──────────────────┐ │
|
||||
│ │ │ │ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ ┌─────────────────┐ ┌─────────────┐ ┌──────────────────┐ │
|
||||
│ │ API Deployment │ │ Worker │ │ NUQ Worker │ │
|
||||
│ │ (firecrawl-api)│ │ Deployment │ │ Deployment │ │
|
||||
│ │ │ │(firecrawl- │ │ (firecrawl-api) │ │
|
||||
│ │ Entrypoint: │ │ api) │ │ │ │
|
||||
│ │ dist/src/ │ │ │ │ Entrypoint: │ │
|
||||
│ │ index.js │ │ Entrypoint: │ │ dist/src/ │ │
|
||||
│ │ │ │ dist/src/ │ │ services/worker/│ │
|
||||
│ │ 4-6GB / 2 CPU │ │ services/ │ │ nuq-worker.js │ │
|
||||
│ │ │ │ queue- │ │ │ │
|
||||
│ │ Replicas: 1 │ │ worker.js │ │ 3-4GB / 1 CPU │ │
|
||||
│ │ │ │ │ │ │ │
|
||||
│ │ Health: │ │ 3-4GB/1 CPU │ │ Replicas: 1 │ │
|
||||
│ │ /v0/health/* │ │ │ │ │ │
|
||||
│ │ │ │ Replicas: 1 │ │ │ │
|
||||
│ └────────┬────────┘ └──────┬──────┘ └────────┬─────────┘ │
|
||||
│ │ │ │ │
|
||||
│ └─────────┬────────┴──────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌─────────┼──────────────┬──────────────┐ │
|
||||
│ │ │ │ │ │
|
||||
│ ▼ ▼ ▼ ▼ │
|
||||
│ ┌────────────┐ ┌──────────┐ ┌──────────┐ ┌───────────────┐ │
|
||||
│ │ Playwright │ │ Redis │ │PostgreSQL│ │ RabbitMQ │ │
|
||||
│ │ Service │ │ │ │ │ │ │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ Deployment │ │Deployment│ │StatefulSet│ │ Deployment │ │
|
||||
│ │ (Harbor) │ │(Upstream)│ │ (Harbor) │ │ (Upstream) │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ Service: │ │ Service: │ │ Service: │ │ Service: │ │
|
||||
│ │ 3000 │ │ 6379 │ │ 5432 │ │ 5672, 15672 │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ 4GB/2 CPU │ │ 1GB/0.5 │ │ 2GB/1 CPU│ │ 1GB/0.5 CPU │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ tmpfs: │ │ │ │ PVC: │ │ Healthcheck: │ │
|
||||
│ │ 1GB │ │ │ │ 10GB │ │ Required │ │
|
||||
│ └────────────┘ └──────────┘ └──────────┘ └───────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────┐ │
|
||||
│ │ NFS PVC │ │
|
||||
│ │ 10GB │ │
|
||||
│ │(nfs- │ │
|
||||
│ │emporium) │ │
|
||||
│ └──────────┘ │
|
||||
│ │
|
||||
│ Configuration: │
|
||||
│ ┌──────────────┐ ┌────────────────┐ │
|
||||
│ │ ConfigMap │ │ ExternalSecret │ │
|
||||
│ │ (firecrawl- │ │ (firecrawl- │ │
|
||||
│ │ config) │ │ secrets) │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ - URLs │ │ ┌──────────┐ │ │
|
||||
│ │ - Ports │ │ │1Password │ │ │
|
||||
│ │ - Tuning │ │ │ Vault │ │ │
|
||||
│ │ │ │ └────┬─────┘ │ │
|
||||
│ └──────────────┘ │ │ │ │
|
||||
│ │ ▼ │ │
|
||||
│ │ - postgres- │ │
|
||||
│ │ password │ │
|
||||
│ │ - bull-auth- │ │
|
||||
│ │ key │ │
|
||||
│ └────────────────┘ │
|
||||
└────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Service Communication Flow
|
||||
|
||||
### API Request Flow
|
||||
```
|
||||
External User
|
||||
│
|
||||
▼
|
||||
Gateway API (TLS termination)
|
||||
│
|
||||
▼
|
||||
API Service (port 3002)
|
||||
│
|
||||
├─────► Playwright Service (browser automation)
|
||||
│ └─► Returns rendered HTML/Markdown
|
||||
│
|
||||
├─────► Redis (queue jobs, cache results)
|
||||
│
|
||||
├─────► PostgreSQL (store job metadata)
|
||||
│
|
||||
└─────► RabbitMQ (publish job events)
|
||||
```
|
||||
|
||||
### Background Job Processing Flow
|
||||
```
|
||||
API receives request
|
||||
│
|
||||
▼
|
||||
Job queued in Redis
|
||||
│
|
||||
▼
|
||||
RabbitMQ notifies workers
|
||||
│
|
||||
├─────► Worker picks up job
|
||||
│ └─► Processes scraping tasks
|
||||
│
|
||||
└─────► NUQ Worker picks up database jobs
|
||||
└─► Processes queue from PostgreSQL
|
||||
```
|
||||
|
||||
### Database Queue Flow (NUQ)
|
||||
```
|
||||
Job created in PostgreSQL (nuq.queue_scrape table)
|
||||
│
|
||||
▼
|
||||
NUQ Worker polls for jobs (prefetch)
|
||||
│
|
||||
├─► Status: queued → active
|
||||
│
|
||||
├─► Worker processes job
|
||||
│ └─► Calls Playwright or direct fetch
|
||||
│
|
||||
└─► Status: active → completed/failed
|
||||
└─► Results stored in returnvalue column
|
||||
```
|
||||
|
||||
## Build Pipeline Flow
|
||||
|
||||
```
|
||||
GitHub: mendableai/firecrawl
|
||||
│
|
||||
▼
|
||||
Tekton Pipeline (innoventions namespace)
|
||||
│
|
||||
├─────► firecrawl-api-build
|
||||
│ │
|
||||
│ ├─► Git Clone Task
|
||||
│ │
|
||||
│ ├─► Kaniko Build Task
|
||||
│ │ └─► Multi-stage: Go → Node → Runtime
|
||||
│ │
|
||||
│ └─► Push to Harbor
|
||||
│ └─► the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest
|
||||
│
|
||||
├─────► firecrawl-playwright-build
|
||||
│ │
|
||||
│ ├─► Git Clone Task
|
||||
│ │
|
||||
│ ├─► Kaniko Build Task
|
||||
│ │ └─► Node.js + Chromium install
|
||||
│ │
|
||||
│ └─► Push to Harbor
|
||||
│ └─► .../firecrawl-playwright:latest
|
||||
│
|
||||
└─────► firecrawl-postgres-build
|
||||
│
|
||||
├─► Git Clone Task
|
||||
│
|
||||
├─► Kaniko Build Task
|
||||
│ └─► postgres:16 + pg_cron + nuq.sql
|
||||
│
|
||||
└─► Push to Harbor
|
||||
└─► .../firecrawl-postgres:latest
|
||||
```
|
||||
|
||||
## Deployment Flow (ArgoCD)
|
||||
|
||||
```
|
||||
Gitea Repository (homelab)
|
||||
│
|
||||
└─► cluster/applications/firecrawl/
|
||||
│
|
||||
▼
|
||||
ArgoCD Application (sync)
|
||||
│
|
||||
├─► Wave 0: Namespace
|
||||
│
|
||||
├─► Wave 1: ConfigMap, ExternalSecret
|
||||
│
|
||||
├─► Wave 2: PostgreSQL StatefulSet + PVC
|
||||
│ Redis Deployment
|
||||
│ RabbitMQ Deployment
|
||||
│
|
||||
├─► Wave 3: Playwright Deployment
|
||||
│ (waits for infrastructure)
|
||||
│
|
||||
├─► Wave 4: API Deployment
|
||||
│ Worker Deployments
|
||||
│ (waits for all dependencies)
|
||||
│
|
||||
└─► Wave 5: Services, HTTPRoute
|
||||
```
|
||||
|
||||
## Resource Distribution
|
||||
|
||||
```
|
||||
Total Cluster Capacity: ~48 CPU / ~96GB RAM (6 nodes)
|
||||
|
||||
Firecrawl Allocation:
|
||||
┌────────────────────────────────────┐
|
||||
│ API: 2 CPU / 4-6GB │ ████████████
|
||||
│ Worker: 1 CPU / 3-4GB │ ██████
|
||||
│ NUQ Worker: 1 CPU / 3-4GB │ ██████
|
||||
│ Playwright: 2 CPU / 4GB │ ████████████
|
||||
│ PostgreSQL: 1 CPU / 2GB │ ██████
|
||||
│ Redis: 0.5 CPU / 1GB │ ███
|
||||
│ RabbitMQ: 0.5 CPU / 1GB │ ███
|
||||
├────────────────────────────────────┤
|
||||
│ TOTAL: 8 CPU / 22GB RAM │
|
||||
└────────────────────────────────────┘
|
||||
|
||||
Percentage of cluster: ~17% CPU, ~23% RAM
|
||||
Headroom available: ✅ Excellent
|
||||
```
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Scrape Request Example
|
||||
```
|
||||
1. User → POST /v1/scrape {"url": "https://example.com"}
|
||||
│
|
||||
2. API validates request
|
||||
│
|
||||
3. API creates job in PostgreSQL (nuq.queue_scrape)
|
||||
│
|
||||
4. API queues job in Redis
|
||||
│
|
||||
5. RabbitMQ notifies workers
|
||||
│
|
||||
6. Worker picks up job
|
||||
│
|
||||
7. Worker calls Playwright service
|
||||
│ └─► Playwright launches Chromium
|
||||
│ └─► Renders page (handles JS)
|
||||
│ └─► Returns HTML
|
||||
│
|
||||
8. Worker converts HTML → Markdown (Go library)
|
||||
│
|
||||
9. Worker stores result in PostgreSQL (returnvalue column)
|
||||
│
|
||||
10. Worker updates job status: completed
|
||||
│
|
||||
11. API returns result to user
|
||||
└─► {"markdown": "...", "html": "...", "metadata": {...}}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Network Policies (Future Enhancement)
|
||||
|
||||
```
|
||||
firecrawl namespace:
|
||||
│
|
||||
├─► Ingress Rules:
|
||||
│ ├─ Allow: Gateway API → API Service (port 3002)
|
||||
│ └─ Deny: All other external traffic
|
||||
│
|
||||
├─► Egress Rules:
|
||||
│ ├─ Allow: API → Playwright (port 3000)
|
||||
│ ├─ Allow: API → Redis (port 6379)
|
||||
│ ├─ Allow: API → PostgreSQL (port 5432)
|
||||
│ ├─ Allow: API → RabbitMQ (port 5672)
|
||||
│ ├─ Allow: All → Internet (for web scraping)
|
||||
│ └─ Deny: All other cluster traffic
|
||||
│
|
||||
└─► Inter-Pod Rules:
|
||||
├─ Allow: API → All infrastructure services
|
||||
├─ Allow: Workers → All infrastructure services
|
||||
├─ Deny: PostgreSQL → Internet (security)
|
||||
└─ Deny: Redis → Internet (security)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Observability (Future Enhancement)
|
||||
|
||||
```
|
||||
Prometheus Metrics:
|
||||
│
|
||||
├─► API Metrics (port 3002/metrics)
|
||||
│ ├─ Request rate
|
||||
│ ├─ Response times
|
||||
│ ├─ Job queue depth
|
||||
│ └─ Error rates
|
||||
│
|
||||
├─► Worker Metrics (port 3005/metrics)
|
||||
│ ├─ Jobs processed
|
||||
│ ├─ Processing times
|
||||
│ └─ Success/failure rates
|
||||
│
|
||||
├─► PostgreSQL Metrics
|
||||
│ ├─ Connection pool usage
|
||||
│ ├─ Query performance
|
||||
│ └─ Table sizes
|
||||
│
|
||||
└─► Playwright Metrics
|
||||
├─ Browser pool usage
|
||||
├─ Page load times
|
||||
└─ Chromium memory usage
|
||||
|
||||
Grafana Dashboards:
|
||||
├─ Firecrawl Overview
|
||||
├─ Job Processing Metrics
|
||||
├─ Service Health
|
||||
└─ Resource Utilization
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Diagram Version:** 1.0
|
||||
**Last Updated:** June 6, 2026
|
||||
**Created By:** Rocket Raccoon (CI/CD Specialist)
|
||||
412
cluster/applications/firecrawl/ENVIRONMENT_VARIABLES.md
Normal file
412
cluster/applications/firecrawl/ENVIRONMENT_VARIABLES.md
Normal file
@@ -0,0 +1,412 @@
|
||||
# Firecrawl Environment Variables Reference
|
||||
|
||||
Complete reference for all environment variables used by Firecrawl services.
|
||||
|
||||
---
|
||||
|
||||
## Required Variables (CRITICAL)
|
||||
|
||||
These variables MUST be set for Firecrawl to function.
|
||||
|
||||
### Server Configuration
|
||||
```yaml
|
||||
HOST: "0.0.0.0" # Listen address
|
||||
PORT: "3002" # Main API port
|
||||
WORKER_PORT: "3005" # Worker liveness check port
|
||||
EXTRACT_WORKER_PORT: "3004" # Extract worker port
|
||||
```
|
||||
|
||||
### Database (PostgreSQL)
|
||||
```yaml
|
||||
POSTGRES_USER: "postgres" # Database username
|
||||
POSTGRES_PASSWORD: "<SECRET>" # 🔐 Database password (from 1Password)
|
||||
POSTGRES_DB: "postgres" # Database name
|
||||
POSTGRES_HOST: "nuq-postgres" # K8s service name
|
||||
POSTGRES_PORT: "5432" # PostgreSQL port
|
||||
```
|
||||
|
||||
### Redis (Queue & Cache)
|
||||
```yaml
|
||||
REDIS_URL: "redis://redis:6379"
|
||||
REDIS_RATE_LIMIT_URL: "redis://redis:6379"
|
||||
```
|
||||
|
||||
### RabbitMQ (Message Broker)
|
||||
```yaml
|
||||
NUQ_RABBITMQ_URL: "amqp://rabbitmq:5672"
|
||||
```
|
||||
|
||||
### Playwright Service
|
||||
```yaml
|
||||
PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000/scrape"
|
||||
```
|
||||
|
||||
### Authentication
|
||||
```yaml
|
||||
USE_DB_AUTHENTICATION: "false" # Set "true" for production with Supabase
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Variables (REQUIRED)
|
||||
|
||||
### Admin UI Protection
|
||||
```yaml
|
||||
BULL_AUTH_KEY: "<SECRET>" # 🔐 Queue admin UI password (from 1Password)
|
||||
# URL: /admin/{BULL_AUTH_KEY}/queues
|
||||
```
|
||||
|
||||
### API Testing
|
||||
```yaml
|
||||
TEST_API_KEY: "<SECRET>" # 🔐 Optional - API key for testing (from 1Password)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Optional Variables (Features)
|
||||
|
||||
### AI Features (JSON Format, Extract API)
|
||||
```yaml
|
||||
# OpenAI Configuration
|
||||
OPENAI_API_KEY: "<SECRET>" # 🔐 OpenAI API key (from 1Password)
|
||||
OPENAI_BASE_URL: "" # Custom OpenAI-compatible endpoint
|
||||
MODEL_NAME: "" # Override default model (e.g., gpt-4)
|
||||
MODEL_EMBEDDING_NAME: "" # Override embedding model
|
||||
|
||||
# Ollama (Alternative to OpenAI)
|
||||
OLLAMA_BASE_URL: "" # E.g., http://localhost:11434/api
|
||||
# When using Ollama, set:
|
||||
# MODEL_NAME: "deepseek-r1:7b"
|
||||
# MODEL_EMBEDDING_NAME: "nomic-embed-text"
|
||||
```
|
||||
|
||||
### Proxy Configuration
|
||||
```yaml
|
||||
PROXY_SERVER: "" # Full URL (http://0.1.2.3:1234) or IP:port
|
||||
PROXY_USERNAME: "" # 🔐 Proxy username (from 1Password)
|
||||
PROXY_PASSWORD: "" # 🔐 Proxy password (from 1Password)
|
||||
```
|
||||
|
||||
### Search API Configuration
|
||||
```yaml
|
||||
# By default, uses Google search
|
||||
# Optionally use SearXNG instead:
|
||||
SEARXNG_ENDPOINT: "" # E.g., http://your.searxng.server
|
||||
SEARXNG_ENGINES: "" # Comma-separated engine list
|
||||
SEARXNG_CATEGORIES: "" # Comma-separated categories
|
||||
```
|
||||
|
||||
### Monitoring & Logging
|
||||
```yaml
|
||||
LOGGING_LEVEL: "info" # debug, info, warn, error
|
||||
SLACK_WEBHOOK_URL: "" # 🔐 Slack webhook for alerts (from 1Password)
|
||||
```
|
||||
|
||||
### Supabase Integration (Advanced)
|
||||
```yaml
|
||||
# Note: Not currently configurable for self-hosted instances
|
||||
SUPABASE_ANON_TOKEN: "" # For DB authentication
|
||||
SUPABASE_URL: "" # Supabase project URL
|
||||
SUPABASE_SERVICE_TOKEN: "" # Supabase service role key
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### Worker Pools
|
||||
```yaml
|
||||
NUM_WORKERS_PER_QUEUE: "8" # Number of workers per queue
|
||||
CRAWL_CONCURRENT_REQUESTS: "10" # Concurrent crawl requests
|
||||
MAX_CONCURRENT_JOBS: "5" # Maximum concurrent jobs
|
||||
```
|
||||
|
||||
### Browser Pool (Playwright)
|
||||
```yaml
|
||||
BROWSER_POOL_SIZE: "5" # Browser instance pool size
|
||||
MAX_CONCURRENT_PAGES: "10" # Max concurrent pages per browser
|
||||
```
|
||||
|
||||
### Resource Limits (Self-Protection)
|
||||
```yaml
|
||||
MAX_CPU: "0.8" # 0.0-1.0, reject jobs above threshold
|
||||
MAX_RAM: "0.8" # 0.0-1.0, reject jobs above threshold
|
||||
```
|
||||
|
||||
### Timeouts
|
||||
```yaml
|
||||
HARNESS_STARTUP_TIMEOUT_MS: "60000" # 60 seconds startup timeout
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Playwright-Specific Variables
|
||||
|
||||
Set on `playwright-service` pods only:
|
||||
|
||||
```yaml
|
||||
PORT: "3000" # Playwright service port
|
||||
PROXY_SERVER: "" # Same as API proxy (if needed)
|
||||
PROXY_USERNAME: "" # Same as API proxy (if needed)
|
||||
PROXY_PASSWORD: "" # Same as API proxy (if needed)
|
||||
ALLOW_LOCAL_WEBHOOKS: "false" # Security: block local webhooks
|
||||
BLOCK_MEDIA: "" # Optional: block media resources
|
||||
MAX_CONCURRENT_PAGES: "10" # Browser concurrency (same as API)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment-Specific Variables
|
||||
|
||||
### Development
|
||||
```yaml
|
||||
ENV: "local"
|
||||
```
|
||||
|
||||
### Production
|
||||
```yaml
|
||||
ENV: "production"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## FlyIO-Specific Variables (Not Used in K8s)
|
||||
|
||||
These variables are set by FlyIO platform and not needed for Kubernetes deployment:
|
||||
|
||||
```yaml
|
||||
FLY_PROCESS_GROUP: "app" # Not used in K8s
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deprecated / Unused Variables
|
||||
|
||||
Variables found in examples but not required for self-hosted deployment:
|
||||
|
||||
```yaml
|
||||
AUTUMN_SECRET_KEY: "" # Mendable platform-specific
|
||||
SELF_HOSTED_WEBHOOK_URL: "" # Custom webhook endpoint
|
||||
LLAMAPARSE_API_KEY: "" # PDF parsing service (optional)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Variable Precedence
|
||||
|
||||
1. **ExternalSecret** (from 1Password) - Highest priority for secrets
|
||||
2. **ConfigMap** - Non-sensitive configuration
|
||||
3. **Deployment env:** - Direct environment variables (override)
|
||||
4. **Dockerfile defaults** - Lowest priority
|
||||
|
||||
---
|
||||
|
||||
## ConfigMap Example
|
||||
|
||||
Non-sensitive variables suitable for ConfigMap:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: firecrawl-config
|
||||
namespace: firecrawl
|
||||
data:
|
||||
# Service URLs
|
||||
REDIS_URL: "redis://redis:6379"
|
||||
REDIS_RATE_LIMIT_URL: "redis://redis:6379"
|
||||
POSTGRES_HOST: "nuq-postgres"
|
||||
POSTGRES_PORT: "5432"
|
||||
POSTGRES_USER: "postgres"
|
||||
POSTGRES_DB: "postgres"
|
||||
NUQ_RABBITMQ_URL: "amqp://rabbitmq:5672"
|
||||
PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000/scrape"
|
||||
|
||||
# Server Configuration
|
||||
HOST: "0.0.0.0"
|
||||
PORT: "3002"
|
||||
WORKER_PORT: "3005"
|
||||
EXTRACT_WORKER_PORT: "3004"
|
||||
USE_DB_AUTHENTICATION: "false"
|
||||
ENV: "production"
|
||||
|
||||
# Performance Tuning
|
||||
NUM_WORKERS_PER_QUEUE: "8"
|
||||
CRAWL_CONCURRENT_REQUESTS: "10"
|
||||
MAX_CONCURRENT_JOBS: "5"
|
||||
BROWSER_POOL_SIZE: "5"
|
||||
MAX_CONCURRENT_PAGES: "10"
|
||||
HARNESS_STARTUP_TIMEOUT_MS: "60000"
|
||||
|
||||
# Logging
|
||||
LOGGING_LEVEL: "info"
|
||||
|
||||
# Security
|
||||
ALLOW_LOCAL_WEBHOOKS: "false"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ExternalSecret Example
|
||||
|
||||
Sensitive variables synced from 1Password:
|
||||
|
||||
```yaml
|
||||
apiVersion: external-secrets.io/v1beta1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
name: firecrawl-secrets
|
||||
namespace: firecrawl
|
||||
spec:
|
||||
refreshInterval: 1h
|
||||
secretStoreRef:
|
||||
name: onepassword-connect
|
||||
kind: ClusterSecretStore
|
||||
target:
|
||||
name: firecrawl-secrets
|
||||
creationPolicy: Owner
|
||||
data:
|
||||
# Required secrets
|
||||
- secretKey: POSTGRES_PASSWORD
|
||||
remoteRef:
|
||||
key: firecrawl
|
||||
property: postgres-password
|
||||
|
||||
- secretKey: BULL_AUTH_KEY
|
||||
remoteRef:
|
||||
key: firecrawl
|
||||
property: bull-auth-key
|
||||
|
||||
# Optional secrets (uncomment when added to 1Password)
|
||||
# - secretKey: OPENAI_API_KEY
|
||||
# remoteRef:
|
||||
# key: firecrawl
|
||||
# property: openai-api-key
|
||||
|
||||
# - secretKey: TEST_API_KEY
|
||||
# remoteRef:
|
||||
# key: firecrawl
|
||||
# property: test-api-key
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Usage in Deployments
|
||||
|
||||
### Combining ConfigMap and Secret
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: api
|
||||
namespace: firecrawl
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: api
|
||||
image: the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest
|
||||
envFrom:
|
||||
# Load all non-sensitive variables
|
||||
- configMapRef:
|
||||
name: firecrawl-config
|
||||
# Load all secrets
|
||||
- secretRef:
|
||||
name: firecrawl-secrets
|
||||
env:
|
||||
# Override specific variables if needed
|
||||
- name: FLY_PROCESS_GROUP
|
||||
value: "app"
|
||||
```
|
||||
|
||||
### Playwright-Specific ConfigMap
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: playwright-service-config
|
||||
namespace: firecrawl
|
||||
data:
|
||||
PORT: "3000"
|
||||
ALLOW_LOCAL_WEBHOOKS: "false"
|
||||
MAX_CONCURRENT_PAGES: "10"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Checklist
|
||||
|
||||
Before deployment, ensure:
|
||||
|
||||
- [ ] `POSTGRES_PASSWORD` set in 1Password
|
||||
- [ ] `BULL_AUTH_KEY` set in 1Password (strong random value)
|
||||
- [ ] All service URLs use correct K8s service names
|
||||
- [ ] Ports match service definitions (3002, 3000, 5432, 6379, 5672)
|
||||
- [ ] `USE_DB_AUTHENTICATION` is "false" (Supabase not available)
|
||||
- [ ] `ENV` is "production" (not "local")
|
||||
- [ ] Worker pool sizes appropriate for cluster capacity
|
||||
- [ ] `ALLOW_LOCAL_WEBHOOKS` is "false" (security)
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Supabase client is not configured"
|
||||
**Expected warning** - Supabase is not available for self-hosted instances. Safe to ignore.
|
||||
|
||||
### "You're bypassing authentication"
|
||||
**Expected warning** - When `USE_DB_AUTHENTICATION=false`. Normal for self-hosted deployment.
|
||||
|
||||
### Connection refused errors
|
||||
**Check:**
|
||||
- Service names match environment variables
|
||||
- Services are running: `kubectl get svc -n firecrawl`
|
||||
- Pods are ready: `kubectl get pods -n firecrawl`
|
||||
|
||||
### Build failures
|
||||
**Check:**
|
||||
- Required build args are set
|
||||
- Kaniko has sufficient memory (4GB for API build)
|
||||
|
||||
---
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
1. **Generate strong secrets:**
|
||||
```bash
|
||||
# Generate BULL_AUTH_KEY
|
||||
openssl rand -base64 32
|
||||
|
||||
# Generate POSTGRES_PASSWORD
|
||||
openssl rand -base64 24
|
||||
```
|
||||
|
||||
2. **Never expose PostgreSQL externally:**
|
||||
- Use ClusterIP service only
|
||||
- No LoadBalancer or NodePort
|
||||
- Access via kubectl port-forward for maintenance
|
||||
|
||||
3. **Protect admin UI:**
|
||||
- Keep BULL_AUTH_KEY secret
|
||||
- Don't commit to git
|
||||
- Rotate periodically
|
||||
|
||||
4. **Use ExternalSecrets:**
|
||||
- Never put secrets in ConfigMaps
|
||||
- Never commit secrets to git
|
||||
- Store all sensitive data in 1Password
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- **Upstream .env Template:** `/tmp/firecrawl/SELF_HOST.md`
|
||||
- **Docker Compose Reference:** `/tmp/firecrawl/docker-compose.yaml`
|
||||
- **K8s Example ConfigMap:** `/tmp/firecrawl/examples/kubernetes/cluster-install/configmap.yaml`
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** June 6, 2026
|
||||
**Maintained By:** Rocket Raccoon (CI/CD Specialist)
|
||||
375
cluster/applications/firecrawl/README.md
Normal file
375
cluster/applications/firecrawl/README.md
Normal file
@@ -0,0 +1,375 @@
|
||||
# Firecrawl - Web Scraping & Crawling Service
|
||||
|
||||
**Project:** Platform Buildout - Firecrawl Deployment
|
||||
**Service Name:** Spaceship Earth (EPCOT themed)
|
||||
**DNS:** spaceship-earth.local.mk-labs.cloud (primary), firecrawl.local.mk-labs.cloud (alias)
|
||||
**Namespace:** firecrawl
|
||||
**Owner:** Rocket Raccoon (CI/CD Specialist)
|
||||
**Status:** 🚧 IN PROGRESS - Day 1 Complete
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Firecrawl is a self-hosted web scraping and crawling API that converts URLs to LLM-ready content (Markdown, JSON, HTML). This deployment enables JARVIS web search capability.
|
||||
|
||||
**Upstream:** https://github.com/mendableai/firecrawl
|
||||
**License:** AGPLv3 (open source)
|
||||
|
||||
### Capabilities
|
||||
- **Search:** Search the web and get full page content
|
||||
- **Scrape:** Convert URLs to markdown, HTML, screenshots, or structured JSON
|
||||
- **Crawl:** Scrape all URLs of a website with a single request
|
||||
- **Interact:** Click, scroll, write, wait before extracting (JS-heavy sites)
|
||||
- **Map:** Discover all URLs on a website
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### Services (5 components)
|
||||
|
||||
| Service | Image | CPU | Memory | Storage | Purpose |
|
||||
|---------|-------|-----|--------|---------|---------|
|
||||
| **API** | Harbor: firecrawl-api:latest | 2.0 | 4-6GB | - | Main REST API |
|
||||
| **Worker** | Harbor: firecrawl-api:latest | 1.0 | 3-4GB | - | Background job processor |
|
||||
| **NUQ Worker** | Harbor: firecrawl-api:latest | 1.0 | 3-4GB | - | Database queue worker |
|
||||
| **Playwright** | Harbor: firecrawl-playwright:latest | 2.0 | 4GB | 1GB tmpfs | Browser automation |
|
||||
| **PostgreSQL** | Harbor: firecrawl-postgres:latest | 1.0 | 2GB | 10GB PVC | Data storage |
|
||||
| **Redis** | Upstream: redis:alpine | 0.5 | 1GB | - | Queue & cache |
|
||||
| **RabbitMQ** | Upstream: rabbitmq:3-management | 0.5 | 1GB | - | Message broker |
|
||||
|
||||
**Total Resources:** ~8 CPU, ~22GB RAM, 11GB storage
|
||||
|
||||
### Service Dependencies
|
||||
|
||||
```
|
||||
API Service ─┬─► Redis (queue/cache)
|
||||
├─► PostgreSQL (data storage)
|
||||
├─► RabbitMQ (message broker) ⚠️ HEALTH CHECK REQUIRED
|
||||
└─► Playwright Service (browser automation)
|
||||
|
||||
Worker ──────┬─► Redis
|
||||
├─► PostgreSQL
|
||||
└─► RabbitMQ
|
||||
|
||||
NUQ Worker ──┴─► PostgreSQL
|
||||
```
|
||||
|
||||
**Startup Order:**
|
||||
1. Redis, PostgreSQL, RabbitMQ (infrastructure)
|
||||
2. Playwright Service
|
||||
3. API, Workers (after all dependencies ready)
|
||||
|
||||
---
|
||||
|
||||
## Build Strategy
|
||||
|
||||
**Hybrid Approach:** Build custom images via Tekton, use upstream for infrastructure.
|
||||
|
||||
### Custom Builds (Tekton → Harbor)
|
||||
|
||||
1. **firecrawl-api** (Multi-stage: Go + Node.js + Rust)
|
||||
- Source: `apps/api/Dockerfile`
|
||||
- Registry: `the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest`
|
||||
- Build time: ~15 minutes (first), ~5 minutes (cached)
|
||||
- Used by: API, Worker, NUQ Worker (different entrypoints)
|
||||
|
||||
2. **firecrawl-playwright** (Node.js + Chromium)
|
||||
- Source: `apps/playwright-service-ts/Dockerfile`
|
||||
- Registry: `the-seas.local.mk-labs.cloud/applications/firecrawl-playwright:latest`
|
||||
- Build time: ~10 minutes
|
||||
|
||||
3. **firecrawl-postgres** (PostgreSQL + pg_cron + init script)
|
||||
- Source: `apps/nuq-postgres/Dockerfile`
|
||||
- Registry: `the-seas.local.mk-labs.cloud/applications/firecrawl-postgres:latest`
|
||||
- Build time: ~3 minutes
|
||||
- Note: Custom build required for pg_cron extension and nuq.sql schema
|
||||
|
||||
### Upstream Images
|
||||
|
||||
- **Redis:** `redis:alpine`
|
||||
- **RabbitMQ:** `rabbitmq:3-management`
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Required Secrets (1Password)
|
||||
|
||||
Managed via ExternalSecret → 1Password vault item: `firecrawl`
|
||||
|
||||
- `POSTGRES_PASSWORD` - Database password (CRITICAL)
|
||||
- `BULL_AUTH_KEY` - Queue admin UI authentication (CRITICAL)
|
||||
- `OPENAI_API_KEY` - Optional, for AI features
|
||||
- `TEST_API_KEY` - Optional, for testing
|
||||
|
||||
### ConfigMap (Non-sensitive)
|
||||
|
||||
- Service URLs (Redis, PostgreSQL, RabbitMQ, Playwright)
|
||||
- Port configuration (3002 API, 3005 Worker)
|
||||
- Performance tuning (worker pools, concurrency limits)
|
||||
- Logging level
|
||||
|
||||
### Environment Variables Reference
|
||||
|
||||
See [ENVIRONMENT_VARIABLES.md](./ENVIRONMENT_VARIABLES.md) for complete list.
|
||||
|
||||
---
|
||||
|
||||
## Deployment Timeline
|
||||
|
||||
### Day 1 (June 6) - Investigation & Planning ✅
|
||||
- Repository analysis complete
|
||||
- Architecture decisions finalized
|
||||
- Environment configuration researched
|
||||
- Day 2 plan created
|
||||
|
||||
### Day 2 (June 7) - Tekton Pipelines 🚧
|
||||
- Create 3 build pipelines (API, Playwright, PostgreSQL)
|
||||
- Test builds and push to Harbor
|
||||
- Validate image integrity
|
||||
|
||||
### Day 3 (June 8) - Kubernetes Manifests
|
||||
- Create Deployments, Services, StatefulSets
|
||||
- Configure ConfigMaps and ExternalSecrets
|
||||
- Set up HTTPRoute for ingress
|
||||
|
||||
### Day 4 (June 9) - Secrets & Configuration
|
||||
- Create 1Password vault item
|
||||
- Configure ExternalSecret sync
|
||||
- Validate configuration
|
||||
|
||||
### Day 5 (June 10) - Deployment & Testing
|
||||
- ArgoCD Application creation
|
||||
- Deploy to cluster
|
||||
- Service health validation
|
||||
- API functionality testing
|
||||
|
||||
### Day 6 (June 11) - JARVIS Integration
|
||||
- Configure JARVIS environment variables
|
||||
- Test web search functionality
|
||||
- End-to-end validation
|
||||
- Documentation delivery
|
||||
|
||||
**Target Completion:** June 12, 2026
|
||||
|
||||
---
|
||||
|
||||
## Access & URLs
|
||||
|
||||
**Primary Access:**
|
||||
- API: https://spaceship-earth.local.mk-labs.cloud
|
||||
- Alias: https://firecrawl.local.mk-labs.cloud
|
||||
- Queue UI: https://spaceship-earth.local.mk-labs.cloud/admin/[BULL_AUTH_KEY]/queues
|
||||
|
||||
**Health Endpoints:**
|
||||
- Liveness: https://spaceship-earth.local.mk-labs.cloud/v0/health/liveness
|
||||
- Readiness: https://spaceship-earth.local.mk-labs.cloud/v0/health/readiness
|
||||
|
||||
**Internal Services (cluster-only):**
|
||||
- Playwright: http://playwright-service.firecrawl.svc:3000
|
||||
- PostgreSQL: postgresql://nuq-postgres.firecrawl.svc:5432
|
||||
- Redis: redis://redis.firecrawl.svc:6379
|
||||
- RabbitMQ: amqp://rabbitmq.firecrawl.svc:5672
|
||||
- RabbitMQ Mgmt: http://rabbitmq.firecrawl.svc:15672
|
||||
|
||||
---
|
||||
|
||||
## API Usage Examples
|
||||
|
||||
### Scrape a URL
|
||||
```bash
|
||||
curl -X POST https://spaceship-earth.local.mk-labs.cloud/v1/scrape \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"url": "https://example.com"
|
||||
}'
|
||||
```
|
||||
|
||||
### Search the Web
|
||||
```bash
|
||||
curl -X POST https://spaceship-earth.local.mk-labs.cloud/v1/search \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"query": "kubernetes best practices"
|
||||
}'
|
||||
```
|
||||
|
||||
### Crawl a Website
|
||||
```bash
|
||||
curl -X POST https://spaceship-earth.local.mk-labs.cloud/v1/crawl \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"url": "https://docs.example.com"
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
cluster/applications/firecrawl/
|
||||
├── README.md # This file
|
||||
├── ENVIRONMENT_VARIABLES.md # Complete env var reference
|
||||
├── namespace.yaml # Namespace definition
|
||||
├── configmap.yaml # Non-sensitive configuration
|
||||
├── externalsecret.yaml # 1Password secret sync
|
||||
├── postgresql/
|
||||
│ ├── statefulset.yaml # PostgreSQL StatefulSet
|
||||
│ ├── service.yaml # PostgreSQL Service
|
||||
│ └── pvc.yaml # Persistent Volume Claim
|
||||
├── redis/
|
||||
│ ├── deployment.yaml # Redis Deployment
|
||||
│ └── service.yaml # Redis Service
|
||||
├── rabbitmq/
|
||||
│ ├── deployment.yaml # RabbitMQ Deployment
|
||||
│ └── service.yaml # RabbitMQ Service
|
||||
├── playwright/
|
||||
│ ├── deployment.yaml # Playwright Deployment
|
||||
│ └── service.yaml # Playwright Service
|
||||
├── api/
|
||||
│ ├── deployment.yaml # API Deployment
|
||||
│ └── service.yaml # API Service
|
||||
├── workers/
|
||||
│ ├── worker-deployment.yaml # Queue Worker Deployment
|
||||
│ └── nuq-worker-deployment.yaml # NUQ Worker Deployment
|
||||
├── ingress/
|
||||
│ └── httproute.yaml # Gateway API HTTPRoute
|
||||
└── argocd/
|
||||
└── application.yaml # ArgoCD Application manifest
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tekton Pipelines
|
||||
|
||||
Build pipelines located in `cluster/tekton/pipelines/`:
|
||||
- `firecrawl-api-build.yaml` - API service build
|
||||
- `firecrawl-playwright-build.yaml` - Playwright service build
|
||||
- `firecrawl-postgres-build.yaml` - PostgreSQL build
|
||||
|
||||
**Trigger Builds:**
|
||||
```bash
|
||||
# API build
|
||||
kubectl create -f cluster/tekton/pipelines/firecrawl-api-build.yaml
|
||||
|
||||
# Playwright build
|
||||
kubectl create -f cluster/tekton/pipelines/firecrawl-playwright-build.yaml
|
||||
|
||||
# PostgreSQL build
|
||||
kubectl create -f cluster/tekton/pipelines/firecrawl-postgres-build.yaml
|
||||
```
|
||||
|
||||
**Monitor Builds:**
|
||||
```bash
|
||||
# List pipeline runs
|
||||
tkn pipelinerun list -n innoventions
|
||||
|
||||
# Watch logs
|
||||
tkn pipelinerun logs -f <pipelinerun-name> -n innoventions
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### API Pod Not Starting
|
||||
**Check:**
|
||||
1. RabbitMQ health status (API depends on healthy RabbitMQ)
|
||||
2. Environment variables (ConfigMap and Secret)
|
||||
3. Database connectivity (PostgreSQL)
|
||||
|
||||
**Commands:**
|
||||
```bash
|
||||
kubectl logs -n firecrawl deployment/api
|
||||
kubectl describe pod -n firecrawl -l app=api
|
||||
kubectl get externalsecret -n firecrawl
|
||||
```
|
||||
|
||||
### Build Failures
|
||||
**Check:**
|
||||
1. Harbor connectivity
|
||||
2. Harbor credentials secret
|
||||
3. Build resource limits (increase if OOM)
|
||||
|
||||
**Commands:**
|
||||
```bash
|
||||
tkn pipelinerun describe <name> -n innoventions
|
||||
kubectl logs -n innoventions <kaniko-pod>
|
||||
```
|
||||
|
||||
### Database Connection Errors
|
||||
**Check:**
|
||||
1. PostgreSQL pod status
|
||||
2. PVC binding
|
||||
3. Init script execution
|
||||
|
||||
**Commands:**
|
||||
```bash
|
||||
kubectl logs -n firecrawl statefulset/nuq-postgres
|
||||
kubectl exec -it -n firecrawl nuq-postgres-0 -- psql -U postgres -d postgres -c '\d nuq.queue_scrape'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Resource Usage:**
|
||||
```bash
|
||||
kubectl top pods -n firecrawl
|
||||
```
|
||||
|
||||
**Service Health:**
|
||||
```bash
|
||||
# API liveness
|
||||
curl -k https://spaceship-earth.local.mk-labs.cloud/v0/health/liveness
|
||||
|
||||
# API readiness
|
||||
curl -k https://spaceship-earth.local.mk-labs.cloud/v0/health/readiness
|
||||
|
||||
# Playwright health
|
||||
kubectl exec -n firecrawl deployment/playwright-service -- curl localhost:3000/health
|
||||
```
|
||||
|
||||
**Queue Status:**
|
||||
Navigate to: https://spaceship-earth.local.mk-labs.cloud/admin/[BULL_AUTH_KEY]/queues
|
||||
|
||||
---
|
||||
|
||||
## Security Notes
|
||||
|
||||
1. **PostgreSQL Credentials:** Stored in 1Password, synced via ExternalSecret
|
||||
2. **Admin UI:** Protected by BULL_AUTH_KEY (in URL path)
|
||||
3. **Database Port:** NOT exposed outside cluster (ClusterIP only)
|
||||
4. **TLS:** All external traffic encrypted via cert-manager certificates
|
||||
5. **RBAC:** Service accounts scoped to firecrawl namespace
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- **Upstream Docs:** https://docs.firecrawl.dev
|
||||
- **GitHub:** https://github.com/mendableai/firecrawl
|
||||
- **Self-Hosting Guide:** https://github.com/mendableai/firecrawl/blob/main/SELF_HOST.md
|
||||
- **K8s Examples:** `/tmp/firecrawl/examples/kubernetes/`
|
||||
- **Mission Brief:** `~/friday/inbox/agents/rocket/FIRECRAWL-DEPLOYMENT-MISSION-BRIEF.md`
|
||||
|
||||
---
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-06-06 - Day 1 Complete
|
||||
- Initial repository structure created
|
||||
- Architecture decisions finalized
|
||||
- Build strategy documented
|
||||
- Environment variables researched
|
||||
- Ready for Day 2 (pipeline creation)
|
||||
|
||||
---
|
||||
|
||||
**Contact:** Rocket Raccoon (CI/CD Specialist)
|
||||
**Project Manager:** Pepper Potts
|
||||
**Cluster:** fastpass (Talos Kubernetes)
|
||||
**Last Updated:** June 6, 2026
|
||||
Reference in New Issue
Block a user