rblundon/homelab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
24735f7e5c90a4bb13fcd3a56bed145947570eb3
homelab/cluster/applications/firecrawl/ENVIRONMENT_VARIABLES.md
Hermes Agent service account d974c75d7c feat(jmri): headless JMRI server with Leviton layout power monitor and X11 GUI mode 
- Stable udev device symlinks (/dev/jmri/nce, /dev/jmri/loconet, /dev/jmri/lcc)
- jmri-monitor: polls Leviton Decora Smart switch to start/stop JMRI automatically
  - Quiet hours 1-10 AM (no polling)
  - 30s off-delay before shutdown
- LCRR config cloned from Gitea (ssh://gitea.mk-labs.cloud:2221/rblundon/LCRR.git)
- ~/.jmri symlinked to LCRR repo for GitOps config management
- jmri-gui: X11 remote GUI access (PanelPro/DecoderPro) via ssh -X as jmri user
  - Stops daemon, launches GUI, restarts daemon on exit if layout still on
- jmri user gets login shell + SSH key for GUI sessions
- Full JRE installed (openjdk-21-jre) for AWT/X11 support
2026-07-29 00:43:23 -05:00

10 KiB
Raw Blame History

Firecrawl Environment Variables Reference

Complete reference for all environment variables used by Firecrawl services.

Required Variables (CRITICAL)

These variables MUST be set for Firecrawl to function.

Server Configuration

HOST: "0.0.0.0"                    # Listen address
PORT: "3002"                       # Main API port
WORKER_PORT: "3005"                # Worker liveness check port
EXTRACT_WORKER_PORT: "3004"        # Extract worker port

Database (PostgreSQL)

POSTGRES_USER: "postgres"          # Database username
POSTGRES_PASSWORD: "<SECRET>"      # 🔐 Database password (from 1Password)
POSTGRES_DB: "postgres"            # Database name
POSTGRES_HOST: "nuq-postgres"      # K8s service name
POSTGRES_PORT: "5432"              # PostgreSQL port

Redis (Queue & Cache)

REDIS_URL: "redis://redis:6379"
REDIS_RATE_LIMIT_URL: "redis://redis:6379"

RabbitMQ (Message Broker)

NUQ_RABBITMQ_URL: "amqp://rabbitmq:5672"

Playwright Service

PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000/scrape"

Authentication

USE_DB_AUTHENTICATION: "false"     # Set "true" for production with Supabase

Security Variables (REQUIRED)

Admin UI Protection

BULL_AUTH_KEY: "<SECRET>"          # 🔐 Queue admin UI password (from 1Password)
                                   # URL: /admin/{BULL_AUTH_KEY}/queues

API Testing

TEST_API_KEY: "<SECRET>"           # 🔐 Optional - API key for testing (from 1Password)

Optional Variables (Features)

AI Features (JSON Format, Extract API)

# OpenAI Configuration
OPENAI_API_KEY: "<SECRET>"         # 🔐 OpenAI API key (from 1Password)
OPENAI_BASE_URL: ""                # Custom OpenAI-compatible endpoint
MODEL_NAME: ""                     # Override default model (e.g., gpt-4)
MODEL_EMBEDDING_NAME: ""           # Override embedding model

# Ollama (Alternative to OpenAI)
OLLAMA_BASE_URL: ""                # E.g., http://localhost:11434/api
# When using Ollama, set:
# MODEL_NAME: "deepseek-r1:7b"
# MODEL_EMBEDDING_NAME: "nomic-embed-text"

Proxy Configuration

PROXY_SERVER: ""                   # Full URL (http://0.1.2.3:1234) or IP:port
PROXY_USERNAME: ""                 # 🔐 Proxy username (from 1Password)
PROXY_PASSWORD: ""                 # 🔐 Proxy password (from 1Password)

Search API Configuration

# By default, uses Google search
# Optionally use SearXNG instead:
SEARXNG_ENDPOINT: ""               # E.g., http://your.searxng.server
SEARXNG_ENGINES: ""                # Comma-separated engine list
SEARXNG_CATEGORIES: ""             # Comma-separated categories

Monitoring & Logging

LOGGING_LEVEL: "info"              # debug, info, warn, error
SLACK_WEBHOOK_URL: ""              # 🔐 Slack webhook for alerts (from 1Password)

Supabase Integration (Advanced)

# Note: Not currently configurable for self-hosted instances
SUPABASE_ANON_TOKEN: ""            # For DB authentication
SUPABASE_URL: ""                   # Supabase project URL
SUPABASE_SERVICE_TOKEN: ""         # Supabase service role key

Performance Tuning

Worker Pools

NUM_WORKERS_PER_QUEUE: "8"         # Number of workers per queue
CRAWL_CONCURRENT_REQUESTS: "10"    # Concurrent crawl requests
MAX_CONCURRENT_JOBS: "5"           # Maximum concurrent jobs

Browser Pool (Playwright)

BROWSER_POOL_SIZE: "5"             # Browser instance pool size
MAX_CONCURRENT_PAGES: "10"         # Max concurrent pages per browser

Resource Limits (Self-Protection)

MAX_CPU: "0.8"                     # 0.0-1.0, reject jobs above threshold
MAX_RAM: "0.8"                     # 0.0-1.0, reject jobs above threshold

Timeouts

HARNESS_STARTUP_TIMEOUT_MS: "60000" # 60 seconds startup timeout

Playwright-Specific Variables

Set on playwright-service pods only:

PORT: "3000"                       # Playwright service port
PROXY_SERVER: ""                   # Same as API proxy (if needed)
PROXY_USERNAME: ""                 # Same as API proxy (if needed)
PROXY_PASSWORD: ""                 # Same as API proxy (if needed)
ALLOW_LOCAL_WEBHOOKS: "false"      # Security: block local webhooks
BLOCK_MEDIA: ""                    # Optional: block media resources
MAX_CONCURRENT_PAGES: "10"         # Browser concurrency (same as API)

Environment-Specific Variables

Development

ENV: "local"

Production

ENV: "production"

FlyIO-Specific Variables (Not Used in K8s)

These variables are set by FlyIO platform and not needed for Kubernetes deployment:

FLY_PROCESS_GROUP: "app"           # Not used in K8s

Deprecated / Unused Variables

Variables found in examples but not required for self-hosted deployment:

AUTUMN_SECRET_KEY: ""              # Mendable platform-specific
SELF_HOSTED_WEBHOOK_URL: ""        # Custom webhook endpoint
LLAMAPARSE_API_KEY: ""             # PDF parsing service (optional)

Variable Precedence

  1. ExternalSecret (from 1Password) - Highest priority for secrets
  2. ConfigMap - Non-sensitive configuration
  3. Deployment env: - Direct environment variables (override)
  4. Dockerfile defaults - Lowest priority

ConfigMap Example

Non-sensitive variables suitable for ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: firecrawl-config
  namespace: firecrawl
data:
  # Service URLs
  REDIS_URL: "redis://redis:6379"
  REDIS_RATE_LIMIT_URL: "redis://redis:6379"
  POSTGRES_HOST: "nuq-postgres"
  POSTGRES_PORT: "5432"
  POSTGRES_USER: "postgres"
  POSTGRES_DB: "postgres"
  NUQ_RABBITMQ_URL: "amqp://rabbitmq:5672"
  PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000/scrape"
  
  # Server Configuration
  HOST: "0.0.0.0"
  PORT: "3002"
  WORKER_PORT: "3005"
  EXTRACT_WORKER_PORT: "3004"
  USE_DB_AUTHENTICATION: "false"
  ENV: "production"
  
  # Performance Tuning
  NUM_WORKERS_PER_QUEUE: "8"
  CRAWL_CONCURRENT_REQUESTS: "10"
  MAX_CONCURRENT_JOBS: "5"
  BROWSER_POOL_SIZE: "5"
  MAX_CONCURRENT_PAGES: "10"
  HARNESS_STARTUP_TIMEOUT_MS: "60000"
  
  # Logging
  LOGGING_LEVEL: "info"
  
  # Security
  ALLOW_LOCAL_WEBHOOKS: "false"

ExternalSecret Example

Sensitive variables synced from 1Password:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: firecrawl-secrets
  namespace: firecrawl
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: onepassword-connect
    kind: ClusterSecretStore
  target:
    name: firecrawl-secrets
    creationPolicy: Owner
  data:
    # Required secrets
    - secretKey: POSTGRES_PASSWORD
      remoteRef:
        key: firecrawl
        property: postgres-password
    
    - secretKey: BULL_AUTH_KEY
      remoteRef:
        key: firecrawl
        property: bull-auth-key
    
    # Optional secrets (uncomment when added to 1Password)
    # - secretKey: OPENAI_API_KEY
    #   remoteRef:
    #     key: firecrawl
    #     property: openai-api-key
    
    # - secretKey: TEST_API_KEY
    #   remoteRef:
    #     key: firecrawl
    #     property: test-api-key

Usage in Deployments

Combining ConfigMap and Secret

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: firecrawl
spec:
  template:
    spec:
      containers:
        - name: api
          image: the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest
          envFrom:
            # Load all non-sensitive variables
            - configMapRef:
                name: firecrawl-config
            # Load all secrets
            - secretRef:
                name: firecrawl-secrets
          env:
            # Override specific variables if needed
            - name: FLY_PROCESS_GROUP
              value: "app"

Playwright-Specific ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: playwright-service-config
  namespace: firecrawl
data:
  PORT: "3000"
  ALLOW_LOCAL_WEBHOOKS: "false"
  MAX_CONCURRENT_PAGES: "10"

Validation Checklist

Before deployment, ensure:

  • POSTGRES_PASSWORD set in 1Password
  • BULL_AUTH_KEY set in 1Password (strong random value)
  • All service URLs use correct K8s service names
  • Ports match service definitions (3002, 3000, 5432, 6379, 5672)
  • USE_DB_AUTHENTICATION is "false" (Supabase not available)
  • ENV is "production" (not "local")
  • Worker pool sizes appropriate for cluster capacity
  • ALLOW_LOCAL_WEBHOOKS is "false" (security)

Troubleshooting

"Supabase client is not configured"

Expected warning - Supabase is not available for self-hosted instances. Safe to ignore.

"You're bypassing authentication"

Expected warning - When USE_DB_AUTHENTICATION=false. Normal for self-hosted deployment.

Connection refused errors

Check:

  • Service names match environment variables
  • Services are running: kubectl get svc -n firecrawl
  • Pods are ready: kubectl get pods -n firecrawl

Build failures

Check:

  • Required build args are set
  • Kaniko has sufficient memory (4GB for API build)

Security Best Practices

  1. Generate strong secrets:

    # Generate BULL_AUTH_KEY
openssl rand -base64 32

# Generate POSTGRES_PASSWORD
openssl rand -base64 24

  2. Never expose PostgreSQL externally:

    • Use ClusterIP service only
    • No LoadBalancer or NodePort
    • Access via kubectl port-forward for maintenance

  3. Protect admin UI:

    • Keep BULL_AUTH_KEY secret
    • Don't commit to git
    • Rotate periodically

  4. Use ExternalSecrets:

    • Never put secrets in ConfigMaps
    • Never commit secrets to git
    • Store all sensitive data in 1Password

References

  • Upstream .env Template: /tmp/firecrawl/SELF_HOST.md
  • Docker Compose Reference: /tmp/firecrawl/docker-compose.yaml
  • K8s Example ConfigMap: /tmp/firecrawl/examples/kubernetes/cluster-install/configmap.yaml

Last Updated: June 6, 2026
Maintained By: Rocket Raccoon (CI/CD Specialist)

Reference in New Issue View Git Blame Copy Permalink