# Firecrawl Environment Variables Reference Complete reference for all environment variables used by Firecrawl services. --- ## Required Variables (CRITICAL) These variables MUST be set for Firecrawl to function. ### Server Configuration ```yaml HOST: "0.0.0.0" # Listen address PORT: "3002" # Main API port WORKER_PORT: "3005" # Worker liveness check port EXTRACT_WORKER_PORT: "3004" # Extract worker port ``` ### Database (PostgreSQL) ```yaml POSTGRES_USER: "postgres" # Database username POSTGRES_PASSWORD: "" # 🔐 Database password (from 1Password) POSTGRES_DB: "postgres" # Database name POSTGRES_HOST: "nuq-postgres" # K8s service name POSTGRES_PORT: "5432" # PostgreSQL port ``` ### Redis (Queue & Cache) ```yaml REDIS_URL: "redis://redis:6379" REDIS_RATE_LIMIT_URL: "redis://redis:6379" ``` ### RabbitMQ (Message Broker) ```yaml NUQ_RABBITMQ_URL: "amqp://rabbitmq:5672" ``` ### Playwright Service ```yaml PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000/scrape" ``` ### Authentication ```yaml USE_DB_AUTHENTICATION: "false" # Set "true" for production with Supabase ``` --- ## Security Variables (REQUIRED) ### Admin UI Protection ```yaml BULL_AUTH_KEY: "" # 🔐 Queue admin UI password (from 1Password) # URL: /admin/{BULL_AUTH_KEY}/queues ``` ### API Testing ```yaml TEST_API_KEY: "" # 🔐 Optional - API key for testing (from 1Password) ``` --- ## Optional Variables (Features) ### AI Features (JSON Format, Extract API) ```yaml # OpenAI Configuration OPENAI_API_KEY: "" # 🔐 OpenAI API key (from 1Password) OPENAI_BASE_URL: "" # Custom OpenAI-compatible endpoint MODEL_NAME: "" # Override default model (e.g., gpt-4) MODEL_EMBEDDING_NAME: "" # Override embedding model # Ollama (Alternative to OpenAI) OLLAMA_BASE_URL: "" # E.g., http://localhost:11434/api # When using Ollama, set: # MODEL_NAME: "deepseek-r1:7b" # MODEL_EMBEDDING_NAME: "nomic-embed-text" ``` ### Proxy Configuration ```yaml PROXY_SERVER: "" # Full URL (http://0.1.2.3:1234) or IP:port PROXY_USERNAME: "" # 🔐 Proxy username (from 1Password) PROXY_PASSWORD: "" # 🔐 Proxy password (from 1Password) ``` ### Search API Configuration ```yaml # By default, uses Google search # Optionally use SearXNG instead: SEARXNG_ENDPOINT: "" # E.g., http://your.searxng.server SEARXNG_ENGINES: "" # Comma-separated engine list SEARXNG_CATEGORIES: "" # Comma-separated categories ``` ### Monitoring & Logging ```yaml LOGGING_LEVEL: "info" # debug, info, warn, error SLACK_WEBHOOK_URL: "" # 🔐 Slack webhook for alerts (from 1Password) ``` ### Supabase Integration (Advanced) ```yaml # Note: Not currently configurable for self-hosted instances SUPABASE_ANON_TOKEN: "" # For DB authentication SUPABASE_URL: "" # Supabase project URL SUPABASE_SERVICE_TOKEN: "" # Supabase service role key ``` --- ## Performance Tuning ### Worker Pools ```yaml NUM_WORKERS_PER_QUEUE: "8" # Number of workers per queue CRAWL_CONCURRENT_REQUESTS: "10" # Concurrent crawl requests MAX_CONCURRENT_JOBS: "5" # Maximum concurrent jobs ``` ### Browser Pool (Playwright) ```yaml BROWSER_POOL_SIZE: "5" # Browser instance pool size MAX_CONCURRENT_PAGES: "10" # Max concurrent pages per browser ``` ### Resource Limits (Self-Protection) ```yaml MAX_CPU: "0.8" # 0.0-1.0, reject jobs above threshold MAX_RAM: "0.8" # 0.0-1.0, reject jobs above threshold ``` ### Timeouts ```yaml HARNESS_STARTUP_TIMEOUT_MS: "60000" # 60 seconds startup timeout ``` --- ## Playwright-Specific Variables Set on `playwright-service` pods only: ```yaml PORT: "3000" # Playwright service port PROXY_SERVER: "" # Same as API proxy (if needed) PROXY_USERNAME: "" # Same as API proxy (if needed) PROXY_PASSWORD: "" # Same as API proxy (if needed) ALLOW_LOCAL_WEBHOOKS: "false" # Security: block local webhooks BLOCK_MEDIA: "" # Optional: block media resources MAX_CONCURRENT_PAGES: "10" # Browser concurrency (same as API) ``` --- ## Environment-Specific Variables ### Development ```yaml ENV: "local" ``` ### Production ```yaml ENV: "production" ``` --- ## FlyIO-Specific Variables (Not Used in K8s) These variables are set by FlyIO platform and not needed for Kubernetes deployment: ```yaml FLY_PROCESS_GROUP: "app" # Not used in K8s ``` --- ## Deprecated / Unused Variables Variables found in examples but not required for self-hosted deployment: ```yaml AUTUMN_SECRET_KEY: "" # Mendable platform-specific SELF_HOSTED_WEBHOOK_URL: "" # Custom webhook endpoint LLAMAPARSE_API_KEY: "" # PDF parsing service (optional) ``` --- ## Variable Precedence 1. **ExternalSecret** (from 1Password) - Highest priority for secrets 2. **ConfigMap** - Non-sensitive configuration 3. **Deployment env:** - Direct environment variables (override) 4. **Dockerfile defaults** - Lowest priority --- ## ConfigMap Example Non-sensitive variables suitable for ConfigMap: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: firecrawl-config namespace: firecrawl data: # Service URLs REDIS_URL: "redis://redis:6379" REDIS_RATE_LIMIT_URL: "redis://redis:6379" POSTGRES_HOST: "nuq-postgres" POSTGRES_PORT: "5432" POSTGRES_USER: "postgres" POSTGRES_DB: "postgres" NUQ_RABBITMQ_URL: "amqp://rabbitmq:5672" PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000/scrape" # Server Configuration HOST: "0.0.0.0" PORT: "3002" WORKER_PORT: "3005" EXTRACT_WORKER_PORT: "3004" USE_DB_AUTHENTICATION: "false" ENV: "production" # Performance Tuning NUM_WORKERS_PER_QUEUE: "8" CRAWL_CONCURRENT_REQUESTS: "10" MAX_CONCURRENT_JOBS: "5" BROWSER_POOL_SIZE: "5" MAX_CONCURRENT_PAGES: "10" HARNESS_STARTUP_TIMEOUT_MS: "60000" # Logging LOGGING_LEVEL: "info" # Security ALLOW_LOCAL_WEBHOOKS: "false" ``` --- ## ExternalSecret Example Sensitive variables synced from 1Password: ```yaml apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: firecrawl-secrets namespace: firecrawl spec: refreshInterval: 1h secretStoreRef: name: onepassword-connect kind: ClusterSecretStore target: name: firecrawl-secrets creationPolicy: Owner data: # Required secrets - secretKey: POSTGRES_PASSWORD remoteRef: key: firecrawl property: postgres-password - secretKey: BULL_AUTH_KEY remoteRef: key: firecrawl property: bull-auth-key # Optional secrets (uncomment when added to 1Password) # - secretKey: OPENAI_API_KEY # remoteRef: # key: firecrawl # property: openai-api-key # - secretKey: TEST_API_KEY # remoteRef: # key: firecrawl # property: test-api-key ``` --- ## Usage in Deployments ### Combining ConfigMap and Secret ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: api namespace: firecrawl spec: template: spec: containers: - name: api image: the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest envFrom: # Load all non-sensitive variables - configMapRef: name: firecrawl-config # Load all secrets - secretRef: name: firecrawl-secrets env: # Override specific variables if needed - name: FLY_PROCESS_GROUP value: "app" ``` ### Playwright-Specific ConfigMap ```yaml apiVersion: v1 kind: ConfigMap metadata: name: playwright-service-config namespace: firecrawl data: PORT: "3000" ALLOW_LOCAL_WEBHOOKS: "false" MAX_CONCURRENT_PAGES: "10" ``` --- ## Validation Checklist Before deployment, ensure: - [ ] `POSTGRES_PASSWORD` set in 1Password - [ ] `BULL_AUTH_KEY` set in 1Password (strong random value) - [ ] All service URLs use correct K8s service names - [ ] Ports match service definitions (3002, 3000, 5432, 6379, 5672) - [ ] `USE_DB_AUTHENTICATION` is "false" (Supabase not available) - [ ] `ENV` is "production" (not "local") - [ ] Worker pool sizes appropriate for cluster capacity - [ ] `ALLOW_LOCAL_WEBHOOKS` is "false" (security) --- ## Troubleshooting ### "Supabase client is not configured" **Expected warning** - Supabase is not available for self-hosted instances. Safe to ignore. ### "You're bypassing authentication" **Expected warning** - When `USE_DB_AUTHENTICATION=false`. Normal for self-hosted deployment. ### Connection refused errors **Check:** - Service names match environment variables - Services are running: `kubectl get svc -n firecrawl` - Pods are ready: `kubectl get pods -n firecrawl` ### Build failures **Check:** - Required build args are set - Kaniko has sufficient memory (4GB for API build) --- ## Security Best Practices 1. **Generate strong secrets:** ```bash # Generate BULL_AUTH_KEY openssl rand -base64 32 # Generate POSTGRES_PASSWORD openssl rand -base64 24 ``` 2. **Never expose PostgreSQL externally:** - Use ClusterIP service only - No LoadBalancer or NodePort - Access via kubectl port-forward for maintenance 3. **Protect admin UI:** - Keep BULL_AUTH_KEY secret - Don't commit to git - Rotate periodically 4. **Use ExternalSecrets:** - Never put secrets in ConfigMaps - Never commit secrets to git - Store all sensitive data in 1Password --- ## References - **Upstream .env Template:** `/tmp/firecrawl/SELF_HOST.md` - **Docker Compose Reference:** `/tmp/firecrawl/docker-compose.yaml` - **K8s Example ConfigMap:** `/tmp/firecrawl/examples/kubernetes/cluster-install/configmap.yaml` --- **Last Updated:** June 6, 2026 **Maintained By:** Rocket Raccoon (CI/CD Specialist)