feat(jmri): headless JMRI server with Leviton layout power monitor and X11 GUI mode
- Stable udev device symlinks (/dev/jmri/nce, /dev/jmri/loconet, /dev/jmri/lcc) - jmri-monitor: polls Leviton Decora Smart switch to start/stop JMRI automatically - Quiet hours 1-10 AM (no polling) - 30s off-delay before shutdown - LCRR config cloned from Gitea (ssh://gitea.mk-labs.cloud:2221/rblundon/LCRR.git) - ~/.jmri symlinked to LCRR repo for GitOps config management - jmri-gui: X11 remote GUI access (PanelPro/DecoderPro) via ssh -X as jmri user - Stops daemon, launches GUI, restarts daemon on exit if layout still on - jmri user gets login shell + SSH key for GUI sessions - Full JRE installed (openjdk-21-jre) for AWT/X11 support
This commit is contained in:
Hermes Agent service account
412
cluster/applications/firecrawl/ENVIRONMENT_VARIABLES.md
Normal file
412
cluster/applications/firecrawl/ENVIRONMENT_VARIABLES.md
Normal file
@@ -0,0 +1,412 @@
|
||||
# Firecrawl Environment Variables Reference
|
||||
|
||||
Complete reference for all environment variables used by Firecrawl services.
|
||||
|
||||
---
|
||||
|
||||
## Required Variables (CRITICAL)
|
||||
|
||||
These variables MUST be set for Firecrawl to function.
|
||||
|
||||
### Server Configuration
|
||||
```yaml
|
||||
HOST: "0.0.0.0" # Listen address
|
||||
PORT: "3002" # Main API port
|
||||
WORKER_PORT: "3005" # Worker liveness check port
|
||||
EXTRACT_WORKER_PORT: "3004" # Extract worker port
|
||||
```
|
||||
|
||||
### Database (PostgreSQL)
|
||||
```yaml
|
||||
POSTGRES_USER: "postgres" # Database username
|
||||
POSTGRES_PASSWORD: "<SECRET>" # 🔐 Database password (from 1Password)
|
||||
POSTGRES_DB: "postgres" # Database name
|
||||
POSTGRES_HOST: "nuq-postgres" # K8s service name
|
||||
POSTGRES_PORT: "5432" # PostgreSQL port
|
||||
```
|
||||
|
||||
### Redis (Queue & Cache)
|
||||
```yaml
|
||||
REDIS_URL: "redis://redis:6379"
|
||||
REDIS_RATE_LIMIT_URL: "redis://redis:6379"
|
||||
```
|
||||
|
||||
### RabbitMQ (Message Broker)
|
||||
```yaml
|
||||
NUQ_RABBITMQ_URL: "amqp://rabbitmq:5672"
|
||||
```
|
||||
|
||||
### Playwright Service
|
||||
```yaml
|
||||
PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000/scrape"
|
||||
```
|
||||
|
||||
### Authentication
|
||||
```yaml
|
||||
USE_DB_AUTHENTICATION: "false" # Set "true" for production with Supabase
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Variables (REQUIRED)
|
||||
|
||||
### Admin UI Protection
|
||||
```yaml
|
||||
BULL_AUTH_KEY: "<SECRET>" # 🔐 Queue admin UI password (from 1Password)
|
||||
# URL: /admin/{BULL_AUTH_KEY}/queues
|
||||
```
|
||||
|
||||
### API Testing
|
||||
```yaml
|
||||
TEST_API_KEY: "<SECRET>" # 🔐 Optional - API key for testing (from 1Password)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Optional Variables (Features)
|
||||
|
||||
### AI Features (JSON Format, Extract API)
|
||||
```yaml
|
||||
# OpenAI Configuration
|
||||
OPENAI_API_KEY: "<SECRET>" # 🔐 OpenAI API key (from 1Password)
|
||||
OPENAI_BASE_URL: "" # Custom OpenAI-compatible endpoint
|
||||
MODEL_NAME: "" # Override default model (e.g., gpt-4)
|
||||
MODEL_EMBEDDING_NAME: "" # Override embedding model
|
||||
|
||||
# Ollama (Alternative to OpenAI)
|
||||
OLLAMA_BASE_URL: "" # E.g., http://localhost:11434/api
|
||||
# When using Ollama, set:
|
||||
# MODEL_NAME: "deepseek-r1:7b"
|
||||
# MODEL_EMBEDDING_NAME: "nomic-embed-text"
|
||||
```
|
||||
|
||||
### Proxy Configuration
|
||||
```yaml
|
||||
PROXY_SERVER: "" # Full URL (http://0.1.2.3:1234) or IP:port
|
||||
PROXY_USERNAME: "" # 🔐 Proxy username (from 1Password)
|
||||
PROXY_PASSWORD: "" # 🔐 Proxy password (from 1Password)
|
||||
```
|
||||
|
||||
### Search API Configuration
|
||||
```yaml
|
||||
# By default, uses Google search
|
||||
# Optionally use SearXNG instead:
|
||||
SEARXNG_ENDPOINT: "" # E.g., http://your.searxng.server
|
||||
SEARXNG_ENGINES: "" # Comma-separated engine list
|
||||
SEARXNG_CATEGORIES: "" # Comma-separated categories
|
||||
```
|
||||
|
||||
### Monitoring & Logging
|
||||
```yaml
|
||||
LOGGING_LEVEL: "info" # debug, info, warn, error
|
||||
SLACK_WEBHOOK_URL: "" # 🔐 Slack webhook for alerts (from 1Password)
|
||||
```
|
||||
|
||||
### Supabase Integration (Advanced)
|
||||
```yaml
|
||||
# Note: Not currently configurable for self-hosted instances
|
||||
SUPABASE_ANON_TOKEN: "" # For DB authentication
|
||||
SUPABASE_URL: "" # Supabase project URL
|
||||
SUPABASE_SERVICE_TOKEN: "" # Supabase service role key
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### Worker Pools
|
||||
```yaml
|
||||
NUM_WORKERS_PER_QUEUE: "8" # Number of workers per queue
|
||||
CRAWL_CONCURRENT_REQUESTS: "10" # Concurrent crawl requests
|
||||
MAX_CONCURRENT_JOBS: "5" # Maximum concurrent jobs
|
||||
```
|
||||
|
||||
### Browser Pool (Playwright)
|
||||
```yaml
|
||||
BROWSER_POOL_SIZE: "5" # Browser instance pool size
|
||||
MAX_CONCURRENT_PAGES: "10" # Max concurrent pages per browser
|
||||
```
|
||||
|
||||
### Resource Limits (Self-Protection)
|
||||
```yaml
|
||||
MAX_CPU: "0.8" # 0.0-1.0, reject jobs above threshold
|
||||
MAX_RAM: "0.8" # 0.0-1.0, reject jobs above threshold
|
||||
```
|
||||
|
||||
### Timeouts
|
||||
```yaml
|
||||
HARNESS_STARTUP_TIMEOUT_MS: "60000" # 60 seconds startup timeout
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Playwright-Specific Variables
|
||||
|
||||
Set on `playwright-service` pods only:
|
||||
|
||||
```yaml
|
||||
PORT: "3000" # Playwright service port
|
||||
PROXY_SERVER: "" # Same as API proxy (if needed)
|
||||
PROXY_USERNAME: "" # Same as API proxy (if needed)
|
||||
PROXY_PASSWORD: "" # Same as API proxy (if needed)
|
||||
ALLOW_LOCAL_WEBHOOKS: "false" # Security: block local webhooks
|
||||
BLOCK_MEDIA: "" # Optional: block media resources
|
||||
MAX_CONCURRENT_PAGES: "10" # Browser concurrency (same as API)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment-Specific Variables
|
||||
|
||||
### Development
|
||||
```yaml
|
||||
ENV: "local"
|
||||
```
|
||||
|
||||
### Production
|
||||
```yaml
|
||||
ENV: "production"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## FlyIO-Specific Variables (Not Used in K8s)
|
||||
|
||||
These variables are set by FlyIO platform and not needed for Kubernetes deployment:
|
||||
|
||||
```yaml
|
||||
FLY_PROCESS_GROUP: "app" # Not used in K8s
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deprecated / Unused Variables
|
||||
|
||||
Variables found in examples but not required for self-hosted deployment:
|
||||
|
||||
```yaml
|
||||
AUTUMN_SECRET_KEY: "" # Mendable platform-specific
|
||||
SELF_HOSTED_WEBHOOK_URL: "" # Custom webhook endpoint
|
||||
LLAMAPARSE_API_KEY: "" # PDF parsing service (optional)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Variable Precedence
|
||||
|
||||
1. **ExternalSecret** (from 1Password) - Highest priority for secrets
|
||||
2. **ConfigMap** - Non-sensitive configuration
|
||||
3. **Deployment env:** - Direct environment variables (override)
|
||||
4. **Dockerfile defaults** - Lowest priority
|
||||
|
||||
---
|
||||
|
||||
## ConfigMap Example
|
||||
|
||||
Non-sensitive variables suitable for ConfigMap:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: firecrawl-config
|
||||
namespace: firecrawl
|
||||
data:
|
||||
# Service URLs
|
||||
REDIS_URL: "redis://redis:6379"
|
||||
REDIS_RATE_LIMIT_URL: "redis://redis:6379"
|
||||
POSTGRES_HOST: "nuq-postgres"
|
||||
POSTGRES_PORT: "5432"
|
||||
POSTGRES_USER: "postgres"
|
||||
POSTGRES_DB: "postgres"
|
||||
NUQ_RABBITMQ_URL: "amqp://rabbitmq:5672"
|
||||
PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000/scrape"
|
||||
|
||||
# Server Configuration
|
||||
HOST: "0.0.0.0"
|
||||
PORT: "3002"
|
||||
WORKER_PORT: "3005"
|
||||
EXTRACT_WORKER_PORT: "3004"
|
||||
USE_DB_AUTHENTICATION: "false"
|
||||
ENV: "production"
|
||||
|
||||
# Performance Tuning
|
||||
NUM_WORKERS_PER_QUEUE: "8"
|
||||
CRAWL_CONCURRENT_REQUESTS: "10"
|
||||
MAX_CONCURRENT_JOBS: "5"
|
||||
BROWSER_POOL_SIZE: "5"
|
||||
MAX_CONCURRENT_PAGES: "10"
|
||||
HARNESS_STARTUP_TIMEOUT_MS: "60000"
|
||||
|
||||
# Logging
|
||||
LOGGING_LEVEL: "info"
|
||||
|
||||
# Security
|
||||
ALLOW_LOCAL_WEBHOOKS: "false"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ExternalSecret Example
|
||||
|
||||
Sensitive variables synced from 1Password:
|
||||
|
||||
```yaml
|
||||
apiVersion: external-secrets.io/v1beta1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
name: firecrawl-secrets
|
||||
namespace: firecrawl
|
||||
spec:
|
||||
refreshInterval: 1h
|
||||
secretStoreRef:
|
||||
name: onepassword-connect
|
||||
kind: ClusterSecretStore
|
||||
target:
|
||||
name: firecrawl-secrets
|
||||
creationPolicy: Owner
|
||||
data:
|
||||
# Required secrets
|
||||
- secretKey: POSTGRES_PASSWORD
|
||||
remoteRef:
|
||||
key: firecrawl
|
||||
property: postgres-password
|
||||
|
||||
- secretKey: BULL_AUTH_KEY
|
||||
remoteRef:
|
||||
key: firecrawl
|
||||
property: bull-auth-key
|
||||
|
||||
# Optional secrets (uncomment when added to 1Password)
|
||||
# - secretKey: OPENAI_API_KEY
|
||||
# remoteRef:
|
||||
# key: firecrawl
|
||||
# property: openai-api-key
|
||||
|
||||
# - secretKey: TEST_API_KEY
|
||||
# remoteRef:
|
||||
# key: firecrawl
|
||||
# property: test-api-key
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Usage in Deployments
|
||||
|
||||
### Combining ConfigMap and Secret
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: api
|
||||
namespace: firecrawl
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: api
|
||||
image: the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest
|
||||
envFrom:
|
||||
# Load all non-sensitive variables
|
||||
- configMapRef:
|
||||
name: firecrawl-config
|
||||
# Load all secrets
|
||||
- secretRef:
|
||||
name: firecrawl-secrets
|
||||
env:
|
||||
# Override specific variables if needed
|
||||
- name: FLY_PROCESS_GROUP
|
||||
value: "app"
|
||||
```
|
||||
|
||||
### Playwright-Specific ConfigMap
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: playwright-service-config
|
||||
namespace: firecrawl
|
||||
data:
|
||||
PORT: "3000"
|
||||
ALLOW_LOCAL_WEBHOOKS: "false"
|
||||
MAX_CONCURRENT_PAGES: "10"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Checklist
|
||||
|
||||
Before deployment, ensure:
|
||||
|
||||
- [ ] `POSTGRES_PASSWORD` set in 1Password
|
||||
- [ ] `BULL_AUTH_KEY` set in 1Password (strong random value)
|
||||
- [ ] All service URLs use correct K8s service names
|
||||
- [ ] Ports match service definitions (3002, 3000, 5432, 6379, 5672)
|
||||
- [ ] `USE_DB_AUTHENTICATION` is "false" (Supabase not available)
|
||||
- [ ] `ENV` is "production" (not "local")
|
||||
- [ ] Worker pool sizes appropriate for cluster capacity
|
||||
- [ ] `ALLOW_LOCAL_WEBHOOKS` is "false" (security)
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Supabase client is not configured"
|
||||
**Expected warning** - Supabase is not available for self-hosted instances. Safe to ignore.
|
||||
|
||||
### "You're bypassing authentication"
|
||||
**Expected warning** - When `USE_DB_AUTHENTICATION=false`. Normal for self-hosted deployment.
|
||||
|
||||
### Connection refused errors
|
||||
**Check:**
|
||||
- Service names match environment variables
|
||||
- Services are running: `kubectl get svc -n firecrawl`
|
||||
- Pods are ready: `kubectl get pods -n firecrawl`
|
||||
|
||||
### Build failures
|
||||
**Check:**
|
||||
- Required build args are set
|
||||
- Kaniko has sufficient memory (4GB for API build)
|
||||
|
||||
---
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
1. **Generate strong secrets:**
|
||||
```bash
|
||||
# Generate BULL_AUTH_KEY
|
||||
openssl rand -base64 32
|
||||
|
||||
# Generate POSTGRES_PASSWORD
|
||||
openssl rand -base64 24
|
||||
```
|
||||
|
||||
2. **Never expose PostgreSQL externally:**
|
||||
- Use ClusterIP service only
|
||||
- No LoadBalancer or NodePort
|
||||
- Access via kubectl port-forward for maintenance
|
||||
|
||||
3. **Protect admin UI:**
|
||||
- Keep BULL_AUTH_KEY secret
|
||||
- Don't commit to git
|
||||
- Rotate periodically
|
||||
|
||||
4. **Use ExternalSecrets:**
|
||||
- Never put secrets in ConfigMaps
|
||||
- Never commit secrets to git
|
||||
- Store all sensitive data in 1Password
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- **Upstream .env Template:** `/tmp/firecrawl/SELF_HOST.md`
|
||||
- **Docker Compose Reference:** `/tmp/firecrawl/docker-compose.yaml`
|
||||
- **K8s Example ConfigMap:** `/tmp/firecrawl/examples/kubernetes/cluster-install/configmap.yaml`
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** June 6, 2026
|
||||
**Maintained By:** Rocket Raccoon (CI/CD Specialist)
|
||||
Reference in New Issue
Block a user