refactor(firecrawl): Convert to production-ready Helm chart with persistent storage

CRITICAL FIXES:
 Add PersistentVolumeClaims for all stateful services
   - PostgreSQL: 20Gi PVC on nfs-emporium (data persistence)
   - Redis: 10Gi PVC on nfs-emporium (cache and queue)
   - RabbitMQ: 5Gi PVC on nfs-emporium (message queue)

 Pin all image versions (no more 'latest' tags)
   - Firecrawl API/Worker: v1.0.0
   - Playwright Service: v1.0.0
   - PostgreSQL (nuq-postgres): v1.0.0
   - Redis: 7.4.1-alpine
   - RabbitMQ: 3.13.7-management-alpine

 Convert raw manifests to proper Helm chart
   - Template-based configuration
   - Centralized values.yaml
   - Proper Helm helpers and labels
   - Easy configuration management

WHAT CHANGED:
- Created chart/ directory with full Helm chart structure
- Moved old manifests to old-manifests/ for reference
- Updated ArgoCD Application to use Helm chart source
- Added comprehensive README and MIGRATION docs
- All services now use nfs-emporium storage class
- Redis configured with AOF persistence
- Proper resource limits and health checks
- Gateway/HTTPRoute configs integrated into chart

DEPLOYMENT:
ArgoCD will automatically sync and apply changes.
Old ephemeral data will be lost (fresh start with persistence).

Resolves data loss issues and brings deployment to production standards.
This commit is contained in:
Hermes Agent service account
2026-06-04 16:56:29 -05:00
parent f4181349f8
commit 6bcb6fa93f
31 changed files with 1354 additions and 7 deletions

View File

@@ -0,0 +1,97 @@
# Migration to Helm Chart
**Date**: 2026-06-04
**Reason**: Production-ready refactoring
## Critical Issues Fixed
### 1. ❌ No Persistent Storage → ✅ PersistentVolumeClaims
**Old**: PostgreSQL and Redis used `emptyDir` volumes
- All data lost on pod restart
- Not production-ready
**New**: All stateful services have PVCs
- PostgreSQL: 20Gi PVC on nfs-emporium
- Redis: 10Gi PVC on nfs-emporium
- RabbitMQ: 5Gi PVC on nfs-emporium
### 2. ❌ Using `latest` Tags → ✅ Pinned Versions
**Old**: Images used `latest` or unspecified tags
- Unpredictable updates
- Hard to rollback
- No version control
**New**: All images pinned to specific versions
- Firecrawl: `v1.0.0`
- Redis: `7.4.1-alpine`
- RabbitMQ: `3.13.7-management-alpine`
- PostgreSQL: `v1.0.0`
- Playwright: `v1.0.0`
### 3. ❌ Raw Manifests → ✅ Helm Chart
**Old**: Individual YAML files
- Hard to maintain
- No templating
- Difficult to customize
**New**: Proper Helm chart structure
- Template-based configuration
- Easy value overrides
- Industry standard deployment
## Migration Steps
1. **Backup** (if needed): Old deployment had ephemeral storage, no data to preserve
2. **ArgoCD Auto-Sync**: Push changes and ArgoCD will:
- Delete old resources
- Create new Helm release
- Provision PVCs
- Deploy with pinned versions
## Old Files Location
All previous manifests are archived in `old-manifests/` directory for reference.
## Rollback Procedure
If needed to rollback:
```bash
# Revert git commit
git revert HEAD
# ArgoCD will automatically sync back to old state
# Or manually:
argocd app sync firecrawl
```
## Verification
After deployment:
```bash
# Check all PVCs are bound
kubectl -n firecrawl get pvc
# Verify pods are running
kubectl -n firecrawl get pods
# Check image versions
kubectl -n firecrawl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}'
# Test API
curl https://spaceship-earth.local.mk-labs.cloud/v0/health/liveness
```
## Benefits
**Data Persistence**: No more data loss on restarts
**Version Control**: Explicit version management
**Configuration Management**: Clean Helm values
**Maintainability**: Standard Helm patterns
**Production Ready**: Follows Kubernetes best practices
## Notes
- Storage class `nfs-emporium` provides persistent NFS-backed storage
- All resources properly labeled with Helm chart metadata
- Gateway/HTTPRoute configurations preserved
- EPCOT theming (spaceship-earth) maintained

View File

@@ -10,9 +10,15 @@
# - Firecrawl API (main service)
# - Firecrawl Worker (background job processor)
# - Playwright Service (browser automation)
# - Redis (cache and job queue)
# - PostgreSQL (state management)
# - RabbitMQ (message queue)
# - Redis (cache and job queue) - WITH PERSISTENT STORAGE
# - PostgreSQL (state management) - WITH PERSISTENT STORAGE
# - RabbitMQ (message queue) - WITH PERSISTENT STORAGE
#
# PRODUCTION-READY FEATURES:
# - All stateful services use PersistentVolumeClaims
# - All images pinned to specific versions
# - Helm chart for configuration management
# - Resource limits and requests properly configured
# ------------------------------------------------------------------------------
apiVersion: argoproj.io/v1alpha1
kind: Application
@@ -30,10 +36,13 @@ spec:
source:
repoURL: https://gitea.mk-labs.cloud/rblundon/homelab.git
targetRevision: main
path: cluster/applications/firecrawl
directory:
recurse: false
exclude: application.yaml # prevent self-reference loop
path: cluster/applications/firecrawl/chart
helm:
releaseName: firecrawl
values: |
# Production configuration
# Storage classes use nfs-emporium (Synology NFS)
# All images pinned to specific versions for stability
destination:
server: https://kubernetes.default.svc
namespace: firecrawl

View File

@@ -0,0 +1,8 @@
apiVersion: v2
name: firecrawl
description: A Helm chart for deploying Firecrawl web scraping service
type: application
version: 1.0.0
appVersion: "1.0.0"
maintainers:
- name: mk-labs

View File

@@ -0,0 +1,309 @@
# Firecrawl Helm Chart
Production-ready Helm chart for deploying Firecrawl web scraping service.
## Overview
Firecrawl is a web scraping and search service that provides JARVIS with web context capabilities. This deployment includes all necessary components with proper persistent storage and pinned image versions.
## Components
### Application Services
- **Firecrawl API** - Main REST API service
- Port: 3002
- Health checks: `/v0/health/liveness`, `/v0/health/readiness`
- Resources: 4Gi memory, 2 CPU cores
- **Firecrawl Worker** - Background job processor
- Processes crawl jobs from queue
- Resources: 3Gi memory, 1 CPU core
- **Playwright Service** - Browser automation
- Port: 3000
- Headless Chrome/Firefox for JavaScript rendering
- Resources: 2Gi memory, 1 CPU core
### Infrastructure Services
- **PostgreSQL** (nuq-postgres) - State management
- Port: 5432
- Custom image with queue extensions
- **Persistent Storage**: 20Gi PVC on nfs-emporium
- **Redis** - Cache and job queue
- Port: 6379
- AOF persistence enabled
- **Persistent Storage**: 10Gi PVC on nfs-emporium
- **RabbitMQ** - Message queue
- AMQP Port: 5672
- Management UI: 15672
- **Persistent Storage**: 5Gi PVC on nfs-emporium
## Production Features
### ✅ Persistent Storage
All stateful services use PersistentVolumeClaims backed by NFS storage:
- PostgreSQL: 20Gi (database state)
- Redis: 10Gi (cache and job queue)
- RabbitMQ: 5Gi (message queue data)
### ✅ Pinned Image Versions
All container images use specific version tags (no `latest`):
- Firecrawl API/Worker: `ghcr.io/mendableai/firecrawl:v1.0.0`
- Playwright Service: `ghcr.io/mendableai/firecrawl/playwright-service:v1.0.0`
- PostgreSQL: `ghcr.io/mendableai/firecrawl/nuq-postgres:v1.0.0`
- Redis: `redis:7.4.1-alpine`
- RabbitMQ: `rabbitmq:3.13.7-management-alpine`
### ✅ Helm Configuration Management
Proper Helm chart structure with:
- Configurable values via `values.yaml`
- Template helpers for consistent naming
- Resource requests and limits
- Health checks and probes
- Service discovery
## Configuration
### Key Values
```yaml
# Image versions (pinned)
image:
repository: ghcr.io/mendableai/firecrawl
tag: "v1.0.0"
# Persistent storage
postgres.persistence:
enabled: true
storageClass: "nfs-emporium"
size: 20Gi
redis.persistence:
enabled: true
storageClass: "nfs-emporium"
size: 10Gi
rabbitmq.persistence:
enabled: true
storageClass: "nfs-emporium"
size: 5Gi
# Performance tuning
config:
NUM_WORKERS_PER_QUEUE: "8"
CRAWL_CONCURRENT_REQUESTS: "10"
MAX_CONCURRENT_JOBS: "5"
```
See `values.yaml` for all configuration options.
## Deployment
### ArgoCD (Recommended)
This application is managed by ArgoCD in wave 20 (applications tier):
```bash
# Check application status
kubectl -n argocd get application firecrawl
# Sync manually if needed
argocd app sync firecrawl
# View application details
argocd app get firecrawl
```
### Manual Helm Deployment
```bash
# From repository root
cd cluster/applications/firecrawl
# Install
helm install firecrawl ./chart -n firecrawl --create-namespace
# Upgrade
helm upgrade firecrawl ./chart -n firecrawl
# Uninstall
helm uninstall firecrawl -n firecrawl
```
## Access
### External Access (via Gateway API)
- **Primary**: https://spaceship-earth.local.mk-labs.cloud
- **Secondary**: https://firecrawl.local.mk-labs.cloud
DNS automatically configured via ExternalDNS pointing to Gateway LB (10.1.71.90).
### Internal Access
```
http://firecrawl-api.firecrawl.svc.cluster.local:3002
```
### API Usage Example
```bash
curl -X POST https://spaceship-earth.local.mk-labs.cloud/v0/scrape \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'
```
## Monitoring
### View Pods
```bash
kubectl -n firecrawl get pods
```
### Check Logs
```bash
# API logs
kubectl -n firecrawl logs -l app.kubernetes.io/component=api -f
# Worker logs
kubectl -n firecrawl logs -l app.kubernetes.io/component=worker -f
# PostgreSQL logs
kubectl -n firecrawl logs -l app.kubernetes.io/component=postgres -f
# Redis logs
kubectl -n firecrawl logs -l app.kubernetes.io/component=redis -f
```
### Check Storage
```bash
# View PVCs
kubectl -n firecrawl get pvc
# Check PVC details
kubectl -n firecrawl describe pvc firecrawl-postgres-pvc
kubectl -n firecrawl describe pvc firecrawl-redis-pvc
kubectl -n firecrawl describe pvc firecrawl-rabbitmq-pvc
```
## Troubleshooting
### Pods Not Starting
```bash
kubectl -n firecrawl describe pod <pod-name>
kubectl -n firecrawl get events --sort-by='.lastTimestamp'
```
### Storage Issues
```bash
# Check PVC binding
kubectl -n firecrawl get pvc
# Verify NFS storage class
kubectl get storageclass nfs-emporium -o yaml
# Check NFS CSI driver
kubectl -n kube-system get pods -l app=csi-nfs-controller
```
### Database Connection Issues
```bash
# Test PostgreSQL connectivity from API pod
kubectl -n firecrawl exec -it deployment/firecrawl-api -- \
sh -c 'apt-get update && apt-get install -y postgresql-client && \
psql -h firecrawl-postgres -U postgres -d postgres -c "SELECT 1"'
# Test Redis connectivity
kubectl -n firecrawl exec -it deployment/firecrawl-api -- \
sh -c 'apt-get update && apt-get install -y redis-tools && \
redis-cli -h firecrawl-redis PING'
```
### Gateway Routing Issues
```bash
# Verify ReferenceGrant
kubectl -n gateway get referencegrant allow-httproutes -o yaml
# Check HTTPRoutes
kubectl -n firecrawl get httproute
# View Gateway status
kubectl -n gateway get gateway fastpass-gateway -o yaml
```
## Upgrading
### Update Image Versions
1. Edit `values.yaml` and update image tags
2. Commit and push changes
3. ArgoCD will automatically sync
4. Or manually: `helm upgrade firecrawl ./chart -n firecrawl`
### Migrate from Old Deployment
The old raw manifest deployment will be automatically replaced by ArgoCD:
1. Commit this Helm chart
2. Push to repository
3. ArgoCD detects the change and applies Helm chart
4. Old resources are pruned
5. New resources with PVCs are created
**Note**: Old PostgreSQL/Redis data was ephemeral and will be lost. Fresh start with persistent storage.
## Storage Architecture
### NFS Provisioner
- **Storage Class**: `nfs-emporium`
- **Provisioner**: `nfs.csi.k8s.io`
- **NFS Server**: emporium (Synology DS1621+) - 10.1.71.9
- **NFS Export**: `/volume1/fastpass`
- **Reclaim Policy**: Delete
- **Binding Mode**: Immediate
### PVC Structure
Each PVC gets a unique subdirectory:
```
/volume1/fastpass/firecrawl/firecrawl-postgres-pvc/
/volume1/fastpass/firecrawl/firecrawl-redis-pvc/
/volume1/fastpass/firecrawl/firecrawl-rabbitmq-pvc/
```
## Theme
Part of the EPCOT-themed namespace with primary hostname **spaceship-earth** representing Firecrawl's role as the information and knowledge hub for JARVIS.
## Maintenance
### Backup Recommendations
```bash
# PostgreSQL backup (manual)
kubectl -n firecrawl exec deployment/firecrawl-postgres -- \
pg_dump -U postgres postgres > firecrawl-backup-$(date +%Y%m%d).sql
# NFS snapshots (via Synology)
# Use Synology Snapshot Replication for point-in-time backups
```
### Scaling
```bash
# Scale API replicas
helm upgrade firecrawl ./chart -n firecrawl \
--set api.replicaCount=2
# Scale workers
helm upgrade firecrawl ./chart -n firecrawl \
--set worker.replicaCount=3
```
## Future Enhancements
- [ ] Add Prometheus metrics and ServiceMonitor
- [ ] Implement resource quotas and LimitRanges
- [ ] Add PodDisruptionBudgets for HA
- [ ] Configure NetworkPolicies for security
- [ ] Add external secrets management (HashiCorp Vault or Sealed Secrets)
- [ ] Implement horizontal pod autoscaling
- [ ] Add extract worker deployment
- [ ] Add nuq worker deployment
## Support
For issues or questions:
- Check ArgoCD application status
- Review pod logs
- Verify storage bindings
- Consult Firecrawl documentation: https://docs.firecrawl.dev

View File

@@ -0,0 +1,49 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "firecrawl.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create a default fully qualified app name.
*/}}
{{- define "firecrawl.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "firecrawl.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Common labels
*/}}
{{- define "firecrawl.labels" -}}
helm.sh/chart: {{ include "firecrawl.chart" . }}
{{ include "firecrawl.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{/*
Selector labels
*/}}
{{- define "firecrawl.selectorLabels" -}}
app.kubernetes.io/name: {{ include "firecrawl.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

View File

@@ -0,0 +1,81 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "firecrawl.fullname" . }}-api
namespace: {{ .Values.namespace.name }}
labels:
{{- include "firecrawl.labels" . | nindent 4 }}
app.kubernetes.io/component: api
epcot.theme/service: spaceship-earth
spec:
replicas: {{ .Values.api.replicaCount }}
selector:
matchLabels:
{{- include "firecrawl.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: api
template:
metadata:
labels:
{{- include "firecrawl.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: api
epcot.theme/service: spaceship-earth
spec:
terminationGracePeriodSeconds: 180
containers:
- name: api
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
command: ["node"]
args: ["--max-old-space-size=6144", "dist/src/index.js"]
envFrom:
- configMapRef:
name: {{ include "firecrawl.fullname" . }}-config
env:
- name: FLY_PROCESS_GROUP
value: "app"
- name: NUQ_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
ports:
- containerPort: 3002
name: http
resources:
{{- toYaml .Values.api.resources | nindent 12 }}
livenessProbe:
httpGet:
path: /v0/health/liveness
port: 3002
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /v0/health/readiness
port: 3002
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 3
---
apiVersion: v1
kind: Service
metadata:
name: {{ include "firecrawl.fullname" . }}-api
namespace: {{ .Values.namespace.name }}
labels:
{{- include "firecrawl.labels" . | nindent 4 }}
app.kubernetes.io/component: api
spec:
type: {{ .Values.api.service.type }}
ports:
- port: {{ .Values.api.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "firecrawl.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: api

View File

@@ -0,0 +1,89 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "firecrawl.fullname" . }}-config
namespace: {{ .Values.namespace.name }}
labels:
{{- include "firecrawl.labels" . | nindent 4 }}
data:
# Environment
ENV: {{ .Values.config.ENV | quote }}
IS_KUBERNETES: {{ .Values.config.IS_KUBERNETES | quote }}
LOGGING_LEVEL: {{ .Values.config.LOGGING_LEVEL | quote }}
# Service configuration
PORT: {{ .Values.config.PORT | quote }}
HOST: {{ .Values.config.HOST | quote }}
WORKER_PORT: {{ .Values.config.WORKER_PORT | quote }}
# Performance tuning
NUM_WORKERS_PER_QUEUE: {{ .Values.config.NUM_WORKERS_PER_QUEUE | quote }}
CRAWL_CONCURRENT_REQUESTS: {{ .Values.config.CRAWL_CONCURRENT_REQUESTS | quote }}
MAX_CONCURRENT_JOBS: {{ .Values.config.MAX_CONCURRENT_JOBS | quote }}
# Authentication
USE_DB_AUTHENTICATION: {{ .Values.config.USE_DB_AUTHENTICATION | quote }}
# Service URLs
REDIS_URL: {{ .Values.config.REDIS_URL | default (printf "redis://%s-redis:%d" (include "firecrawl.fullname" .) (.Values.redis.service.port | int)) | quote }}
REDIS_RATE_LIMIT_URL: {{ .Values.config.REDIS_RATE_LIMIT_URL | default (printf "redis://%s-redis:%d" (include "firecrawl.fullname" .) (.Values.redis.service.port | int)) | quote }}
PLAYWRIGHT_MICROSERVICE_URL: {{ .Values.config.PLAYWRIGHT_MICROSERVICE_URL | default (printf "http://%s-playwright:%d/scrape" (include "firecrawl.fullname" .) (.Values.playwrightService.service.port | int)) | quote }}
# Database configuration
POSTGRES_HOST: {{ printf "%s-postgres" (include "firecrawl.fullname" .) | quote }}
POSTGRES_PORT: {{ .Values.postgres.service.port | quote }}
POSTGRES_USER: {{ .Values.postgres.auth.username | quote }}
POSTGRES_PASSWORD: {{ .Values.postgres.auth.password | quote }}
POSTGRES_DB: {{ .Values.postgres.auth.database | quote }}
# RabbitMQ configuration
NUQ_RABBITMQ_URL: {{ printf "amqp://%s-rabbitmq:%d" (include "firecrawl.fullname" .) (.Values.rabbitmq.service.amqpPort | int) | quote }}
# Optional external services
{{- if .Values.config.OPENAI_BASE_URL }}
OPENAI_BASE_URL: {{ .Values.config.OPENAI_BASE_URL | quote }}
{{- end }}
{{- if .Values.config.MODEL_NAME }}
MODEL_NAME: {{ .Values.config.MODEL_NAME | quote }}
{{- end }}
{{- if .Values.config.MODEL_EMBEDDING_NAME }}
MODEL_EMBEDDING_NAME: {{ .Values.config.MODEL_EMBEDDING_NAME | quote }}
{{- end }}
{{- if .Values.config.OLLAMA_BASE_URL }}
OLLAMA_BASE_URL: {{ .Values.config.OLLAMA_BASE_URL | quote }}
{{- end }}
{{- if .Values.config.PROXY_SERVER }}
PROXY_SERVER: {{ .Values.config.PROXY_SERVER | quote }}
{{- end }}
{{- if .Values.config.PROXY_USERNAME }}
PROXY_USERNAME: {{ .Values.config.PROXY_USERNAME | quote }}
{{- end }}
{{- if .Values.config.SEARXNG_ENDPOINT }}
SEARXNG_ENDPOINT: {{ .Values.config.SEARXNG_ENDPOINT | quote }}
{{- end }}
{{- if .Values.config.SEARXNG_ENGINES }}
SEARXNG_ENGINES: {{ .Values.config.SEARXNG_ENGINES | quote }}
{{- end }}
{{- if .Values.config.SEARXNG_CATEGORIES }}
SEARXNG_CATEGORIES: {{ .Values.config.SEARXNG_CATEGORIES | quote }}
{{- end }}
{{- if .Values.config.SELF_HOSTED_WEBHOOK_URL }}
SELF_HOSTED_WEBHOOK_URL: {{ .Values.config.SELF_HOSTED_WEBHOOK_URL | quote }}
{{- end }}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "firecrawl.fullname" . }}-playwright-config
namespace: {{ .Values.namespace.name }}
labels:
{{- include "firecrawl.labels" . | nindent 4 }}
data:
PORT: {{ .Values.playwrightConfig.PORT | quote }}
MAX_CONCURRENT_PAGES: {{ .Values.playwrightConfig.MAX_CONCURRENT_PAGES | quote }}
{{- if .Values.playwrightConfig.ALLOW_LOCAL_WEBHOOKS }}
ALLOW_LOCAL_WEBHOOKS: {{ .Values.playwrightConfig.ALLOW_LOCAL_WEBHOOKS | quote }}
{{- end }}
{{- if .Values.playwrightConfig.BLOCK_MEDIA }}
BLOCK_MEDIA: {{ .Values.playwrightConfig.BLOCK_MEDIA | quote }}
{{- end }}

View File

@@ -0,0 +1,87 @@
# ------------------------------------------------------------------------------
# HTTPRoute — Firecrawl via Cilium Gateway
# Both primary (spaceship-earth) and secondary (firecrawl) DNS names
# ------------------------------------------------------------------------------
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: spaceship-earth-tls
namespace: {{ .Values.namespace.name }}
spec:
secretName: spaceship-earth-tls
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
dnsNames:
- spaceship-earth.local.mk-labs.cloud
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: firecrawl-tls
namespace: {{ .Values.namespace.name }}
spec:
secretName: firecrawl-tls
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
dnsNames:
- firecrawl.local.mk-labs.cloud
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: spaceship-earth
namespace: {{ .Values.namespace.name }}
annotations:
external-dns.alpha.kubernetes.io/hostname: spaceship-earth.local.mk-labs.cloud
external-dns.alpha.kubernetes.io/target: "10.1.71.90"
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: fastpass-gateway
namespace: gateway
sectionName: https
hostnames:
- spaceship-earth.local.mk-labs.cloud
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: ""
kind: Service
name: {{ include "firecrawl.fullname" . }}-api
port: {{ .Values.api.service.port }}
weight: 1
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: firecrawl
namespace: {{ .Values.namespace.name }}
annotations:
external-dns.alpha.kubernetes.io/hostname: firecrawl.local.mk-labs.cloud
external-dns.alpha.kubernetes.io/target: "10.1.71.90"
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: fastpass-gateway
namespace: gateway
sectionName: https
hostnames:
- firecrawl.local.mk-labs.cloud
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: ""
kind: Service
name: {{ include "firecrawl.fullname" . }}-api
port: {{ .Values.api.service.port }}
weight: 1

View File

@@ -0,0 +1,6 @@
apiVersion: v1
kind: Namespace
metadata:
name: {{ .Values.namespace.name }}
labels:
{{- toYaml .Values.namespace.labels | nindent 4 }}

View File

@@ -0,0 +1,58 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "firecrawl.fullname" . }}-playwright
namespace: {{ .Values.namespace.name }}
labels:
{{- include "firecrawl.labels" . | nindent 4 }}
app.kubernetes.io/component: playwright
spec:
replicas: {{ .Values.playwrightService.replicaCount }}
selector:
matchLabels:
{{- include "firecrawl.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: playwright
template:
metadata:
labels:
{{- include "firecrawl.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: playwright
spec:
containers:
- name: playwright
image: "{{ .Values.playwright.repository }}:{{ .Values.playwright.tag }}"
imagePullPolicy: {{ .Values.playwright.pullPolicy }}
envFrom:
- configMapRef:
name: {{ include "firecrawl.fullname" . }}-playwright-config
ports:
- containerPort: 3000
name: http
resources:
{{- toYaml .Values.playwrightService.resources | nindent 12 }}
volumeMounts:
- name: cache
mountPath: /tmp/.cache
volumes:
- name: cache
emptyDir:
sizeLimit: 1Gi
---
apiVersion: v1
kind: Service
metadata:
name: {{ include "firecrawl.fullname" . }}-playwright
namespace: {{ .Values.namespace.name }}
labels:
{{- include "firecrawl.labels" . | nindent 4 }}
app.kubernetes.io/component: playwright
spec:
type: {{ .Values.playwrightService.service.type }}
ports:
- port: {{ .Values.playwrightService.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "firecrawl.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: playwright

View File

@@ -0,0 +1,67 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "firecrawl.fullname" . }}-postgres
namespace: {{ .Values.namespace.name }}
labels:
{{- include "firecrawl.labels" . | nindent 4 }}
app.kubernetes.io/component: postgres
spec:
replicas: {{ .Values.postgres.replicaCount }}
selector:
matchLabels:
{{- include "firecrawl.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: postgres
template:
metadata:
labels:
{{- include "firecrawl.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: postgres
spec:
containers:
- name: postgres
image: "{{ .Values.postgres.image.repository }}:{{ .Values.postgres.image.tag }}"
imagePullPolicy: {{ .Values.postgres.image.pullPolicy }}
env:
- name: POSTGRES_USER
value: {{ .Values.postgres.auth.username | quote }}
- name: POSTGRES_PASSWORD
value: {{ .Values.postgres.auth.password | quote }}
- name: POSTGRES_DB
value: {{ .Values.postgres.auth.database | quote }}
ports:
- containerPort: 5432
name: postgres
resources:
{{- toYaml .Values.postgres.resources | nindent 12 }}
volumeMounts:
- name: postgres-data
mountPath: /var/lib/postgresql/data
subPath: postgres
volumes:
- name: postgres-data
{{- if .Values.postgres.persistence.enabled }}
persistentVolumeClaim:
claimName: {{ include "firecrawl.fullname" . }}-postgres-pvc
{{- else }}
emptyDir: {}
{{- end }}
---
apiVersion: v1
kind: Service
metadata:
name: {{ include "firecrawl.fullname" . }}-postgres
namespace: {{ .Values.namespace.name }}
labels:
{{- include "firecrawl.labels" . | nindent 4 }}
app.kubernetes.io/component: postgres
spec:
type: {{ .Values.postgres.service.type }}
ports:
- port: {{ .Values.postgres.service.port }}
targetPort: postgres
protocol: TCP
name: postgres
selector:
{{- include "firecrawl.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: postgres

View File

@@ -0,0 +1,53 @@
{{- if .Values.postgres.persistence.enabled }}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: {{ include "firecrawl.fullname" . }}-postgres-pvc
namespace: {{ .Values.namespace.name }}
labels:
{{- include "firecrawl.labels" . | nindent 4 }}
app.kubernetes.io/component: postgres
spec:
accessModes:
- {{ .Values.postgres.persistence.accessMode }}
storageClassName: {{ .Values.postgres.persistence.storageClass | quote }}
resources:
requests:
storage: {{ .Values.postgres.persistence.size | quote }}
{{- end }}
---
{{- if .Values.redis.persistence.enabled }}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: {{ include "firecrawl.fullname" . }}-redis-pvc
namespace: {{ .Values.namespace.name }}
labels:
{{- include "firecrawl.labels" . | nindent 4 }}
app.kubernetes.io/component: redis
spec:
accessModes:
- {{ .Values.redis.persistence.accessMode }}
storageClassName: {{ .Values.redis.persistence.storageClass | quote }}
resources:
requests:
storage: {{ .Values.redis.persistence.size | quote }}
{{- end }}
---
{{- if and .Values.rabbitmq.enabled .Values.rabbitmq.persistence.enabled }}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: {{ include "firecrawl.fullname" . }}-rabbitmq-pvc
namespace: {{ .Values.namespace.name }}
labels:
{{- include "firecrawl.labels" . | nindent 4 }}
app.kubernetes.io/component: rabbitmq
spec:
accessModes:
- {{ .Values.rabbitmq.persistence.accessMode }}
storageClassName: {{ .Values.rabbitmq.persistence.storageClass | quote }}
resources:
requests:
storage: {{ .Values.rabbitmq.persistence.size | quote }}
{{- end }}

View File

@@ -0,0 +1,82 @@
{{- if .Values.rabbitmq.enabled }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "firecrawl.fullname" . }}-rabbitmq
namespace: {{ .Values.namespace.name }}
labels:
{{- include "firecrawl.labels" . | nindent 4 }}
app.kubernetes.io/component: rabbitmq
spec:
replicas: {{ .Values.rabbitmq.replicaCount }}
selector:
matchLabels:
{{- include "firecrawl.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: rabbitmq
template:
metadata:
labels:
{{- include "firecrawl.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: rabbitmq
spec:
containers:
- name: rabbitmq
image: "{{ .Values.rabbitmq.image.repository }}:{{ .Values.rabbitmq.image.tag }}"
imagePullPolicy: {{ .Values.rabbitmq.image.pullPolicy }}
command: ["rabbitmq-server"]
ports:
- containerPort: 5672
name: amqp
- containerPort: 15672
name: management
resources:
{{- toYaml .Values.rabbitmq.resources | nindent 12 }}
livenessProbe:
exec:
command: ["rabbitmq-diagnostics", "-q", "check_running"]
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
exec:
command: ["rabbitmq-diagnostics", "-q", "check_running"]
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 5
failureThreshold: 3
volumeMounts:
{{- if .Values.rabbitmq.persistence.enabled }}
- name: rabbitmq-data
mountPath: /var/lib/rabbitmq
{{- end }}
volumes:
{{- if .Values.rabbitmq.persistence.enabled }}
- name: rabbitmq-data
persistentVolumeClaim:
claimName: {{ include "firecrawl.fullname" . }}-rabbitmq-pvc
{{- end }}
---
apiVersion: v1
kind: Service
metadata:
name: {{ include "firecrawl.fullname" . }}-rabbitmq
namespace: {{ .Values.namespace.name }}
labels:
{{- include "firecrawl.labels" . | nindent 4 }}
app.kubernetes.io/component: rabbitmq
spec:
type: {{ .Values.rabbitmq.service.type }}
ports:
- port: {{ .Values.rabbitmq.service.amqpPort }}
targetPort: amqp
protocol: TCP
name: amqp
- port: {{ .Values.rabbitmq.service.managementPort }}
targetPort: management
protocol: TCP
name: management
selector:
{{- include "firecrawl.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: rabbitmq
{{- end }}

View File

@@ -0,0 +1,73 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "firecrawl.fullname" . }}-redis
namespace: {{ .Values.namespace.name }}
labels:
{{- include "firecrawl.labels" . | nindent 4 }}
app.kubernetes.io/component: redis
spec:
replicas: {{ .Values.redis.replicaCount }}
selector:
matchLabels:
{{- include "firecrawl.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: redis
template:
metadata:
labels:
{{- include "firecrawl.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: redis
spec:
containers:
- name: redis
image: "{{ .Values.redis.image.repository }}:{{ .Values.redis.image.tag }}"
imagePullPolicy: {{ .Values.redis.image.pullPolicy }}
command: ["redis-server"]
args:
- "--bind"
- "0.0.0.0"
{{- if .Values.redis.persistence.enabled }}
- "--appendonly"
- "yes"
- "--save"
- "900 1"
- "--save"
- "300 10"
- "--save"
- "60 10000"
{{- end }}
ports:
- containerPort: 6379
name: redis
resources:
{{- toYaml .Values.redis.resources | nindent 12 }}
volumeMounts:
{{- if .Values.redis.persistence.enabled }}
- name: redis-data
mountPath: /data
{{- end }}
volumes:
{{- if .Values.redis.persistence.enabled }}
- name: redis-data
persistentVolumeClaim:
claimName: {{ include "firecrawl.fullname" . }}-redis-pvc
{{- end }}
---
apiVersion: v1
kind: Service
metadata:
name: {{ include "firecrawl.fullname" . }}-redis
namespace: {{ .Values.namespace.name }}
labels:
{{- include "firecrawl.labels" . | nindent 4 }}
app.kubernetes.io/component: redis
spec:
type: {{ .Values.redis.service.type }}
ports:
- port: {{ .Values.redis.service.port }}
targetPort: redis
protocol: TCP
name: redis
selector:
{{- include "firecrawl.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: redis

View File

@@ -0,0 +1,39 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "firecrawl.fullname" . }}-worker
namespace: {{ .Values.namespace.name }}
labels:
{{- include "firecrawl.labels" . | nindent 4 }}
app.kubernetes.io/component: worker
spec:
replicas: {{ .Values.worker.replicaCount }}
selector:
matchLabels:
{{- include "firecrawl.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: worker
template:
metadata:
labels:
{{- include "firecrawl.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: worker
spec:
terminationGracePeriodSeconds: 180
containers:
- name: worker
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
command: ["node"]
args: ["--max-old-space-size=4096", "dist/src/services/queue-worker.js"]
envFrom:
- configMapRef:
name: {{ include "firecrawl.fullname" . }}-config
env:
- name: FLY_PROCESS_GROUP
value: "worker"
- name: NUQ_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
resources:
{{- toYaml .Values.worker.resources | nindent 12 }}

View File

@@ -0,0 +1,240 @@
# ==============================================================================
# Firecrawl Helm Chart Values
# Production-ready configuration for mk-labs homelab
# ==============================================================================
# ------------------------------------------------------------------------------
# Image Configuration
# ------------------------------------------------------------------------------
image:
# Main Firecrawl API and Worker image
repository: ghcr.io/mendableai/firecrawl
tag: "v1.0.0" # Pinned version - update explicitly
pullPolicy: IfNotPresent
playwright:
repository: ghcr.io/mendableai/firecrawl/playwright-service
tag: "v1.0.0" # Pinned version - update explicitly
pullPolicy: IfNotPresent
# ------------------------------------------------------------------------------
# API Service Configuration
# ------------------------------------------------------------------------------
api:
replicaCount: 1
resources:
requests:
memory: "4Gi"
cpu: "2000m"
limits:
memory: "6Gi"
cpu: "2000m"
service:
type: ClusterIP
port: 3002
# ------------------------------------------------------------------------------
# Worker Service Configuration
# ------------------------------------------------------------------------------
worker:
replicaCount: 1
resources:
requests:
memory: "3Gi"
cpu: "1000m"
limits:
memory: "4Gi"
cpu: "1000m"
# ------------------------------------------------------------------------------
# Playwright Service Configuration
# ------------------------------------------------------------------------------
playwrightService:
replicaCount: 1
resources:
requests:
memory: "2Gi"
cpu: "1000m"
limits:
memory: "4Gi"
cpu: "2000m"
service:
type: ClusterIP
port: 3000
# ------------------------------------------------------------------------------
# PostgreSQL Configuration (nuq-postgres)
# ------------------------------------------------------------------------------
postgres:
image:
repository: ghcr.io/mendableai/firecrawl/nuq-postgres
tag: "v1.0.0" # Pinned version
pullPolicy: IfNotPresent
replicaCount: 1
auth:
username: postgres
password: postgres
database: postgres
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "500m"
# Persistent storage configuration
persistence:
enabled: true
storageClass: "nfs-emporium"
accessMode: ReadWriteOnce
size: 20Gi
service:
type: ClusterIP
port: 5432
# ------------------------------------------------------------------------------
# Redis Configuration
# ------------------------------------------------------------------------------
redis:
image:
repository: redis
tag: "7.4.1-alpine" # Pinned stable version
pullPolicy: IfNotPresent
replicaCount: 1
resources:
requests:
memory: "256Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "500m"
# Persistent storage configuration
persistence:
enabled: true
storageClass: "nfs-emporium"
accessMode: ReadWriteOnce
size: 10Gi
service:
type: ClusterIP
port: 6379
# ------------------------------------------------------------------------------
# RabbitMQ Configuration
# ------------------------------------------------------------------------------
rabbitmq:
enabled: true
image:
repository: rabbitmq
tag: "3.13.7-management-alpine" # Pinned stable version
pullPolicy: IfNotPresent
replicaCount: 1
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "500m"
# Persistent storage configuration
persistence:
enabled: true
storageClass: "nfs-emporium"
accessMode: ReadWriteOnce
size: 5Gi
service:
type: ClusterIP
amqpPort: 5672
managementPort: 15672
# ------------------------------------------------------------------------------
# Application Configuration
# ------------------------------------------------------------------------------
config:
# Environment
ENV: "production"
IS_KUBERNETES: "true"
LOGGING_LEVEL: "INFO"
# Service configuration
PORT: "3002"
HOST: "0.0.0.0"
WORKER_PORT: "3005"
# Performance tuning
NUM_WORKERS_PER_QUEUE: "8"
CRAWL_CONCURRENT_REQUESTS: "10"
MAX_CONCURRENT_JOBS: "5"
# Authentication
USE_DB_AUTHENTICATION: "false"
# URLs (auto-generated if empty)
REDIS_URL: ""
REDIS_RATE_LIMIT_URL: ""
PLAYWRIGHT_MICROSERVICE_URL: ""
# Optional external services
OPENAI_BASE_URL: ""
MODEL_NAME: ""
MODEL_EMBEDDING_NAME: ""
OLLAMA_BASE_URL: ""
PROXY_SERVER: ""
PROXY_USERNAME: ""
SEARXNG_ENDPOINT: ""
SEARXNG_ENGINES: ""
SEARXNG_CATEGORIES: ""
SELF_HOSTED_WEBHOOK_URL: ""
# ------------------------------------------------------------------------------
# Playwright Service Configuration
# ------------------------------------------------------------------------------
playwrightConfig:
PORT: "3000"
ALLOW_LOCAL_WEBHOOKS: ""
BLOCK_MEDIA: ""
MAX_CONCURRENT_PAGES: "10"
# ------------------------------------------------------------------------------
# Secrets (Optional - use ExternalSecrets in production)
# ------------------------------------------------------------------------------
secrets:
OPENAI_API_KEY: ""
SLACK_WEBHOOK_URL: ""
LLAMAPARSE_API_KEY: ""
BULL_AUTH_KEY: ""
TEST_API_KEY: ""
FIRE_ENGINE_BETA_URL: ""
SUPABASE_ANON_TOKEN: ""
SUPABASE_URL: ""
SUPABASE_SERVICE_TOKEN: ""
PROXY_PASSWORD: ""
SELF_HOSTED_WEBHOOK_HMAC_SECRET: ""
# ------------------------------------------------------------------------------
# Namespace Configuration
# ------------------------------------------------------------------------------
namespace:
name: firecrawl
labels:
app.kubernetes.io/name: firecrawl
app.kubernetes.io/part-of: mk-labs
epcot.theme/name: spaceship-earth