homelab/cluster/applications/firecrawl/README.md

# Firecrawl - Web Scraping & Crawling Service

**Project:** Platform Buildout - Firecrawl Deployment  
**Service Name:** Spaceship Earth (EPCOT themed)  
**DNS:** spaceship-earth.local.mk-labs.cloud (primary), firecrawl.local.mk-labs.cloud (alias)  
**Namespace:** firecrawl  
**Owner:** Rocket Raccoon (CI/CD Specialist)  
**Status:** 🚧 IN PROGRESS - Day 1 Complete

---

## Overview

Firecrawl is a self-hosted web scraping and crawling API that converts URLs to LLM-ready content (Markdown, JSON, HTML). This deployment enables JARVIS web search capability.

**Upstream:** https://github.com/mendableai/firecrawl  
**License:** AGPLv3 (open source)

### Capabilities
- **Search:** Search the web and get full page content
- **Scrape:** Convert URLs to markdown, HTML, screenshots, or structured JSON
- **Crawl:** Scrape all URLs of a website with a single request
- **Interact:** Click, scroll, write, wait before extracting (JS-heavy sites)
- **Map:** Discover all URLs on a website

---

## Architecture

### Services (5 components)

| Service | Image | CPU | Memory | Storage | Purpose |
|---------|-------|-----|--------|---------|---------|
| **API** | Harbor: firecrawl-api:latest | 2.0 | 4-6GB | - | Main REST API |
| **Worker** | Harbor: firecrawl-api:latest | 1.0 | 3-4GB | - | Background job processor |
| **NUQ Worker** | Harbor: firecrawl-api:latest | 1.0 | 3-4GB | - | Database queue worker |
| **Playwright** | Harbor: firecrawl-playwright:latest | 2.0 | 4GB | 1GB tmpfs | Browser automation |
| **PostgreSQL** | Harbor: firecrawl-postgres:latest | 1.0 | 2GB | 10GB PVC | Data storage |
| **Redis** | Upstream: redis:alpine | 0.5 | 1GB | - | Queue & cache |
| **RabbitMQ** | Upstream: rabbitmq:3-management | 0.5 | 1GB | - | Message broker |

**Total Resources:** ~8 CPU, ~22GB RAM, 11GB storage

### Service Dependencies

```
API Service ─┬─► Redis (queue/cache)
             ├─► PostgreSQL (data storage)
             ├─► RabbitMQ (message broker) ⚠️ HEALTH CHECK REQUIRED
             └─► Playwright Service (browser automation)

Worker ──────┬─► Redis
             ├─► PostgreSQL
             └─► RabbitMQ

NUQ Worker ──┴─► PostgreSQL
```

**Startup Order:**
1. Redis, PostgreSQL, RabbitMQ (infrastructure)
2. Playwright Service
3. API, Workers (after all dependencies ready)

---

## Build Strategy

**Hybrid Approach:** Build custom images via Tekton, use upstream for infrastructure.

### Custom Builds (Tekton → Harbor)

1. **firecrawl-api** (Multi-stage: Go + Node.js + Rust)
   - Source: `apps/api/Dockerfile`
   - Registry: `the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest`
   - Build time: ~15 minutes (first), ~5 minutes (cached)
   - Used by: API, Worker, NUQ Worker (different entrypoints)

2. **firecrawl-playwright** (Node.js + Chromium)
   - Source: `apps/playwright-service-ts/Dockerfile`
   - Registry: `the-seas.local.mk-labs.cloud/applications/firecrawl-playwright:latest`
   - Build time: ~10 minutes

3. **firecrawl-postgres** (PostgreSQL + pg_cron + init script)
   - Source: `apps/nuq-postgres/Dockerfile`
   - Registry: `the-seas.local.mk-labs.cloud/applications/firecrawl-postgres:latest`
   - Build time: ~3 minutes
   - Note: Custom build required for pg_cron extension and nuq.sql schema

### Upstream Images

- **Redis:** `redis:alpine`
- **RabbitMQ:** `rabbitmq:3-management`

---

## Configuration

### Required Secrets (1Password)

Managed via ExternalSecret → 1Password vault item: `firecrawl`

- `POSTGRES_PASSWORD` - Database password (CRITICAL)
- `BULL_AUTH_KEY` - Queue admin UI authentication (CRITICAL)
- `OPENAI_API_KEY` - Optional, for AI features
- `TEST_API_KEY` - Optional, for testing

### ConfigMap (Non-sensitive)

- Service URLs (Redis, PostgreSQL, RabbitMQ, Playwright)
- Port configuration (3002 API, 3005 Worker)
- Performance tuning (worker pools, concurrency limits)
- Logging level

### Environment Variables Reference

See [ENVIRONMENT_VARIABLES.md](./ENVIRONMENT_VARIABLES.md) for complete list.

---

## Deployment Timeline

### Day 1 (June 6) - Investigation & Planning ✅
- Repository analysis complete
- Architecture decisions finalized
- Environment configuration researched
- Day 2 plan created

### Day 2 (June 7) - Tekton Pipelines 🚧
- Create 3 build pipelines (API, Playwright, PostgreSQL)
- Test builds and push to Harbor
- Validate image integrity

### Day 3 (June 8) - Kubernetes Manifests
- Create Deployments, Services, StatefulSets
- Configure ConfigMaps and ExternalSecrets
- Set up HTTPRoute for ingress

### Day 4 (June 9) - Secrets & Configuration
- Create 1Password vault item
- Configure ExternalSecret sync
- Validate configuration

### Day 5 (June 10) - Deployment & Testing
- ArgoCD Application creation
- Deploy to cluster
- Service health validation
- API functionality testing

### Day 6 (June 11) - JARVIS Integration
- Configure JARVIS environment variables
- Test web search functionality
- End-to-end validation
- Documentation delivery

**Target Completion:** June 12, 2026

---

## Access & URLs

**Primary Access:**
- API: https://spaceship-earth.local.mk-labs.cloud
- Alias: https://firecrawl.local.mk-labs.cloud
- Queue UI: https://spaceship-earth.local.mk-labs.cloud/admin/[BULL_AUTH_KEY]/queues

**Health Endpoints:**
- Liveness: https://spaceship-earth.local.mk-labs.cloud/v0/health/liveness
- Readiness: https://spaceship-earth.local.mk-labs.cloud/v0/health/readiness

**Internal Services (cluster-only):**
- Playwright: http://playwright-service.firecrawl.svc:3000
- PostgreSQL: postgresql://nuq-postgres.firecrawl.svc:5432
- Redis: redis://redis.firecrawl.svc:6379
- RabbitMQ: amqp://rabbitmq.firecrawl.svc:5672
- RabbitMQ Mgmt: http://rabbitmq.firecrawl.svc:15672

---

## API Usage Examples

### Scrape a URL
```bash
curl -X POST https://spaceship-earth.local.mk-labs.cloud/v1/scrape \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com"
  }'
```

### Search the Web
```bash
curl -X POST https://spaceship-earth.local.mk-labs.cloud/v1/search \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "kubernetes best practices"
  }'
```

### Crawl a Website
```bash
curl -X POST https://spaceship-earth.local.mk-labs.cloud/v1/crawl \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://docs.example.com"
  }'
```

---

## Directory Structure

```
cluster/applications/firecrawl/
├── README.md                      # This file
├── ENVIRONMENT_VARIABLES.md       # Complete env var reference
├── namespace.yaml                 # Namespace definition
├── configmap.yaml                 # Non-sensitive configuration
├── externalsecret.yaml            # 1Password secret sync
├── postgresql/
│   ├── statefulset.yaml          # PostgreSQL StatefulSet
│   ├── service.yaml              # PostgreSQL Service
│   └── pvc.yaml                  # Persistent Volume Claim
├── redis/
│   ├── deployment.yaml           # Redis Deployment
│   └── service.yaml              # Redis Service
├── rabbitmq/
│   ├── deployment.yaml           # RabbitMQ Deployment
│   └── service.yaml              # RabbitMQ Service
├── playwright/
│   ├── deployment.yaml           # Playwright Deployment
│   └── service.yaml              # Playwright Service
├── api/
│   ├── deployment.yaml           # API Deployment
│   └── service.yaml              # API Service
├── workers/
│   ├── worker-deployment.yaml    # Queue Worker Deployment
│   └── nuq-worker-deployment.yaml # NUQ Worker Deployment
├── ingress/
│   └── httproute.yaml            # Gateway API HTTPRoute
└── argocd/
    └── application.yaml          # ArgoCD Application manifest
```

---

## Tekton Pipelines

Build pipelines located in `cluster/tekton/pipelines/`:
- `firecrawl-api-build.yaml` - API service build
- `firecrawl-playwright-build.yaml` - Playwright service build
- `firecrawl-postgres-build.yaml` - PostgreSQL build

**Trigger Builds:**
```bash
# API build
kubectl create -f cluster/tekton/pipelines/firecrawl-api-build.yaml

# Playwright build
kubectl create -f cluster/tekton/pipelines/firecrawl-playwright-build.yaml

# PostgreSQL build
kubectl create -f cluster/tekton/pipelines/firecrawl-postgres-build.yaml
```

**Monitor Builds:**
```bash
# List pipeline runs
tkn pipelinerun list -n innoventions

# Watch logs
tkn pipelinerun logs -f <pipelinerun-name> -n innoventions
```

---

## Troubleshooting

### API Pod Not Starting
**Check:**
1. RabbitMQ health status (API depends on healthy RabbitMQ)
2. Environment variables (ConfigMap and Secret)
3. Database connectivity (PostgreSQL)

**Commands:**
```bash
kubectl logs -n firecrawl deployment/api
kubectl describe pod -n firecrawl -l app=api
kubectl get externalsecret -n firecrawl
```

### Build Failures
**Check:**
1. Harbor connectivity
2. Harbor credentials secret
3. Build resource limits (increase if OOM)

**Commands:**
```bash
tkn pipelinerun describe <name> -n innoventions
kubectl logs -n innoventions <kaniko-pod>
```

### Database Connection Errors
**Check:**
1. PostgreSQL pod status
2. PVC binding
3. Init script execution

**Commands:**
```bash
kubectl logs -n firecrawl statefulset/nuq-postgres
kubectl exec -it -n firecrawl nuq-postgres-0 -- psql -U postgres -d postgres -c '\d nuq.queue_scrape'
```

---

## Monitoring

**Resource Usage:**
```bash
kubectl top pods -n firecrawl
```

**Service Health:**
```bash
# API liveness
curl -k https://spaceship-earth.local.mk-labs.cloud/v0/health/liveness

# API readiness
curl -k https://spaceship-earth.local.mk-labs.cloud/v0/health/readiness

# Playwright health
kubectl exec -n firecrawl deployment/playwright-service -- curl localhost:3000/health
```

**Queue Status:**
Navigate to: https://spaceship-earth.local.mk-labs.cloud/admin/[BULL_AUTH_KEY]/queues

---

## Security Notes

1. **PostgreSQL Credentials:** Stored in 1Password, synced via ExternalSecret
2. **Admin UI:** Protected by BULL_AUTH_KEY (in URL path)
3. **Database Port:** NOT exposed outside cluster (ClusterIP only)
4. **TLS:** All external traffic encrypted via cert-manager certificates
5. **RBAC:** Service accounts scoped to firecrawl namespace

---

## References

- **Upstream Docs:** https://docs.firecrawl.dev
- **GitHub:** https://github.com/mendableai/firecrawl
- **Self-Hosting Guide:** https://github.com/mendableai/firecrawl/blob/main/SELF_HOST.md
- **K8s Examples:** `/tmp/firecrawl/examples/kubernetes/`
- **Mission Brief:** `~/friday/inbox/agents/rocket/FIRECRAWL-DEPLOYMENT-MISSION-BRIEF.md`

---

## Changelog

### 2026-06-06 - Day 1 Complete
- Initial repository structure created
- Architecture decisions finalized
- Build strategy documented
- Environment variables researched
- Ready for Day 2 (pipeline creation)

---

**Contact:** Rocket Raccoon (CI/CD Specialist)  
**Project Manager:** Pepper Potts  
**Cluster:** fastpass (Talos Kubernetes)  
**Last Updated:** June 6, 2026