# Firecrawl (Spaceship Earth)

Web scraping and search service for JARVIS's web search capabilities.

## Service Names

- **Primary (EPCOT):** `spaceship-earth.local.mk-labs.cloud`
- **Secondary:** `firecrawl.local.mk-labs.cloud`

## Architecture

### Components

1. **Firecrawl API** - Main REST API service for web scraping
   - Port: 3002
   - Health checks: `/v0/health/liveness`, `/v0/health/readiness`
   - Resources: 4Gi memory, 2 CPU cores (6Gi limit)

2. **Firecrawl Worker** - Background job processor
   - Processes crawl jobs from queue
   - Resources: 3Gi memory, 1 CPU core (4Gi limit)

3. **Playwright Service** - Browser automation
   - Port: 3000
   - Headless Chrome/Firefox for JavaScript rendering
   - Resources: 2Gi memory, 1 CPU core (4Gi limit)

4. **Redis** - Cache and job queue
   - Port: 6379
   - Resources: 256Mi memory, 100m CPU (512Mi limit)

5. **PostgreSQL (nuq-postgres)** - State management
   - Port: 5432
   - Custom image with queue extensions
   - Resources: 512Mi memory, 250m CPU (1Gi limit)
   - **Note:** Currently using emptyDir (ephemeral). Add PVC for persistence if needed.

6. **RabbitMQ** - Message queue for distributed processing
   - AMQP Port: 5672
   - Management UI: 15672
   - Resources: 512Mi memory, 250m CPU (1Gi limit)

### Configuration

Key environment variables in `configmap.yaml`:

- `USE_DB_AUTHENTICATION=false` - Simplified auth for internal use
- `NUM_WORKERS_PER_QUEUE=8` - Worker concurrency
- `CRAWL_CONCURRENT_REQUESTS=10` - Parallel crawl requests
- `MAX_CONCURRENT_JOBS=5` - Maximum simultaneous jobs

## Access

- **HTTPS (Gateway API):** https://spaceship-earth.local.mk-labs.cloud
- **HTTPS (Gateway API):** https://firecrawl.local.mk-labs.cloud
- Internal: `http://firecrawl-api.firecrawl.svc.cluster.local:3002`

## DNS

ExternalDNS automatically configures DNS records pointing to the Gateway load balancer (10.1.71.90).

## TLS

Certificates are automatically provisioned via cert-manager using the `letsencrypt-prod` ClusterIssuer.

## Deployment

Managed by ArgoCD (wave 20):

```bash
# Check application status
kubectl -n argocd get application firecrawl

# View pods
kubectl -n firecrawl get pods

# Check API logs
kubectl -n firecrawl logs -l app=firecrawl-api -f

# Check worker logs
kubectl -n firecrawl logs -l app=firecrawl-worker -f
```

## API Usage

Example: Scrape a webpage

```bash
curl -X POST https://spaceship-earth.local.mk-labs.cloud/v0/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
```

## Troubleshooting

### Pods not starting

Check resource availability:
```bash
kubectl -n firecrawl describe pod <pod-name>
```

### PostgreSQL data persistence

Currently using `emptyDir` for simplicity. To add persistence:

1. Create a PVC in `postgres-pvc.yaml`
2. Update `postgres-deployment.yaml` to use the PVC
3. Commit and push

### Gateway not routing

Verify ReferenceGrant allows firecrawl namespace:
```bash
kubectl -n gateway get referencegrant allow-httproutes -o yaml
```

## Future Enhancements

- [ ] Add persistent volume for PostgreSQL
- [ ] Add extract worker deployment (data extraction)
- [ ] Add nuq worker deployment (advanced queue processing)
- [ ] Configure monitoring/metrics
- [ ] Add resource quotas and limits
- [ ] Integrate with JARVIS via API key authentication