chore: Remove Firecrawl deployment - pausing until platform infra is ready

Removing Firecrawl ArgoCD application and all manifests. The deployment
was failing due to missing container images that need to be built from
source. This requires platform infrastructure we don't have yet.

Will return to Firecrawl deployment after Harbor registry and Tekton
pipelines are deployed and configured.

Note: ArgoCD also needs a thematic EPCOT name at some point.
This commit is contained in:
Hermes Agent service account
2026-06-04 19:10:47 -05:00
parent 461aa1bc54
commit 52e97f3a7c
33 changed files with 0 additions and 2336 deletions

View File

@@ -1,125 +0,0 @@
# Firecrawl (Spaceship Earth)
Web scraping and search service for JARVIS's web search capabilities.
## Service Names
- **Primary (EPCOT):** `spaceship-earth.local.mk-labs.cloud`
- **Secondary:** `firecrawl.local.mk-labs.cloud`
## Architecture
### Components
1. **Firecrawl API** - Main REST API service for web scraping
- Port: 3002
- Health checks: `/v0/health/liveness`, `/v0/health/readiness`
- Resources: 4Gi memory, 2 CPU cores (6Gi limit)
2. **Firecrawl Worker** - Background job processor
- Processes crawl jobs from queue
- Resources: 3Gi memory, 1 CPU core (4Gi limit)
3. **Playwright Service** - Browser automation
- Port: 3000
- Headless Chrome/Firefox for JavaScript rendering
- Resources: 2Gi memory, 1 CPU core (4Gi limit)
4. **Redis** - Cache and job queue
- Port: 6379
- Resources: 256Mi memory, 100m CPU (512Mi limit)
5. **PostgreSQL (nuq-postgres)** - State management
- Port: 5432
- Custom image with queue extensions
- Resources: 512Mi memory, 250m CPU (1Gi limit)
- **Note:** Currently using emptyDir (ephemeral). Add PVC for persistence if needed.
6. **RabbitMQ** - Message queue for distributed processing
- AMQP Port: 5672
- Management UI: 15672
- Resources: 512Mi memory, 250m CPU (1Gi limit)
### Configuration
Key environment variables in `configmap.yaml`:
- `USE_DB_AUTHENTICATION=false` - Simplified auth for internal use
- `NUM_WORKERS_PER_QUEUE=8` - Worker concurrency
- `CRAWL_CONCURRENT_REQUESTS=10` - Parallel crawl requests
- `MAX_CONCURRENT_JOBS=5` - Maximum simultaneous jobs
## Access
- **HTTPS (Gateway API):** https://spaceship-earth.local.mk-labs.cloud
- **HTTPS (Gateway API):** https://firecrawl.local.mk-labs.cloud
- Internal: `http://firecrawl-api.firecrawl.svc.cluster.local:3002`
## DNS
ExternalDNS automatically configures DNS records pointing to the Gateway load balancer (10.1.71.90).
## TLS
Certificates are automatically provisioned via cert-manager using the `letsencrypt-prod` ClusterIssuer.
## Deployment
Managed by ArgoCD (wave 20):
```bash
# Check application status
kubectl -n argocd get application firecrawl
# View pods
kubectl -n firecrawl get pods
# Check API logs
kubectl -n firecrawl logs -l app=firecrawl-api -f
# Check worker logs
kubectl -n firecrawl logs -l app=firecrawl-worker -f
```
## API Usage
Example: Scrape a webpage
```bash
curl -X POST https://spaceship-earth.local.mk-labs.cloud/v0/scrape \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'
```
## Troubleshooting
### Pods not starting
Check resource availability:
```bash
kubectl -n firecrawl describe pod <pod-name>
```
### PostgreSQL data persistence
Currently using `emptyDir` for simplicity. To add persistence:
1. Create a PVC in `postgres-pvc.yaml`
2. Update `postgres-deployment.yaml` to use the PVC
3. Commit and push
### Gateway not routing
Verify ReferenceGrant allows firecrawl namespace:
```bash
kubectl -n gateway get referencegrant allow-httproutes -o yaml
```
## Future Enhancements
- [ ] Add persistent volume for PostgreSQL
- [ ] Add extract worker deployment (data extraction)
- [ ] Add nuq worker deployment (advanced queue processing)
- [ ] Configure monitoring/metrics
- [ ] Add resource quotas and limits
- [ ] Integrate with JARVIS via API key authentication