f4181349f8
- Add Firecrawl application with full stack: - Firecrawl API (main service) - Firecrawl Worker (background jobs) - Playwright Service (browser automation) - Redis (cache & job queue) - PostgreSQL (state management) - RabbitMQ (message queue) - Configure dual DNS names: - Primary: spaceship-earth.local.mk-labs.cloud (EPCOT theme) - Secondary: firecrawl.local.mk-labs.cloud - Add Gateway API HTTPRoutes with TLS certificates - Update ReferenceGrant for firecrawl namespace - Configure ArgoCD application (wave 20) - Set USE_DB_AUTHENTICATION=false for internal deployment This provides JARVIS with web scraping and search capabilities.
126 lines
3.3 KiB
Markdown
126 lines
3.3 KiB
Markdown
# Firecrawl (Spaceship Earth)
|
|
|
|
Web scraping and search service for JARVIS's web search capabilities.
|
|
|
|
## Service Names
|
|
|
|
- **Primary (EPCOT):** `spaceship-earth.local.mk-labs.cloud`
|
|
- **Secondary:** `firecrawl.local.mk-labs.cloud`
|
|
|
|
## Architecture
|
|
|
|
### Components
|
|
|
|
1. **Firecrawl API** - Main REST API service for web scraping
|
|
- Port: 3002
|
|
- Health checks: `/v0/health/liveness`, `/v0/health/readiness`
|
|
- Resources: 4Gi memory, 2 CPU cores (6Gi limit)
|
|
|
|
2. **Firecrawl Worker** - Background job processor
|
|
- Processes crawl jobs from queue
|
|
- Resources: 3Gi memory, 1 CPU core (4Gi limit)
|
|
|
|
3. **Playwright Service** - Browser automation
|
|
- Port: 3000
|
|
- Headless Chrome/Firefox for JavaScript rendering
|
|
- Resources: 2Gi memory, 1 CPU core (4Gi limit)
|
|
|
|
4. **Redis** - Cache and job queue
|
|
- Port: 6379
|
|
- Resources: 256Mi memory, 100m CPU (512Mi limit)
|
|
|
|
5. **PostgreSQL (nuq-postgres)** - State management
|
|
- Port: 5432
|
|
- Custom image with queue extensions
|
|
- Resources: 512Mi memory, 250m CPU (1Gi limit)
|
|
- **Note:** Currently using emptyDir (ephemeral). Add PVC for persistence if needed.
|
|
|
|
6. **RabbitMQ** - Message queue for distributed processing
|
|
- AMQP Port: 5672
|
|
- Management UI: 15672
|
|
- Resources: 512Mi memory, 250m CPU (1Gi limit)
|
|
|
|
### Configuration
|
|
|
|
Key environment variables in `configmap.yaml`:
|
|
|
|
- `USE_DB_AUTHENTICATION=false` - Simplified auth for internal use
|
|
- `NUM_WORKERS_PER_QUEUE=8` - Worker concurrency
|
|
- `CRAWL_CONCURRENT_REQUESTS=10` - Parallel crawl requests
|
|
- `MAX_CONCURRENT_JOBS=5` - Maximum simultaneous jobs
|
|
|
|
## Access
|
|
|
|
- **HTTPS (Gateway API):** https://spaceship-earth.local.mk-labs.cloud
|
|
- **HTTPS (Gateway API):** https://firecrawl.local.mk-labs.cloud
|
|
- Internal: `http://firecrawl-api.firecrawl.svc.cluster.local:3002`
|
|
|
|
## DNS
|
|
|
|
ExternalDNS automatically configures DNS records pointing to the Gateway load balancer (10.1.71.90).
|
|
|
|
## TLS
|
|
|
|
Certificates are automatically provisioned via cert-manager using the `letsencrypt-prod` ClusterIssuer.
|
|
|
|
## Deployment
|
|
|
|
Managed by ArgoCD (wave 20):
|
|
|
|
```bash
|
|
# Check application status
|
|
kubectl -n argocd get application firecrawl
|
|
|
|
# View pods
|
|
kubectl -n firecrawl get pods
|
|
|
|
# Check API logs
|
|
kubectl -n firecrawl logs -l app=firecrawl-api -f
|
|
|
|
# Check worker logs
|
|
kubectl -n firecrawl logs -l app=firecrawl-worker -f
|
|
```
|
|
|
|
## API Usage
|
|
|
|
Example: Scrape a webpage
|
|
|
|
```bash
|
|
curl -X POST https://spaceship-earth.local.mk-labs.cloud/v0/scrape \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"url": "https://example.com"}'
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Pods not starting
|
|
|
|
Check resource availability:
|
|
```bash
|
|
kubectl -n firecrawl describe pod <pod-name>
|
|
```
|
|
|
|
### PostgreSQL data persistence
|
|
|
|
Currently using `emptyDir` for simplicity. To add persistence:
|
|
|
|
1. Create a PVC in `postgres-pvc.yaml`
|
|
2. Update `postgres-deployment.yaml` to use the PVC
|
|
3. Commit and push
|
|
|
|
### Gateway not routing
|
|
|
|
Verify ReferenceGrant allows firecrawl namespace:
|
|
```bash
|
|
kubectl -n gateway get referencegrant allow-httproutes -o yaml
|
|
```
|
|
|
|
## Future Enhancements
|
|
|
|
- [ ] Add persistent volume for PostgreSQL
|
|
- [ ] Add extract worker deployment (data extraction)
|
|
- [ ] Add nuq worker deployment (advanced queue processing)
|
|
- [ ] Configure monitoring/metrics
|
|
- [ ] Add resource quotas and limits
|
|
- [ ] Integrate with JARVIS via API key authentication
|