homelab/cluster/platform/harbor/README.md

# The Seas - Harbor Container Registry

**EPCOT Theme:** The Seas with Nemo & Friends  
**Service:** Harbor Container Registry  
**Wave:** 7

## Overview

Harbor is an open-source container registry that provides trust, compliance, performance, and interoperability. It serves as the central image repository for all custom-built containers in the mk-labs homelab.

## Service Details

- **URL:** https://the-seas.local.mk-labs.cloud
- **Namespace:** harbor
- **Sync Wave:** 7 (deploys after Gateway API, cert-manager, ExternalSecrets)

## Architecture

### Deployment Model
- **Database:** Embedded PostgreSQL (internal to Harbor)
- **Cache:** Embedded Redis (internal to Harbor)
- **HA Mode:** Single instance (no HA for homelab simplicity)
- **Storage:** NFS via nfs-emporium StorageClass

### Versions
- **Helm Chart:** 1.19.1
- **Harbor:** v2.15.1
- **All component images:** Pinned to 2.15.1 tag

## Components

Harbor consists of several microservices:

1. **Portal** - Web UI (1 replica)
2. **Core** - Main API server (1 replica)
3. **Registry** - OCI/Docker registry backend (1 replica)
4. **Job Service** - Async job processor (1 replica)
5. **Database** - PostgreSQL for metadata (embedded)
6. **Redis** - Cache and job queue (embedded)
7. **Trivy** - Vulnerability scanner (enabled)
8. **Exporter** - Prometheus metrics exporter (enabled)
9. **NGINX** - Reverse proxy frontend (1 replica)

## Storage Configuration

| Component | StorageClass | Size | Purpose |
|-----------|-------------|------|---------|
| Registry | nfs-emporium | 100Gi | Container images and Helm charts |
| Database | nfs-emporium | 10Gi | Metadata and configuration |
| Redis | nfs-emporium | 5Gi | Cache and job queue persistence |
| JobService | nfs-emporium | 5Gi | Job logs |
| Trivy | nfs-emporium | 10Gi | Vulnerability database |

**Total:** ~130Gi storage provisioned

## Resource Allocation

Approximate resource requests/limits per component:

- **Core:** 512Mi-1Gi RAM, 0.5-1 CPU
- **Registry:** 1-2Gi RAM, 0.5-1 CPU
- **Database:** 1-2Gi RAM, 0.5-2 CPU
- **Redis:** 256-512Mi RAM, 0.25-0.5 CPU
- **Portal:** 256-512Mi RAM, 0.25-0.5 CPU
- **JobService:** 512Mi-1Gi RAM, 0.5-1 CPU
- **Trivy:** 512Mi-1Gi RAM, 0.5-1 CPU
- **Exporter:** 256-512Mi RAM, 0.25-0.5 CPU
- **NGINX:** 256-512Mi RAM, 0.25-0.5 CPU

**Total:** ~6-10Gi RAM, ~4-8 CPU cores across all pods

## Secrets Management

All credentials are stored in 1Password and synchronized to Kubernetes via ExternalSecrets Operator.

**1Password Item:** `the-seas` (vault: `mk-labs`)

**Required Secrets:**
1. `harbor-admin-password` - Web UI admin user password
2. `database-password` - PostgreSQL password
3. `redis-password` - Redis password
4. `core-secret` - Harbor core component secret
5. `jobservice-secret` - Job service authentication secret
6. `registry-password` - Registry backend password

**Kubernetes Secret:** `the-seas-credentials` (namespace: the-seas)

## Network & TLS

### DNS Name
- **Hostname:** the-seas.local.mk-labs.cloud

DNS resolves to the NGINX Ingress Controller IP (10.1.71.80).

### TLS Certificate
- Issued by Let's Encrypt (letsencrypt-staging ClusterIssuer)
- Managed by cert-manager (automatic via Ingress annotation)
- Certificate stored as Kubernetes secret: `harbor-tls`

### Routing
- **Ingress Controller:** NGINX (nginx-ingress)
- **Protocol:** HTTPS (port 443)
- **Backend:** Harbor NGINX service on port 80
- TLS termination at Ingress level

## Access & Authentication

### Web UI
1. Navigate to https://the-seas.local.mk-labs.cloud or https://harbor.local.mk-labs.cloud
2. Username: `admin`
3. Password: From 1Password item `the-seas`, field `harbor-admin-password`

### Docker CLI
```bash
# Login to registry
docker login the-seas.local.mk-labs.cloud
# or
docker login harbor.local.mk-labs.cloud

# Username: admin
# Password: <from 1Password>
```

### Robot Accounts
For CI/CD automation, robot accounts are created post-deployment:

1. **tekton-builder** - For Tekton CI/CD image pushes
   - Permissions: Push artifacts, Create tags
   - Token stored in 1Password: `the-seas-tekton-robot`

2. **argocd-puller** - For ArgoCD image metadata reads
   - Permissions: Pull artifacts, Read metadata (read-only)
   - Token stored in 1Password: `the-seas-argocd-robot`

## Initial Project Structure

Harbor organizes images into projects. Initial projects:

1. **library** - General-purpose images (default)
2. **platform** - Infrastructure/platform images (Tekton, etc.)
3. **applications** - Application images (Firecrawl, etc.)

All projects are private by default.

## Deployment

### Prerequisites
- ArgoCD deployed and managing cluster
- Gateway API (wave 4) deployed
- cert-manager (wave 3) deployed
- ExternalSecrets Operator (wave 1) deployed
- 1Password Connect (wave 2) deployed
- 1Password item `the-seas` created with required secrets

### Apply
```bash
# From repository root
kubectl apply -f cluster/platform/the-seas/application.yaml

# ArgoCD will sync automatically
```

### Verify
```bash
# Check ArgoCD application status
kubectl get application -n argocd the-seas

# Check all pods are running
kubectl get pods -n the-seas

# Check PVCs are bound
kubectl get pvc -n the-seas

# Check ExternalSecret synced
kubectl get externalsecret -n the-seas the-seas-credentials

# Check certificates issued
kubectl get certificate -n the-seas
```

## Testing

### Basic Functionality
```bash
# 1. Web UI access
curl -k https://the-seas.local.mk-labs.cloud
# Should return Harbor web UI HTML

# 2. Docker login
docker login the-seas.local.mk-labs.cloud
# Should prompt for credentials and succeed

# 3. Push test image
docker pull alpine:latest
docker tag alpine:latest the-seas.local.mk-labs.cloud/library/alpine:test
docker push the-seas.local.mk-labs.cloud/library/alpine:test

# 4. Pull test image
docker rmi the-seas.local.mk-labs.cloud/library/alpine:test
docker pull the-seas.local.mk-labs.cloud/library/alpine:test

# 5. Verify in web UI
# Navigate to Projects → library → Repositories
# Should see alpine image with test tag
```

### Certificate Validation
```bash
# Check TLS certificate
echo | openssl s_client -connect the-seas.local.mk-labs.cloud:443 2>/dev/null | openssl x509 -noout -text | grep -A 2 "Subject:"

# Verify certificate is valid
echo | openssl s_client -connect the-seas.local.mk-labs.cloud:443 2>/dev/null | grep "Verify return code"
# Should show: Verify return code: 0 (ok)
```

## Monitoring

Harbor exports Prometheus metrics on port 8001 for several components:
- Core: `/metrics`
- Registry: `/metrics`
- JobService: `/metrics`
- Exporter: `/metrics`

Metrics are enabled in values.yaml and can be scraped by Prometheus.

## Troubleshooting

### Pods CrashLoopBackOff

Check pod logs:
```bash
kubectl logs -n the-seas <pod-name>
```

Common issues:
- **Database init failures:** Check database password in ExternalSecret
- **Redis connection errors:** Check redis password in ExternalSecret
- **Secret not found:** Verify ExternalSecret synced successfully

### Cannot Access Web UI

1. Check Gateway and HTTPRoute:
```bash
kubectl get gateway -n gateway fastpass-gateway
kubectl get httproute -n the-seas
```

2. Check DNS resolution:
```bash
nslookup the-seas.local.mk-labs.cloud
# Should resolve to 10.1.71.90
```

3. Check certificate:
```bash
kubectl get certificate -n the-seas the-seas-tls
# Should show Ready=True
```

### Docker Login Fails

1. Check DNS resolution
2. Check certificate is trusted
3. Verify admin password from 1Password is correct
4. Check Harbor core service is running:
```bash
kubectl get svc -n the-seas the-seas
kubectl get pods -n the-seas -l component=core
```

### Image Push/Pull Fails

1. Check registry pod:
```bash
kubectl get pods -n the-seas -l component=registry
kubectl logs -n the-seas -l component=registry
```

2. Check storage PVC is bound:
```bash
kubectl get pvc -n the-seas
```

3. Check project permissions in Harbor UI

## Maintenance

### Backup
Harbor data is persisted on NFS volumes. Ensure NFS storage is backed up regularly.

Critical data:
- `/storage` on registry PVC - All container images
- Database PVC - All metadata, users, projects, policies

### Updates
Harbor is managed by ArgoCD. To update:

1. Update `targetRevision` in `application.yaml` to new chart version
2. Review changelog: https://github.com/goharbor/harbor/releases
3. Update image tags in `values.yaml` if needed
4. Commit and push to Git
5. ArgoCD will automatically sync

### Scaling
Current deployment is single-instance. To scale:

1. Update replica counts in `values.yaml`:
   - `portal.replicas`
   - `core.replicas`
   - `registry.replicas`
   - `jobservice.replicas`

2. Consider external PostgreSQL and Redis for HA

## References

- **Harbor Documentation:** https://goharbor.io/docs/
- **Harbor Helm Chart:** https://github.com/goharbor/harbor-helm
- **Repository Path:** `/home/hermes/git/homelab/cluster/platform/the-seas/`
- **ArgoCD Application:** https://argocd.mk-labs.cloud/applications/the-seas
- **1Password Vault:** mk-labs
- **1Password Item:** the-seas

## Next Steps

1. **Phase 2:** Integrate with Tekton (innoventions) for CI/CD
2. **Phase 3:** Configure vulnerability scanning policies
3. **Phase 4:** Implement image signing with Cosign (Notary v2)
4. **Phase 5:** Set up replication to backup registry (if needed)

## Support

For issues or questions:
- Check troubleshooting section above
- Review Harbor logs: `kubectl logs -n the-seas <pod-name>`
- Consult Harbor documentation: https://goharbor.io/docs/
- Escalate to Ryan via `/mnt/mk-labs-pka/ryan-inbox/`