homelab/cluster/platform/the-seas/README.md

The Seas - Harbor Container Registry

EPCOT Theme: The Seas with Nemo & Friends
Service: Harbor Container Registry
Wave: 7

Overview

Harbor is an open-source container registry that provides trust, compliance, performance, and interoperability. It serves as the central image repository for all custom-built containers in the mk-labs homelab.

Service Details

• Primary URL: https://the-seas.local.mk-labs.cloud
• Alternate URL: https://harbor.local.mk-labs.cloud (for intuitive docker login harbor...)
• Namespace: the-seas
• Sync Wave: 7 (deploys after Gateway API, cert-manager, ExternalSecrets)

Architecture

Deployment Model

• Database: Embedded PostgreSQL (internal to Harbor)
• Cache: Embedded Redis (internal to Harbor)
• HA Mode: Single instance (no HA for homelab simplicity)
• Storage: NFS via nfs-emporium StorageClass

Versions

• Helm Chart: 1.19.1
• Harbor: v2.15.1
• All component images: Pinned to 2.15.1 tag

Components

Harbor consists of several microservices:

1. Portal - Web UI (1 replica)
2. Core - Main API server (1 replica)
3. Registry - OCI/Docker registry backend (1 replica)
4. Job Service - Async job processor (1 replica)
5. Database - PostgreSQL for metadata (embedded)
6. Redis - Cache and job queue (embedded)
7. Trivy - Vulnerability scanner (enabled)
8. Exporter - Prometheus metrics exporter (enabled)
9. NGINX - Reverse proxy frontend (1 replica)

Storage Configuration

Component	StorageClass	Size	Purpose
Registry	nfs-emporium	100Gi	Container images and Helm charts
Database	nfs-emporium	10Gi	Metadata and configuration
Redis	nfs-emporium	5Gi	Cache and job queue persistence
JobService	nfs-emporium	5Gi	Job logs
Trivy	nfs-emporium	10Gi	Vulnerability database

Total: ~130Gi storage provisioned

Resource Allocation

Approximate resource requests/limits per component:

• Core: 512Mi-1Gi RAM, 0.5-1 CPU
• Registry: 1-2Gi RAM, 0.5-1 CPU
• Database: 1-2Gi RAM, 0.5-2 CPU
• Redis: 256-512Mi RAM, 0.25-0.5 CPU
• Portal: 256-512Mi RAM, 0.25-0.5 CPU
• JobService: 512Mi-1Gi RAM, 0.5-1 CPU
• Trivy: 512Mi-1Gi RAM, 0.5-1 CPU
• Exporter: 256-512Mi RAM, 0.25-0.5 CPU
• NGINX: 256-512Mi RAM, 0.25-0.5 CPU

Total: ~6-10Gi RAM, ~4-8 CPU cores across all pods

Secrets Management

All credentials are stored in 1Password and synchronized to Kubernetes via ExternalSecrets Operator.

1Password Item: the-seas (vault: mk-labs)

Required Secrets:

1. harbor-admin-password - Web UI admin user password
2. database-password - PostgreSQL password
3. redis-password - Redis password
4. core-secret - Harbor core component secret
5. jobservice-secret - Job service authentication secret
6. registry-password - Registry backend password

Kubernetes Secret: the-seas-credentials (namespace: the-seas)

Network & TLS

DNS Names

• Primary: the-seas.local.mk-labs.cloud
• Alternate: harbor.local.mk-labs.cloud

Both DNS names resolve to the Cilium Gateway IP (10.1.71.90) via external-dns.

TLS Certificates

• Issued by Let's Encrypt (letsencrypt-prod ClusterIssuer)
• Managed by cert-manager
• Separate certificates for each hostname
• Stored as Kubernetes secrets: the-seas-tls, harbor-tls

Routing

• Gateway: fastpass-gateway (namespace: gateway)
• Protocol: HTTPS (port 443)
• Backend: ClusterIP service the-seas on port 80
• TLS termination at Gateway level

Access & Authentication

Web UI

1. Navigate to https://the-seas.local.mk-labs.cloud or https://harbor.local.mk-labs.cloud
2. Username: admin
3. Password: From 1Password item the-seas, field harbor-admin-password

Docker CLI

# Login to registry
docker login the-seas.local.mk-labs.cloud
# or
docker login harbor.local.mk-labs.cloud

# Username: admin
# Password: <from 1Password>

Robot Accounts

For CI/CD automation, robot accounts are created post-deployment:

1. tekton-builder - For Tekton CI/CD image pushes

   • Permissions: Push artifacts, Create tags
   • Token stored in 1Password: the-seas-tekton-robot

2. argocd-puller - For ArgoCD image metadata reads

   • Permissions: Pull artifacts, Read metadata (read-only)
   • Token stored in 1Password: the-seas-argocd-robot

Initial Project Structure

Harbor organizes images into projects. Initial projects:

1. library - General-purpose images (default)
2. platform - Infrastructure/platform images (Tekton, etc.)
3. applications - Application images (Firecrawl, etc.)

All projects are private by default.

Deployment

Prerequisites

• ArgoCD deployed and managing cluster
• Gateway API (wave 4) deployed
• cert-manager (wave 3) deployed
• ExternalSecrets Operator (wave 1) deployed
• 1Password Connect (wave 2) deployed
• 1Password item the-seas created with required secrets

Apply

# From repository root
kubectl apply -f cluster/platform/the-seas/application.yaml

# ArgoCD will sync automatically

Verify

# Check ArgoCD application status
kubectl get application -n argocd the-seas

# Check all pods are running
kubectl get pods -n the-seas

# Check PVCs are bound
kubectl get pvc -n the-seas

# Check ExternalSecret synced
kubectl get externalsecret -n the-seas the-seas-credentials

# Check certificates issued
kubectl get certificate -n the-seas

Testing

Basic Functionality

# 1. Web UI access
curl -k https://the-seas.local.mk-labs.cloud
# Should return Harbor web UI HTML

# 2. Docker login
docker login the-seas.local.mk-labs.cloud
# Should prompt for credentials and succeed

# 3. Push test image
docker pull alpine:latest
docker tag alpine:latest the-seas.local.mk-labs.cloud/library/alpine:test
docker push the-seas.local.mk-labs.cloud/library/alpine:test

# 4. Pull test image
docker rmi the-seas.local.mk-labs.cloud/library/alpine:test
docker pull the-seas.local.mk-labs.cloud/library/alpine:test

# 5. Verify in web UI
# Navigate to Projects → library → Repositories
# Should see alpine image with test tag

Certificate Validation

# Check TLS certificate
echo | openssl s_client -connect the-seas.local.mk-labs.cloud:443 2>/dev/null | openssl x509 -noout -text | grep -A 2 "Subject:"

# Verify certificate is valid
echo | openssl s_client -connect the-seas.local.mk-labs.cloud:443 2>/dev/null | grep "Verify return code"
# Should show: Verify return code: 0 (ok)

Monitoring

Harbor exports Prometheus metrics on port 8001 for several components:

• Core: /metrics
• Registry: /metrics
• JobService: /metrics
• Exporter: /metrics

Metrics are enabled in values.yaml and can be scraped by Prometheus.

Troubleshooting

Pods CrashLoopBackOff

Check pod logs:

kubectl logs -n the-seas <pod-name>

Common issues:

• Database init failures: Check database password in ExternalSecret
• Redis connection errors: Check redis password in ExternalSecret
• Secret not found: Verify ExternalSecret synced successfully

Cannot Access Web UI

1. Check Gateway and HTTPRoute:

kubectl get gateway -n gateway fastpass-gateway
kubectl get httproute -n the-seas

2. Check DNS resolution:

nslookup the-seas.local.mk-labs.cloud
# Should resolve to 10.1.71.90

3. Check certificate:

kubectl get certificate -n the-seas the-seas-tls
# Should show Ready=True

Docker Login Fails

1. Check DNS resolution
2. Check certificate is trusted
3. Verify admin password from 1Password is correct
4. Check Harbor core service is running:

kubectl get svc -n the-seas the-seas
kubectl get pods -n the-seas -l component=core

Image Push/Pull Fails

1. Check registry pod:

kubectl get pods -n the-seas -l component=registry
kubectl logs -n the-seas -l component=registry

2. Check storage PVC is bound:

kubectl get pvc -n the-seas

3. Check project permissions in Harbor UI

Maintenance

Backup

Harbor data is persisted on NFS volumes. Ensure NFS storage is backed up regularly.

Critical data:

• /storage on registry PVC - All container images
• Database PVC - All metadata, users, projects, policies

Updates

Harbor is managed by ArgoCD. To update:

1. Update targetRevision in application.yaml to new chart version
2. Review changelog: https://github.com/goharbor/harbor/releases
3. Update image tags in values.yaml if needed
4. Commit and push to Git
5. ArgoCD will automatically sync

Scaling

Current deployment is single-instance. To scale:

1. Update replica counts in values.yaml:

   • portal.replicas
   • core.replicas
   • registry.replicas
   • jobservice.replicas

2. Consider external PostgreSQL and Redis for HA

References

• Harbor Documentation: https://goharbor.io/docs/
• Harbor Helm Chart: https://github.com/goharbor/harbor-helm
• Repository Path: /home/hermes/git/homelab/cluster/platform/the-seas/
• ArgoCD Application: https://argocd.mk-labs.cloud/applications/the-seas
• 1Password Vault: mk-labs
• 1Password Item: the-seas

Next Steps

1. Phase 2: Integrate with Tekton (innoventions) for CI/CD
2. Phase 3: Configure vulnerability scanning policies
3. Phase 4: Implement image signing with Cosign (Notary v2)
4. Phase 5: Set up replication to backup registry (if needed)

Support

For issues or questions:

• Check troubleshooting section above
• Review Harbor logs: kubectl logs -n the-seas <pod-name>
• Consult Harbor documentation: https://goharbor.io/docs/
• Escalate to Ryan via /mnt/mk-labs-pka/ryan-inbox/