fix(harbor): correct naming convention and use staging certs

- Rename application/namespace: the-seas -> harbor
- Move directory: cluster/platform/the-seas -> cluster/platform/harbor
- Update all resource references (ExternalSecret, HTTPRoutes, Certificates)
- Switch to letsencrypt-staging issuer (avoid ACME rate limits during testing)
- Thematic name 'the-seas' remains in DNS hostnames and comments
This commit is contained in:
Hermes Agent service account
2026-06-04 20:25:25 -05:00
parent 8f190eb188
commit 4b1e8a7cac
8 changed files with 34 additions and 32 deletions

View File

@@ -0,0 +1,341 @@
# The Seas - Harbor Container Registry
**EPCOT Theme:** The Seas with Nemo & Friends
**Service:** Harbor Container Registry
**Wave:** 7
## Overview
Harbor is an open-source container registry that provides trust, compliance, performance, and interoperability. It serves as the central image repository for all custom-built containers in the mk-labs homelab.
## Service Details
- **Primary URL:** https://the-seas.local.mk-labs.cloud
- **Alternate URL:** https://harbor.local.mk-labs.cloud (for intuitive `docker login harbor...`)
- **Namespace:** the-seas
- **Sync Wave:** 7 (deploys after Gateway API, cert-manager, ExternalSecrets)
## Architecture
### Deployment Model
- **Database:** Embedded PostgreSQL (internal to Harbor)
- **Cache:** Embedded Redis (internal to Harbor)
- **HA Mode:** Single instance (no HA for homelab simplicity)
- **Storage:** NFS via nfs-emporium StorageClass
### Versions
- **Helm Chart:** 1.19.1
- **Harbor:** v2.15.1
- **All component images:** Pinned to 2.15.1 tag
## Components
Harbor consists of several microservices:
1. **Portal** - Web UI (1 replica)
2. **Core** - Main API server (1 replica)
3. **Registry** - OCI/Docker registry backend (1 replica)
4. **Job Service** - Async job processor (1 replica)
5. **Database** - PostgreSQL for metadata (embedded)
6. **Redis** - Cache and job queue (embedded)
7. **Trivy** - Vulnerability scanner (enabled)
8. **Exporter** - Prometheus metrics exporter (enabled)
9. **NGINX** - Reverse proxy frontend (1 replica)
## Storage Configuration
| Component | StorageClass | Size | Purpose |
|-----------|-------------|------|---------|
| Registry | nfs-emporium | 100Gi | Container images and Helm charts |
| Database | nfs-emporium | 10Gi | Metadata and configuration |
| Redis | nfs-emporium | 5Gi | Cache and job queue persistence |
| JobService | nfs-emporium | 5Gi | Job logs |
| Trivy | nfs-emporium | 10Gi | Vulnerability database |
**Total:** ~130Gi storage provisioned
## Resource Allocation
Approximate resource requests/limits per component:
- **Core:** 512Mi-1Gi RAM, 0.5-1 CPU
- **Registry:** 1-2Gi RAM, 0.5-1 CPU
- **Database:** 1-2Gi RAM, 0.5-2 CPU
- **Redis:** 256-512Mi RAM, 0.25-0.5 CPU
- **Portal:** 256-512Mi RAM, 0.25-0.5 CPU
- **JobService:** 512Mi-1Gi RAM, 0.5-1 CPU
- **Trivy:** 512Mi-1Gi RAM, 0.5-1 CPU
- **Exporter:** 256-512Mi RAM, 0.25-0.5 CPU
- **NGINX:** 256-512Mi RAM, 0.25-0.5 CPU
**Total:** ~6-10Gi RAM, ~4-8 CPU cores across all pods
## Secrets Management
All credentials are stored in 1Password and synchronized to Kubernetes via ExternalSecrets Operator.
**1Password Item:** `the-seas` (vault: `mk-labs`)
**Required Secrets:**
1. `harbor-admin-password` - Web UI admin user password
2. `database-password` - PostgreSQL password
3. `redis-password` - Redis password
4. `core-secret` - Harbor core component secret
5. `jobservice-secret` - Job service authentication secret
6. `registry-password` - Registry backend password
**Kubernetes Secret:** `the-seas-credentials` (namespace: the-seas)
## Network & TLS
### DNS Names
- **Primary:** the-seas.local.mk-labs.cloud
- **Alternate:** harbor.local.mk-labs.cloud
Both DNS names resolve to the Cilium Gateway IP (10.1.71.90) via external-dns.
### TLS Certificates
- Issued by Let's Encrypt (letsencrypt-prod ClusterIssuer)
- Managed by cert-manager
- Separate certificates for each hostname
- Stored as Kubernetes secrets: `the-seas-tls`, `harbor-tls`
### Routing
- **Gateway:** fastpass-gateway (namespace: gateway)
- **Protocol:** HTTPS (port 443)
- **Backend:** ClusterIP service `the-seas` on port 80
- TLS termination at Gateway level
## Access & Authentication
### Web UI
1. Navigate to https://the-seas.local.mk-labs.cloud or https://harbor.local.mk-labs.cloud
2. Username: `admin`
3. Password: From 1Password item `the-seas`, field `harbor-admin-password`
### Docker CLI
```bash
# Login to registry
docker login the-seas.local.mk-labs.cloud
# or
docker login harbor.local.mk-labs.cloud
# Username: admin
# Password: <from 1Password>
```
### Robot Accounts
For CI/CD automation, robot accounts are created post-deployment:
1. **tekton-builder** - For Tekton CI/CD image pushes
- Permissions: Push artifacts, Create tags
- Token stored in 1Password: `the-seas-tekton-robot`
2. **argocd-puller** - For ArgoCD image metadata reads
- Permissions: Pull artifacts, Read metadata (read-only)
- Token stored in 1Password: `the-seas-argocd-robot`
## Initial Project Structure
Harbor organizes images into projects. Initial projects:
1. **library** - General-purpose images (default)
2. **platform** - Infrastructure/platform images (Tekton, etc.)
3. **applications** - Application images (Firecrawl, etc.)
All projects are private by default.
## Deployment
### Prerequisites
- ArgoCD deployed and managing cluster
- Gateway API (wave 4) deployed
- cert-manager (wave 3) deployed
- ExternalSecrets Operator (wave 1) deployed
- 1Password Connect (wave 2) deployed
- 1Password item `the-seas` created with required secrets
### Apply
```bash
# From repository root
kubectl apply -f cluster/platform/the-seas/application.yaml
# ArgoCD will sync automatically
```
### Verify
```bash
# Check ArgoCD application status
kubectl get application -n argocd the-seas
# Check all pods are running
kubectl get pods -n the-seas
# Check PVCs are bound
kubectl get pvc -n the-seas
# Check ExternalSecret synced
kubectl get externalsecret -n the-seas the-seas-credentials
# Check certificates issued
kubectl get certificate -n the-seas
```
## Testing
### Basic Functionality
```bash
# 1. Web UI access
curl -k https://the-seas.local.mk-labs.cloud
# Should return Harbor web UI HTML
# 2. Docker login
docker login the-seas.local.mk-labs.cloud
# Should prompt for credentials and succeed
# 3. Push test image
docker pull alpine:latest
docker tag alpine:latest the-seas.local.mk-labs.cloud/library/alpine:test
docker push the-seas.local.mk-labs.cloud/library/alpine:test
# 4. Pull test image
docker rmi the-seas.local.mk-labs.cloud/library/alpine:test
docker pull the-seas.local.mk-labs.cloud/library/alpine:test
# 5. Verify in web UI
# Navigate to Projects → library → Repositories
# Should see alpine image with test tag
```
### Certificate Validation
```bash
# Check TLS certificate
echo | openssl s_client -connect the-seas.local.mk-labs.cloud:443 2>/dev/null | openssl x509 -noout -text | grep -A 2 "Subject:"
# Verify certificate is valid
echo | openssl s_client -connect the-seas.local.mk-labs.cloud:443 2>/dev/null | grep "Verify return code"
# Should show: Verify return code: 0 (ok)
```
## Monitoring
Harbor exports Prometheus metrics on port 8001 for several components:
- Core: `/metrics`
- Registry: `/metrics`
- JobService: `/metrics`
- Exporter: `/metrics`
Metrics are enabled in values.yaml and can be scraped by Prometheus.
## Troubleshooting
### Pods CrashLoopBackOff
Check pod logs:
```bash
kubectl logs -n the-seas <pod-name>
```
Common issues:
- **Database init failures:** Check database password in ExternalSecret
- **Redis connection errors:** Check redis password in ExternalSecret
- **Secret not found:** Verify ExternalSecret synced successfully
### Cannot Access Web UI
1. Check Gateway and HTTPRoute:
```bash
kubectl get gateway -n gateway fastpass-gateway
kubectl get httproute -n the-seas
```
2. Check DNS resolution:
```bash
nslookup the-seas.local.mk-labs.cloud
# Should resolve to 10.1.71.90
```
3. Check certificate:
```bash
kubectl get certificate -n the-seas the-seas-tls
# Should show Ready=True
```
### Docker Login Fails
1. Check DNS resolution
2. Check certificate is trusted
3. Verify admin password from 1Password is correct
4. Check Harbor core service is running:
```bash
kubectl get svc -n the-seas the-seas
kubectl get pods -n the-seas -l component=core
```
### Image Push/Pull Fails
1. Check registry pod:
```bash
kubectl get pods -n the-seas -l component=registry
kubectl logs -n the-seas -l component=registry
```
2. Check storage PVC is bound:
```bash
kubectl get pvc -n the-seas
```
3. Check project permissions in Harbor UI
## Maintenance
### Backup
Harbor data is persisted on NFS volumes. Ensure NFS storage is backed up regularly.
Critical data:
- `/storage` on registry PVC - All container images
- Database PVC - All metadata, users, projects, policies
### Updates
Harbor is managed by ArgoCD. To update:
1. Update `targetRevision` in `application.yaml` to new chart version
2. Review changelog: https://github.com/goharbor/harbor/releases
3. Update image tags in `values.yaml` if needed
4. Commit and push to Git
5. ArgoCD will automatically sync
### Scaling
Current deployment is single-instance. To scale:
1. Update replica counts in `values.yaml`:
- `portal.replicas`
- `core.replicas`
- `registry.replicas`
- `jobservice.replicas`
2. Consider external PostgreSQL and Redis for HA
## References
- **Harbor Documentation:** https://goharbor.io/docs/
- **Harbor Helm Chart:** https://github.com/goharbor/harbor-helm
- **Repository Path:** `/home/hermes/git/homelab/cluster/platform/the-seas/`
- **ArgoCD Application:** https://argocd.mk-labs.cloud/applications/the-seas
- **1Password Vault:** mk-labs
- **1Password Item:** the-seas
## Next Steps
1. **Phase 2:** Integrate with Tekton (innoventions) for CI/CD
2. **Phase 3:** Configure vulnerability scanning policies
3. **Phase 4:** Implement image signing with Cosign (Notary v2)
4. **Phase 5:** Set up replication to backup registry (if needed)
## Support
For issues or questions:
- Check troubleshooting section above
- Review Harbor logs: `kubectl logs -n the-seas <pod-name>`
- Consult Harbor documentation: https://goharbor.io/docs/
- Escalate to Ryan via `/mnt/mk-labs-pka/ryan-inbox/`

View File

@@ -0,0 +1,40 @@
# ------------------------------------------------------------------------------
# Platform: harbor (Container Registry)
# Theme: the-seas (EPCOT: The Seas with Nemo & Friends)
# Wave 7 — depends on Gateway API (wave 4), cert-manager (wave 3), ESO (wave 1)
# Multi-source: Helm chart from upstream + values + manifests from repo
# ------------------------------------------------------------------------------
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: harbor
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "7"
spec:
project: default
sources:
# Source 1: Helm chart from upstream Harbor repository
- repoURL: https://helm.goharbor.io
chart: harbor
targetRevision: 1.19.1
helm:
valueFiles:
- $values/cluster/platform/harbor/values.yaml
# Source 2: Repo — values ref + ExternalSecret + HTTPRoutes + Certificates
- repoURL: https://gitea.mk-labs.cloud/rblundon/homelab.git
targetRevision: main
path: cluster/platform/harbor
ref: values
directory:
exclude: "application.yaml"
destination:
server: https://kubernetes.default.svc
namespace: harbor
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=true

View File

@@ -0,0 +1,16 @@
# ------------------------------------------------------------------------------
# Certificate — Harbor Alternate TLS Certificate (staging until stable)
# Alternate DNS: harbor.local.mk-labs.cloud
# ------------------------------------------------------------------------------
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: harbor-alt-tls
namespace: harbor
spec:
secretName: harbor-alt-tls
issuerRef:
name: letsencrypt-staging
kind: ClusterIssuer
dnsNames:
- harbor.local.mk-labs.cloud

View File

@@ -0,0 +1,16 @@
# ------------------------------------------------------------------------------
# Certificate — Harbor TLS Certificate (staging until stable)
# Primary DNS: the-seas.local.mk-labs.cloud
# ------------------------------------------------------------------------------
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: harbor-tls
namespace: harbor
spec:
secretName: harbor-tls
issuerRef:
name: letsencrypt-staging
kind: ClusterIssuer
dnsNames:
- the-seas.local.mk-labs.cloud

View File

@@ -0,0 +1,52 @@
# ------------------------------------------------------------------------------
# ExternalSecret — Harbor Credentials
# Pulled from 1Password via Connect Server, materialized as a K8s secret
# in the-seas namespace for use by Harbor components.
#
# In 1Password, these are stored in the "the-seas" item in the "mk-labs" vault
# with fields: harbor-admin-password, database-password, redis-password,
# core-secret, jobservice-secret, registry-password
# ------------------------------------------------------------------------------
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: harbor-credentials
namespace: harbor
spec:
secretStoreRef:
kind: ClusterSecretStore
name: onepassword-connect
refreshInterval: "1h"
target:
name: harbor-credentials
creationPolicy: Owner
data:
- secretKey: HARBOR_ADMIN_PASSWORD
remoteRef:
key: the-seas
property: harbor-admin-password
- secretKey: DATABASE_PASSWORD
remoteRef:
key: the-seas
property: database-password
- secretKey: REDIS_PASSWORD
remoteRef:
key: the-seas
property: redis-password
- secretKey: CORE_SECRET
remoteRef:
key: the-seas
property: core-secret
- secretKey: JOBSERVICE_SECRET
remoteRef:
key: the-seas
property: jobservice-secret
- secretKey: REGISTRY_PASSWD
remoteRef:
key: the-seas
property: registry-password

View File

@@ -0,0 +1,33 @@
# ------------------------------------------------------------------------------
# ------------------------------------------------------------------------------
# HTTPRoute — Harbor (Alternate) via Cilium Gateway
# Alternate DNS: harbor.local.mk-labs.cloud (functional name)
# ------------------------------------------------------------------------------
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: harbor-alt
namespace: harbor
annotations:
external-dns.alpha.kubernetes.io/hostname: harbor.local.mk-labs.cloud
external-dns.alpha.kubernetes.io/target: "10.1.71.90"
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: fastpass-gateway
namespace: gateway
sectionName: https
hostnames:
- harbor.local.mk-labs.cloud
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: ""
kind: Service
name: the-seas
port: 80
weight: 1

View File

@@ -0,0 +1,32 @@
# ------------------------------------------------------------------------------
# HTTPRoute — Harbor (Primary) via Cilium Gateway
# Primary DNS: the-seas.local.mk-labs.cloud (EPCOT theme)
# ------------------------------------------------------------------------------
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: harbor-the-seas
namespace: harbor
annotations:
external-dns.alpha.kubernetes.io/hostname: the-seas.local.mk-labs.cloud
external-dns.alpha.kubernetes.io/target: "10.1.71.90"
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: fastpass-gateway
namespace: gateway
sectionName: https
hostnames:
- the-seas.local.mk-labs.cloud
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: ""
kind: Service
name: harbor
port: 80
weight: 1

View File

@@ -0,0 +1,273 @@
# ------------------------------------------------------------------------------
# Harbor Helm Chart Values
# Service: the-seas (The Seas with Nemo & Friends)
# Architecture: Embedded PostgreSQL, embedded Redis, single instance
# ------------------------------------------------------------------------------
# External URL for Harbor - primary DNS name
externalURL: https://the-seas.local.mk-labs.cloud
# Expose Harbor via ClusterIP (Gateway API handles ingress)
expose:
type: clusterIP
tls:
enabled: true
# TLS termination handled by Gateway, not ingress
certSource: none
clusterIP:
name: the-seas
ports:
httpPort: 80
httpsPort: 443
# Admin credentials from ExternalSecret
harborAdminPassword: ""
existingSecretAdminPassword: "harbor-credentials"
existingSecretAdminPasswordKey: "HARBOR_ADMIN_PASSWORD"
# Disable internal TLS (Gateway terminates TLS)
internalTLS:
enabled: false
# Embedded PostgreSQL configuration
database:
type: internal
internal:
image:
repository: docker.io/goharbor/harbor-db
tag: "v2.15.1"
# Password from ExternalSecret
password: ""
existingSecret: "harbor-credentials"
resources:
requests:
memory: 1Gi
cpu: 500m
limits:
memory: 2Gi
cpu: 2000m
shmSizeLimit: 512Mi
maxIdleConns: 100
maxOpenConns: 900
podAnnotations: {}
podLabels: {}
# Embedded Redis configuration
redis:
type: internal
internal:
image:
repository: docker.io/goharbor/redis-photon
tag: "v2.15.1"
# Password from ExternalSecret
jobservicePassword: ""
existingSecret: "harbor-credentials"
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 512Mi
cpu: 500m
podAnnotations: {}
podLabels: {}
# Storage configuration - use nfs-emporium StorageClass
persistence:
enabled: true
resourcePolicy: "keep"
persistentVolumeClaim:
registry:
storageClass: nfs-emporium
accessMode: ReadWriteOnce
size: 100Gi
jobservice:
jobLog:
storageClass: nfs-emporium
accessMode: ReadWriteOnce
size: 5Gi
database:
storageClass: nfs-emporium
accessMode: ReadWriteOnce
size: 10Gi
redis:
storageClass: nfs-emporium
accessMode: ReadWriteOnce
size: 5Gi
trivy:
storageClass: nfs-emporium
accessMode: ReadWriteOnce
size: 10Gi
# Image chart storage - use local filesystem (on PVC)
imageChartStorage:
disableredirect: false
type: filesystem
filesystem:
rootdirectory: /storage
# Portal (UI) configuration - single replica
portal:
image:
repository: docker.io/goharbor/harbor-portal
tag: "v2.15.1"
replicas: 1
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 512Mi
cpu: 500m
podAnnotations: {}
podLabels: {}
# Core service configuration - single replica
core:
image:
repository: docker.io/goharbor/harbor-core
tag: "v2.15.1"
replicas: 1
# Core secret from ExternalSecret
secret: ""
existingSecret: "harbor-credentials"
secretName: "CORE_SECRET"
# Enable HTTP/2 for better performance
configureUserSettings: |-
http2_push_preload on;
resources:
requests:
memory: 512Mi
cpu: 500m
limits:
memory: 1Gi
cpu: 1000m
podAnnotations: {}
podLabels: {}
# Job service configuration - single replica
jobservice:
image:
repository: docker.io/goharbor/harbor-jobservice
tag: "v2.15.1"
replicas: 1
# Job service secret from ExternalSecret
secret: ""
existingSecret: "harbor-credentials"
existingSecretKey: "JOBSERVICE_SECRET"
resources:
requests:
memory: 512Mi
cpu: 500m
limits:
memory: 1Gi
cpu: 1000m
podAnnotations: {}
podLabels: {}
# Registry configuration - single replica
registry:
image:
repository: docker.io/goharbor/registry-photon
tag: "v2.15.1"
replicas: 1
# Registry credentials from ExternalSecret
credentials:
username: "harbor_registry_user"
password: ""
existingSecret: "harbor-credentials"
resources:
requests:
memory: 1Gi
cpu: 500m
limits:
memory: 2Gi
cpu: 1000m
podAnnotations: {}
podLabels: {}
# Registry controller
registryctl:
image:
repository: docker.io/goharbor/harbor-registryctl
tag: "v2.15.1"
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 512Mi
cpu: 500m
# Trivy vulnerability scanner
trivy:
enabled: true
image:
repository: docker.io/goharbor/trivy-adapter-photon
tag: "v2.15.1"
resources:
requests:
memory: 512Mi
cpu: 500m
limits:
memory: 1Gi
cpu: 1000m
podAnnotations: {}
podLabels: {}
# Exporter for metrics
exporter:
image:
repository: docker.io/goharbor/harbor-exporter
tag: "v2.15.1"
replicas: 1
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 512Mi
cpu: 500m
podAnnotations: {}
podLabels: {}
# Metrics configuration
metrics:
enabled: true
core:
path: /metrics
port: 8001
registry:
path: /metrics
port: 8001
jobservice:
path: /metrics
port: 8001
exporter:
path: /metrics
port: 8001
# Cache configuration (for performance)
cache:
enabled: true
expireHours: 24
# Update strategy
updateStrategy:
type: RollingUpdate
# NGINX (proxy in front of Harbor)
nginx:
image:
repository: docker.io/goharbor/nginx-photon
tag: "v2.15.1"
replicas: 1
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 512Mi
cpu: 500m
podAnnotations: {}
podLabels: {}