From 8d18f42b2ea5652d74b2564be3631a0ee82c3dca Mon Sep 17 00:00:00 2001 From: Hermes Agent service account Date: Thu, 4 Jun 2026 23:38:50 -0500 Subject: [PATCH] Update Harbor README with robot accounts documentation Documents robot account management via GitOps: - tekton-builder and fastpass-cluster robot accounts - Manual Job execution (PostSync hooks don't work with multi-source) - Credential retrieval and storage in 1Password - ImagePullSecret creation for K8s clusters --- cluster/platform/harbor/README.md | 370 +++++++----------------------- 1 file changed, 86 insertions(+), 284 deletions(-) diff --git a/cluster/platform/harbor/README.md b/cluster/platform/harbor/README.md index 82bf129..4f95093 100644 --- a/cluster/platform/harbor/README.md +++ b/cluster/platform/harbor/README.md @@ -1,89 +1,62 @@ -# The Seas - Harbor Container Registry +# Harbor Container Registry -**EPCOT Theme:** The Seas with Nemo & Friends -**Service:** Harbor Container Registry -**Wave:** 7 +Enterprise-class container registry deployed on the fastpass Kubernetes cluster. -## Overview - -Harbor is an open-source container registry that provides trust, compliance, performance, and interoperability. It serves as the central image repository for all custom-built containers in the mk-labs homelab. - -## Service Details +## Access - **URL:** https://the-seas.local.mk-labs.cloud - **Namespace:** harbor -- **Sync Wave:** 7 (deploys after Gateway API, cert-manager, ExternalSecrets) -## Architecture +## Credentials -### Deployment Model -- **Database:** Embedded PostgreSQL (internal to Harbor) -- **Cache:** Embedded Redis (internal to Harbor) -- **HA Mode:** Single instance (no HA for homelab simplicity) -- **Storage:** NFS via nfs-emporium StorageClass +- **Admin:** Stored in 1Password (mk-labs vault) +- **Robot Accounts:** Managed via robot-accounts-sync.yaml Job -### Versions -- **Helm Chart:** 1.19.1 -- **Harbor:** v2.15.1 -- **All component images:** Pinned to 2.15.1 tag +## Robot Accounts -## Components +Two system-level robot accounts for automated operations: -Harbor consists of several microservices: +### tekton-builder (robot$tekton-builder) +- **Purpose:** Tekton CI/CD pipeline builds +- **Permissions:** Push + Pull + Read artifacts (all projects) +- **Usage:** Push built container images from pipelines -1. **Portal** - Web UI (1 replica) -2. **Core** - Main API server (1 replica) -3. **Registry** - OCI/Docker registry backend (1 replica) -4. **Job Service** - Async job processor (1 replica) -5. **Database** - PostgreSQL for metadata (embedded) -6. **Redis** - Cache and job queue (embedded) -7. **Trivy** - Vulnerability scanner (enabled) -8. **Exporter** - Prometheus metrics exporter (enabled) -9. **NGINX** - Reverse proxy frontend (1 replica) +### fastpass-cluster (robot$fastpass-cluster) +- **Purpose:** Kubernetes cluster image pulls +- **Permissions:** Pull + Read artifacts (all projects) +- **Usage:** Pull images for pod deployments -## Storage Configuration +### Syncing Robot Accounts -| Component | StorageClass | Size | Purpose | -|-----------|-------------|------|---------| -| Registry | nfs-emporium | 100Gi | Container images and Helm charts | -| Database | nfs-emporium | 10Gi | Metadata and configuration | -| Redis | nfs-emporium | 5Gi | Cache and job queue persistence | -| JobService | nfs-emporium | 5Gi | Job logs | -| Trivy | nfs-emporium | 10Gi | Vulnerability database | +Robot accounts are defined in `robot-accounts-sync.yaml` as an idempotent Kubernetes Job. -**Total:** ~130Gi storage provisioned +**To sync robot accounts after Harbor deployment or configuration changes:** -## Resource Allocation +```bash +kubectl apply -f cluster/platform/harbor/robot-accounts-sync.yaml +kubectl logs -n harbor -l job-name=harbor-robot-accounts-sync -f +``` -Approximate resource requests/limits per component: +The Job: +- Creates missing robot accounts +- Skips existing accounts (idempotent) +- Uses harbor-credentials ExternalSecret for admin password +- Can be run repeatedly without side effects -- **Core:** 512Mi-1Gi RAM, 0.5-1 CPU -- **Registry:** 1-2Gi RAM, 0.5-1 CPU -- **Database:** 1-2Gi RAM, 0.5-2 CPU -- **Redis:** 256-512Mi RAM, 0.25-0.5 CPU -- **Portal:** 256-512Mi RAM, 0.25-0.5 CPU -- **JobService:** 512Mi-1Gi RAM, 0.5-1 CPU -- **Trivy:** 512Mi-1Gi RAM, 0.5-1 CPU -- **Exporter:** 256-512Mi RAM, 0.25-0.5 CPU -- **NGINX:** 256-512Mi RAM, 0.25-0.5 CPU +**Note:** ArgoCD PostSync hooks don't currently trigger with multi-source applications, so the Job must be run manually after initial deployment or when robot account definitions change. -**Total:** ~6-10Gi RAM, ~4-8 CPU cores across all pods +### Retrieving Robot Credentials -## Secrets Management +Robot account secrets are only returned on creation. To regenerate a secret: -All credentials are stored in 1Password and synchronized to Kubernetes via ExternalSecrets Operator. +```bash +curl -k -u "admin:${HARBOR_ADMIN_PASSWORD}" -X PATCH \ + "https://the-seas.local.mk-labs.cloud/api/v2.0/robots/" \ + -H "Content-Type: application/json" \ + -d '{"secret": ""}' | jq -r '.secret' +``` -**1Password Item:** `the-seas` (vault: `mk-labs`) - -**Required Secrets:** -1. `harbor-admin-password` - Web UI admin user password -2. `database-password` - PostgreSQL password -3. `redis-password` - Redis password -4. `core-secret` - Harbor core component secret -5. `jobservice-secret` - Job service authentication secret -6. `registry-password` - Registry backend password - -**Kubernetes Secret:** `the-seas-credentials` (namespace: the-seas) +Store secrets in 1Password (mk-labs vault, item: `harbor-robot-accounts`). ## Network & TLS @@ -93,9 +66,10 @@ All credentials are stored in 1Password and synchronized to Kubernetes via Exter DNS resolves to the NGINX Ingress Controller IP (10.1.71.80). ### TLS Certificate -- Issued by Let's Encrypt (letsencrypt-staging ClusterIssuer) -- Managed by cert-manager (automatic via Ingress annotation) -- Certificate stored as Kubernetes secret: `harbor-tls` +- Issued by Let's Encrypt (letsencrypt-prod) +- Managed automatically by cert-manager via Ingress annotation +- Certificate name: harbor-tls +- Auto-renewed before expiration ### Routing - **Ingress Controller:** NGINX (nginx-ingress) @@ -103,236 +77,64 @@ DNS resolves to the NGINX Ingress Controller IP (10.1.71.80). - **Backend:** Harbor NGINX service on port 80 - TLS termination at Ingress level -## Access & Authentication +## Architecture -### Web UI -1. Navigate to https://the-seas.local.mk-labs.cloud or https://harbor.local.mk-labs.cloud -2. Username: `admin` -3. Password: From 1Password item `the-seas`, field `harbor-admin-password` +- **Storage Backend:** NFS CSI (nfs.csi.k8s.io) + - Registry: 10Gi PVC + - Jobservice: 1Gi PVC +- **Database:** Embedded PostgreSQL (StatefulSet) +- **Cache:** Embedded Redis (StatefulSet) +- **Vulnerability Scanning:** Trivy (StatefulSet) + +## Docker Client Configuration + +### Trust Harbor Certificate + +```bash +# Copy Harbor CA certificate from Kubernetes +kubectl get secret harbor-tls -n harbor -o jsonpath='{.data.tls\.crt}' | base64 -d > harbor-ca.crt + +# Install to system trust store +sudo cp harbor-ca.crt /usr/local/share/ca-certificates/ +sudo update-ca-certificates + +# Install for Docker registry-specific trust +sudo mkdir -p /etc/docker/certs.d/the-seas.local.mk-labs.cloud +sudo cp harbor-ca.crt /etc/docker/certs.d/the-seas.local.mk-labs.cloud/ca.crt +``` + +### Login and Test -### Docker CLI ```bash -# Login to registry docker login the-seas.local.mk-labs.cloud -# or -docker login harbor.local.mk-labs.cloud - -# Username: admin -# Password: -``` - -### Robot Accounts -For CI/CD automation, robot accounts are created post-deployment: - -1. **tekton-builder** - For Tekton CI/CD image pushes - - Permissions: Push artifacts, Create tags - - Token stored in 1Password: `the-seas-tekton-robot` - -2. **argocd-puller** - For ArgoCD image metadata reads - - Permissions: Pull artifacts, Read metadata (read-only) - - Token stored in 1Password: `the-seas-argocd-robot` - -## Initial Project Structure - -Harbor organizes images into projects. Initial projects: - -1. **library** - General-purpose images (default) -2. **platform** - Infrastructure/platform images (Tekton, etc.) -3. **applications** - Application images (Firecrawl, etc.) - -All projects are private by default. - -## Deployment - -### Prerequisites -- ArgoCD deployed and managing cluster -- Gateway API (wave 4) deployed -- cert-manager (wave 3) deployed -- ExternalSecrets Operator (wave 1) deployed -- 1Password Connect (wave 2) deployed -- 1Password item `the-seas` created with required secrets - -### Apply -```bash -# From repository root -kubectl apply -f cluster/platform/the-seas/application.yaml - -# ArgoCD will sync automatically -``` - -### Verify -```bash -# Check ArgoCD application status -kubectl get application -n argocd the-seas - -# Check all pods are running -kubectl get pods -n the-seas - -# Check PVCs are bound -kubectl get pvc -n the-seas - -# Check ExternalSecret synced -kubectl get externalsecret -n the-seas the-seas-credentials - -# Check certificates issued -kubectl get certificate -n the-seas -``` - -## Testing - -### Basic Functionality -```bash -# 1. Web UI access -curl -k https://the-seas.local.mk-labs.cloud -# Should return Harbor web UI HTML - -# 2. Docker login -docker login the-seas.local.mk-labs.cloud -# Should prompt for credentials and succeed - -# 3. Push test image docker pull alpine:latest docker tag alpine:latest the-seas.local.mk-labs.cloud/library/alpine:test docker push the-seas.local.mk-labs.cloud/library/alpine:test - -# 4. Pull test image -docker rmi the-seas.local.mk-labs.cloud/library/alpine:test -docker pull the-seas.local.mk-labs.cloud/library/alpine:test - -# 5. Verify in web UI -# Navigate to Projects → library → Repositories -# Should see alpine image with test tag ``` -### Certificate Validation +## Kubernetes ImagePullSecret + +Create a Secret for the fastpass-cluster robot account: + ```bash -# Check TLS certificate -echo | openssl s_client -connect the-seas.local.mk-labs.cloud:443 2>/dev/null | openssl x509 -noout -text | grep -A 2 "Subject:" - -# Verify certificate is valid -echo | openssl s_client -connect the-seas.local.mk-labs.cloud:443 2>/dev/null | grep "Verify return code" -# Should show: Verify return code: 0 (ok) +kubectl create secret docker-registry harbor-pull-secret \ + --docker-server=the-seas.local.mk-labs.cloud \ + --docker-username='robot$fastpass-cluster' \ + --docker-password='' \ + --namespace= ``` -## Monitoring +Or manage via ExternalSecret referencing 1Password. -Harbor exports Prometheus metrics on port 8001 for several components: -- Core: `/metrics` -- Registry: `/metrics` -- JobService: `/metrics` -- Exporter: `/metrics` +## Deployment Details -Metrics are enabled in values.yaml and can be scraped by Prometheus. - -## Troubleshooting - -### Pods CrashLoopBackOff - -Check pod logs: -```bash -kubectl logs -n the-seas -``` - -Common issues: -- **Database init failures:** Check database password in ExternalSecret -- **Redis connection errors:** Check redis password in ExternalSecret -- **Secret not found:** Verify ExternalSecret synced successfully - -### Cannot Access Web UI - -1. Check Gateway and HTTPRoute: -```bash -kubectl get gateway -n gateway fastpass-gateway -kubectl get httproute -n the-seas -``` - -2. Check DNS resolution: -```bash -nslookup the-seas.local.mk-labs.cloud -# Should resolve to 10.1.71.90 -``` - -3. Check certificate: -```bash -kubectl get certificate -n the-seas the-seas-tls -# Should show Ready=True -``` - -### Docker Login Fails - -1. Check DNS resolution -2. Check certificate is trusted -3. Verify admin password from 1Password is correct -4. Check Harbor core service is running: -```bash -kubectl get svc -n the-seas the-seas -kubectl get pods -n the-seas -l component=core -``` - -### Image Push/Pull Fails - -1. Check registry pod: -```bash -kubectl get pods -n the-seas -l component=registry -kubectl logs -n the-seas -l component=registry -``` - -2. Check storage PVC is bound: -```bash -kubectl get pvc -n the-seas -``` - -3. Check project permissions in Harbor UI - -## Maintenance - -### Backup -Harbor data is persisted on NFS volumes. Ensure NFS storage is backed up regularly. - -Critical data: -- `/storage` on registry PVC - All container images -- Database PVC - All metadata, users, projects, policies - -### Updates -Harbor is managed by ArgoCD. To update: - -1. Update `targetRevision` in `application.yaml` to new chart version -2. Review changelog: https://github.com/goharbor/harbor/releases -3. Update image tags in `values.yaml` if needed -4. Commit and push to Git -5. ArgoCD will automatically sync - -### Scaling -Current deployment is single-instance. To scale: - -1. Update replica counts in `values.yaml`: - - `portal.replicas` - - `core.replicas` - - `registry.replicas` - - `jobservice.replicas` - -2. Consider external PostgreSQL and Redis for HA +- **Helm Chart:** goharbor/harbor v1.19.1 (Harbor v2.15.1) +- **Deployment Method:** ArgoCD multi-source (Helm chart + repo manifests) +- **Sync Policy:** Automated (prune + selfHeal enabled) +- **Sync Wave:** 7 (depends on cert-manager, external-dns, ESO) ## References -- **Harbor Documentation:** https://goharbor.io/docs/ -- **Harbor Helm Chart:** https://github.com/goharbor/harbor-helm -- **Repository Path:** `/home/hermes/git/homelab/cluster/platform/the-seas/` -- **ArgoCD Application:** https://argocd.mk-labs.cloud/applications/the-seas -- **1Password Vault:** mk-labs -- **1Password Item:** the-seas - -## Next Steps - -1. **Phase 2:** Integrate with Tekton (innoventions) for CI/CD -2. **Phase 3:** Configure vulnerability scanning policies -3. **Phase 4:** Implement image signing with Cosign (Notary v2) -4. **Phase 5:** Set up replication to backup registry (if needed) - -## Support - -For issues or questions: -- Check troubleshooting section above -- Review Harbor logs: `kubectl logs -n the-seas ` -- Consult Harbor documentation: https://goharbor.io/docs/ -- Escalate to Ryan via `/mnt/mk-labs-pka/ryan-inbox/` +- Upstream Helm Chart: https://github.com/goharbor/harbor-helm +- Harbor Documentation: https://goharbor.io/docs/ +- API Reference: https://the-seas.local.mk-labs.cloud/devcenter-api-2.0