Files
homelab/cluster/platform/harbor
Hermes Agent service account 5a99928c6f Add ServiceMonitors and Grafana dashboards for Harbor, External-DNS, and Ingress-NGINX
- Created ServiceMonitor for Harbor (harbor-core, harbor-exporter, harbor-jobservice, harbor-registry)
  - Port: http-metrics (8001)
  - Scrape interval: 30s
  - Verified metrics: harbor_core_http_request_duration_seconds_count

- Created ServiceMonitor for External-DNS
  - Port: http (7979)
  - Scrape interval: 30s
  - Verified metrics: external_dns_registry_endpoints_total

- Created ServiceMonitor for Ingress-NGINX
  - Port: metrics (10254)
  - Scrape interval: 30s
  - Verified metrics: nginx_ingress_controller_requests

- Added Grafana dashboards:
  - Harbor Overview (dashboard 16366)
  - External-DNS (dashboard 15038)
  - Ingress-NGINX Controller (dashboard 9614)

All ServiceMonitors deployed and actively scraping. Prometheus targets confirmed UP.
2026-06-19 18:26:54 -05:00
..

Harbor Container Registry

Enterprise-class container registry deployed on the fastpass Kubernetes cluster.

Access

Credentials

  • Admin: Stored in 1Password (mk-labs vault)
  • Robot Accounts: Managed via robot-accounts-sync.yaml Job

Robot Accounts

Two system-level robot accounts for automated operations:

tekton-builder (robot$tekton-builder)

  • Purpose: Tekton CI/CD pipeline builds
  • Permissions: Push + Pull + Read artifacts (all projects)
  • Usage: Push built container images from pipelines

fastpass-cluster (robot$fastpass-cluster)

  • Purpose: Kubernetes cluster image pulls
  • Permissions: Pull + Read artifacts (all projects)
  • Usage: Pull images for pod deployments

Syncing Robot Accounts

Robot accounts are defined in robot-accounts-sync.yaml as an idempotent Kubernetes Job.

To sync robot accounts after Harbor deployment or configuration changes:

kubectl apply -f cluster/platform/harbor/robot-accounts-sync.yaml
kubectl logs -n harbor -l job-name=harbor-robot-accounts-sync -f

The Job:

  • Creates missing robot accounts
  • Skips existing accounts (idempotent)
  • Uses harbor-credentials ExternalSecret for admin password
  • Can be run repeatedly without side effects

Note: ArgoCD PostSync hooks don't currently trigger with multi-source applications, so the Job must be run manually after initial deployment or when robot account definitions change.

Managing Robot Credentials

Robot account secrets are managed through 1Password and synced via ExternalSecrets:

  1. Initial Setup: After running robot-accounts-sync.yaml, regenerate secrets to capture full values:

    # Regenerate tekton-builder secret (robot ID: 3)
    curl -k -u "admin:${HARBOR_ADMIN_PASSWORD}" -X PATCH \
      "https://the-seas.local.mk-labs.cloud/api/v2.0/robots/3" \
      -H "Content-Type: application/json" \
      -d '{"secret": ""}' | jq -r '.secret'
    
    # Regenerate fastpass-cluster secret (robot ID: 4)
    curl -k -u "admin:${HARBOR_ADMIN_PASSWORD}" -X PATCH \
      "https://the-seas.local.mk-labs.cloud/api/v2.0/robots/4" \
      -H "Content-Type: application/json" \
      -d '{"secret": ""}' | jq -r '.secret'
    
  2. Store in 1Password: Create item harbor-robot-accounts in mk-labs vault with fields:

    • tekton-builder-username: robot$tekton-builder
    • tekton-builder-password: (secret from above)
    • fastpass-cluster-username: robot$fastpass-cluster
    • fastpass-cluster-password: (secret from above)
  3. ExternalSecrets: Two ExternalSecrets automatically sync credentials from 1Password:

    • externalsecret-tekton-robot.yaml → creates harbor-tekton-robot secret
    • externalsecret-pull-secret.yaml → creates harbor-pull-secret secret

    Both create kubernetes.io/dockerconfigjson type secrets ready for use.

Network & TLS

DNS Name

  • Hostname: the-seas.local.mk-labs.cloud

DNS resolves to the NGINX Ingress Controller IP (10.1.71.80).

TLS Certificate

  • Issued by Let's Encrypt (letsencrypt-prod)
  • Managed automatically by cert-manager via Ingress annotation
  • Certificate name: harbor-tls
  • Auto-renewed before expiration

Routing

  • Ingress Controller: NGINX (nginx-ingress)
  • Protocol: HTTPS (port 443)
  • Backend: Harbor NGINX service on port 80
  • TLS termination at Ingress level

Architecture

  • Storage Backend: NFS CSI (nfs.csi.k8s.io)
    • Registry: 10Gi PVC
    • Jobservice: 1Gi PVC
  • Database: Embedded PostgreSQL (StatefulSet)
  • Cache: Embedded Redis (StatefulSet)
  • Vulnerability Scanning: Trivy (StatefulSet)

Docker Client Configuration

Trust Harbor Certificate

# Copy Harbor CA certificate from Kubernetes
kubectl get secret harbor-tls -n harbor -o jsonpath='{.data.tls\.crt}' | base64 -d > harbor-ca.crt

# Install to system trust store
sudo cp harbor-ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

# Install for Docker registry-specific trust
sudo mkdir -p /etc/docker/certs.d/the-seas.local.mk-labs.cloud
sudo cp harbor-ca.crt /etc/docker/certs.d/the-seas.local.mk-labs.cloud/ca.crt

Login and Test

docker login the-seas.local.mk-labs.cloud
docker pull alpine:latest
docker tag alpine:latest the-seas.local.mk-labs.cloud/library/alpine:test
docker push the-seas.local.mk-labs.cloud/library/alpine:test

Kubernetes ImagePullSecret

Create a Secret for the fastpass-cluster robot account:

kubectl create secret docker-registry harbor-pull-secret \
  --docker-server=the-seas.local.mk-labs.cloud \
  --docker-username='robot$fastpass-cluster' \
  --docker-password='<secret-from-1password>' \
  --namespace=<target-namespace>

Or manage via ExternalSecret referencing 1Password.

Deployment Details

  • Helm Chart: goharbor/harbor v1.19.1 (Harbor v2.15.1)
  • Deployment Method: ArgoCD multi-source (Helm chart + repo manifests)
  • Sync Policy: Automated (prune + selfHeal enabled)
  • Sync Wave: 7 (depends on cert-manager, external-dns, ESO)

References