rblundon/homelab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update Harbor README with robot accounts documentation

Browse Source 
Documents robot account management via GitOps:
- tekton-builder and fastpass-cluster robot accounts
- Manual Job execution (PostSync hooks don't work with multi-source)
- Credential retrieval and storage in 1Password
- ImagePullSecret creation for K8s clusters
This commit is contained in:
Hermes Agent service account
2026-06-04 23:38:50 -05:00
parent 152f10ed8b
commit 8d18f42b2e
1 changed files with 86 additions and 284 deletions
Download Patch File Download Diff File Expand all files Collapse all files

370
cluster/platform/harbor/README.md
View File

@@ -1,89 +1,62 @@
# The Seas - Harbor Container Registry
# Harbor Container Registry
**EPCOT Theme:** The Seas with Nemo & Friends
**Service:** Harbor Container Registry
**Wave:** 7
Enterprise-class container registry deployed on the fastpass Kubernetes cluster.
## Overview
Harbor is an open-source container registry that provides trust, compliance, performance, and interoperability. It serves as the central image repository for all custom-built containers in the mk-labs homelab.
## Service Details
## Access
- **URL:** https://the-seas.local.mk-labs.cloud
- **Namespace:** harbor
- **Sync Wave:** 7 (deploys after Gateway API, cert-manager, ExternalSecrets)
## Architecture
## Credentials
### Deployment Model
- **Database:** Embedded PostgreSQL (internal to Harbor)
- **Cache:** Embedded Redis (internal to Harbor)
- **HA Mode:** Single instance (no HA for homelab simplicity)
- **Storage:** NFS via nfs-emporium StorageClass
- **Admin:** Stored in 1Password (mk-labs vault)
- **Robot Accounts:** Managed via robot-accounts-sync.yaml Job
### Versions
- **Helm Chart:** 1.19.1
- **Harbor:** v2.15.1
- **All component images:** Pinned to 2.15.1 tag
## Robot Accounts
## Components
Two system-level robot accounts for automated operations:
Harbor consists of several microservices:
### tekton-builder (robot$tekton-builder)
- **Purpose:** Tekton CI/CD pipeline builds
- **Permissions:** Push + Pull + Read artifacts (all projects)
- **Usage:** Push built container images from pipelines
1. **Portal** - Web UI (1 replica)
2. **Core** - Main API server (1 replica)
3. **Registry** - OCI/Docker registry backend (1 replica)
4. **Job Service** - Async job processor (1 replica)
5. **Database** - PostgreSQL for metadata (embedded)
6. **Redis** - Cache and job queue (embedded)
7. **Trivy** - Vulnerability scanner (enabled)
8. **Exporter** - Prometheus metrics exporter (enabled)
9. **NGINX** - Reverse proxy frontend (1 replica)
### fastpass-cluster (robot$fastpass-cluster)
- **Purpose:** Kubernetes cluster image pulls
- **Permissions:** Pull + Read artifacts (all projects)
- **Usage:** Pull images for pod deployments
## Storage Configuration
### Syncing Robot Accounts
| Component | StorageClass | Size | Purpose |
|-----------|-------------|------|---------|
| Registry | nfs-emporium | 100Gi | Container images and Helm charts |
| Database | nfs-emporium | 10Gi | Metadata and configuration |
| Redis | nfs-emporium | 5Gi | Cache and job queue persistence |
| JobService | nfs-emporium | 5Gi | Job logs |
| Trivy | nfs-emporium | 10Gi | Vulnerability database |
Robot accounts are defined in `robot-accounts-sync.yaml` as an idempotent Kubernetes Job.
**Total:** ~130Gi storage provisioned
**To sync robot accounts after Harbor deployment or configuration changes:**
## Resource Allocation
```bash
kubectl apply -f cluster/platform/harbor/robot-accounts-sync.yaml
kubectl logs -n harbor -l job-name=harbor-robot-accounts-sync -f
```
Approximate resource requests/limits per component:
The Job:
- Creates missing robot accounts
- Skips existing accounts (idempotent)
- Uses harbor-credentials ExternalSecret for admin password
- Can be run repeatedly without side effects
- **Core:** 512Mi-1Gi RAM, 0.5-1 CPU
- **Registry:** 1-2Gi RAM, 0.5-1 CPU
- **Database:** 1-2Gi RAM, 0.5-2 CPU
- **Redis:** 256-512Mi RAM, 0.25-0.5 CPU
- **Portal:** 256-512Mi RAM, 0.25-0.5 CPU
- **JobService:** 512Mi-1Gi RAM, 0.5-1 CPU
- **Trivy:** 512Mi-1Gi RAM, 0.5-1 CPU
- **Exporter:** 256-512Mi RAM, 0.25-0.5 CPU
- **NGINX:** 256-512Mi RAM, 0.25-0.5 CPU
**Note:** ArgoCD PostSync hooks don't currently trigger with multi-source applications, so the Job must be run manually after initial deployment or when robot account definitions change.
**Total:** ~6-10Gi RAM, ~4-8 CPU cores across all pods
### Retrieving Robot Credentials
## Secrets Management
Robot account secrets are only returned on creation. To regenerate a secret:
All credentials are stored in 1Password and synchronized to Kubernetes via ExternalSecrets Operator.
```bash
curl -k -u "admin:${HARBOR_ADMIN_PASSWORD}" -X PATCH \
"https://the-seas.local.mk-labs.cloud/api/v2.0/robots/<robot_id>" \
-H "Content-Type: application/json" \
-d '{"secret": ""}' | jq -r '.secret'
```
**1Password Item:** `the-seas` (vault: `mk-labs`)
**Required Secrets:**
1. `harbor-admin-password` - Web UI admin user password
2. `database-password` - PostgreSQL password
3. `redis-password` - Redis password
4. `core-secret` - Harbor core component secret
5. `jobservice-secret` - Job service authentication secret
6. `registry-password` - Registry backend password
**Kubernetes Secret:** `the-seas-credentials` (namespace: the-seas)
Store secrets in 1Password (mk-labs vault, item: `harbor-robot-accounts`).
## Network & TLS
@@ -93,9 +66,10 @@ All credentials are stored in 1Password and synchronized to Kubernetes via Exter
DNS resolves to the NGINX Ingress Controller IP (10.1.71.80).
### TLS Certificate
- Issued by Let's Encrypt (letsencrypt-staging ClusterIssuer)
- Managed by cert-manager (automatic via Ingress annotation)
- Certificate stored as Kubernetes secret: `harbor-tls`
- Issued by Let's Encrypt (letsencrypt-prod)
- Managed automatically by cert-manager via Ingress annotation
- Certificate name: harbor-tls
- Auto-renewed before expiration
### Routing
- **Ingress Controller:** NGINX (nginx-ingress)
@@ -103,236 +77,64 @@ DNS resolves to the NGINX Ingress Controller IP (10.1.71.80).
- **Backend:** Harbor NGINX service on port 80
- TLS termination at Ingress level
## Access & Authentication
## Architecture
### Web UI
1. Navigate to https://the-seas.local.mk-labs.cloud or https://harbor.local.mk-labs.cloud
2. Username: `admin`
3. Password: From 1Password item `the-seas`, field `harbor-admin-password`
- **Storage Backend:** NFS CSI (nfs.csi.k8s.io)
- Registry: 10Gi PVC
- Jobservice: 1Gi PVC
- **Database:** Embedded PostgreSQL (StatefulSet)
- **Cache:** Embedded Redis (StatefulSet)
- **Vulnerability Scanning:** Trivy (StatefulSet)
## Docker Client Configuration
### Trust Harbor Certificate
```bash
# Copy Harbor CA certificate from Kubernetes
kubectl get secret harbor-tls -n harbor -o jsonpath='{.data.tls\.crt}' | base64 -d > harbor-ca.crt
# Install to system trust store
sudo cp harbor-ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
# Install for Docker registry-specific trust
sudo mkdir -p /etc/docker/certs.d/the-seas.local.mk-labs.cloud
sudo cp harbor-ca.crt /etc/docker/certs.d/the-seas.local.mk-labs.cloud/ca.crt
```
### Login and Test
### Docker CLI
```bash
# Login to registry
docker login the-seas.local.mk-labs.cloud
# or
docker login harbor.local.mk-labs.cloud
# Username: admin
# Password: <from 1Password>
```
### Robot Accounts
For CI/CD automation, robot accounts are created post-deployment:
1. **tekton-builder** - For Tekton CI/CD image pushes
- Permissions: Push artifacts, Create tags
- Token stored in 1Password: `the-seas-tekton-robot`
2. **argocd-puller** - For ArgoCD image metadata reads
- Permissions: Pull artifacts, Read metadata (read-only)
- Token stored in 1Password: `the-seas-argocd-robot`
## Initial Project Structure
Harbor organizes images into projects. Initial projects:
1. **library** - General-purpose images (default)
2. **platform** - Infrastructure/platform images (Tekton, etc.)
3. **applications** - Application images (Firecrawl, etc.)
All projects are private by default.
## Deployment
### Prerequisites
- ArgoCD deployed and managing cluster
- Gateway API (wave 4) deployed
- cert-manager (wave 3) deployed
- ExternalSecrets Operator (wave 1) deployed
- 1Password Connect (wave 2) deployed
- 1Password item `the-seas` created with required secrets
### Apply
```bash
# From repository root
kubectl apply -f cluster/platform/the-seas/application.yaml
# ArgoCD will sync automatically
```
### Verify
```bash
# Check ArgoCD application status
kubectl get application -n argocd the-seas
# Check all pods are running
kubectl get pods -n the-seas
# Check PVCs are bound
kubectl get pvc -n the-seas
# Check ExternalSecret synced
kubectl get externalsecret -n the-seas the-seas-credentials
# Check certificates issued
kubectl get certificate -n the-seas
```
## Testing
### Basic Functionality
```bash
# 1. Web UI access
curl -k https://the-seas.local.mk-labs.cloud
# Should return Harbor web UI HTML
# 2. Docker login
docker login the-seas.local.mk-labs.cloud
# Should prompt for credentials and succeed
# 3. Push test image
docker pull alpine:latest
docker tag alpine:latest the-seas.local.mk-labs.cloud/library/alpine:test
docker push the-seas.local.mk-labs.cloud/library/alpine:test
# 4. Pull test image
docker rmi the-seas.local.mk-labs.cloud/library/alpine:test
docker pull the-seas.local.mk-labs.cloud/library/alpine:test
# 5. Verify in web UI
# Navigate to Projects → library → Repositories
# Should see alpine image with test tag
```
### Certificate Validation
## Kubernetes ImagePullSecret
Create a Secret for the fastpass-cluster robot account:
```bash
# Check TLS certificate
echo | openssl s_client -connect the-seas.local.mk-labs.cloud:443 2>/dev/null | openssl x509 -noout -text | grep -A 2 "Subject:"
# Verify certificate is valid
echo | openssl s_client -connect the-seas.local.mk-labs.cloud:443 2>/dev/null | grep "Verify return code"
# Should show: Verify return code: 0 (ok)
kubectl create secret docker-registry harbor-pull-secret \
--docker-server=the-seas.local.mk-labs.cloud \
--docker-username='robot$fastpass-cluster' \
--docker-password='<secret-from-1password>' \
--namespace=<target-namespace>
```
## Monitoring
Or manage via ExternalSecret referencing 1Password.
Harbor exports Prometheus metrics on port 8001 for several components:
- Core: `/metrics`
- Registry: `/metrics`
- JobService: `/metrics`
- Exporter: `/metrics`
## Deployment Details
Metrics are enabled in values.yaml and can be scraped by Prometheus.
## Troubleshooting
### Pods CrashLoopBackOff
Check pod logs:
```bash
kubectl logs -n the-seas <pod-name>
```
Common issues:
- **Database init failures:** Check database password in ExternalSecret
- **Redis connection errors:** Check redis password in ExternalSecret
- **Secret not found:** Verify ExternalSecret synced successfully
### Cannot Access Web UI
1. Check Gateway and HTTPRoute:
```bash
kubectl get gateway -n gateway fastpass-gateway
kubectl get httproute -n the-seas
```
2. Check DNS resolution:
```bash
nslookup the-seas.local.mk-labs.cloud
# Should resolve to 10.1.71.90
```
3. Check certificate:
```bash
kubectl get certificate -n the-seas the-seas-tls
# Should show Ready=True
```
### Docker Login Fails
1. Check DNS resolution
2. Check certificate is trusted
3. Verify admin password from 1Password is correct
4. Check Harbor core service is running:
```bash
kubectl get svc -n the-seas the-seas
kubectl get pods -n the-seas -l component=core
```
### Image Push/Pull Fails
1. Check registry pod:
```bash
kubectl get pods -n the-seas -l component=registry
kubectl logs -n the-seas -l component=registry
```
2. Check storage PVC is bound:
```bash
kubectl get pvc -n the-seas
```
3. Check project permissions in Harbor UI
## Maintenance
### Backup
Harbor data is persisted on NFS volumes. Ensure NFS storage is backed up regularly.
Critical data:
- `/storage` on registry PVC - All container images
- Database PVC - All metadata, users, projects, policies
### Updates
Harbor is managed by ArgoCD. To update:
1. Update `targetRevision` in `application.yaml` to new chart version
2. Review changelog: https://github.com/goharbor/harbor/releases
3. Update image tags in `values.yaml` if needed
4. Commit and push to Git
5. ArgoCD will automatically sync
### Scaling
Current deployment is single-instance. To scale:
1. Update replica counts in `values.yaml`:
- `portal.replicas`
- `core.replicas`
- `registry.replicas`
- `jobservice.replicas`
2. Consider external PostgreSQL and Redis for HA
- **Helm Chart:** goharbor/harbor v1.19.1 (Harbor v2.15.1)
- **Deployment Method:** ArgoCD multi-source (Helm chart + repo manifests)
- **Sync Policy:** Automated (prune + selfHeal enabled)
- **Sync Wave:** 7 (depends on cert-manager, external-dns, ESO)
## References
- **Harbor Documentation:** https://goharbor.io/docs/
- **Harbor Helm Chart:** https://github.com/goharbor/harbor-helm
- **Repository Path:** `/home/hermes/git/homelab/cluster/platform/the-seas/`
- **ArgoCD Application:** https://argocd.mk-labs.cloud/applications/the-seas
- **1Password Vault:** mk-labs
- **1Password Item:** the-seas
## Next Steps
1. **Phase 2:** Integrate with Tekton (innoventions) for CI/CD
2. **Phase 3:** Configure vulnerability scanning policies
3. **Phase 4:** Implement image signing with Cosign (Notary v2)
4. **Phase 5:** Set up replication to backup registry (if needed)
## Support
For issues or questions:
- Check troubleshooting section above
- Review Harbor logs: `kubectl logs -n the-seas <pod-name>`
- Consult Harbor documentation: https://goharbor.io/docs/
- Escalate to Ryan via `/mnt/mk-labs-pka/ryan-inbox/`
- Upstream Helm Chart: https://github.com/goharbor/harbor-helm
- Harbor Documentation: https://goharbor.io/docs/
- API Reference: https://the-seas.local.mk-labs.cloud/devcenter-api-2.0