fastpass GitOps Bootstrap

This document is the complete procedure to go from a freshly provisioned Talos cluster to a fully GitOps-managed platform. After completing these steps, all future changes are made via git — never kubectl apply again.

Prerequisites

Talos cluster running (space-mountain, big-thunder-mountain, splash-mountain CP nodes)
Cilium installed as CNI (done at Talos bootstrap via talhelper)
kubectl configured and pointing at fastpass
kubeconfig on city-hall at ~/.kube/config
1Password Connect credentials JSON downloaded from your 1Password account (Developer Tools > Connect Servers > New Server)
1Password Connect token generated for the mk-labs vault

Step 1 — Install ArgoCD

This is the only kubectl apply you will ever run against this cluster.

kubectl create namespace argocd

kubectl apply -n argocd \
  --server-side \
  --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.2/manifests/install.yaml

# Wait for ArgoCD to be healthy
kubectl wait --for=condition=available deployment/argocd-server \
  -n argocd --timeout=120s

Step 2 — Bootstrap the 1Password Connect secret

This is the only secret you will ever manage manually. Everything else flows through ESO from here.

# Create the namespace first
kubectl create namespace onepassword-connect

# Apply the credentials file (downloaded from 1Password)
kubectl create secret generic op-credentials \
  --from-file=1password-credentials.json=/path/to/1password-credentials.json \
  -n onepassword-connect

# Apply the Connect API token
kubectl create secret generic connect-token \
  --from-literal=token=<your-connect-token> \
  -n onepassword-connect

Step 3 — Apply the App of Apps

This is the second and final manual apply. From this point on, ArgoCD manages everything, including itself.

kubectl apply -n argocd -f cluster/argocd/apps-of-apps.yaml

Step 4 — Watch it go

# Watch all Applications come up in wave order
kubectl get applications -n argocd -w

# Or use the ArgoCD UI
kubectl port-forward svc/argocd-server -n argocd 8080:443
# Then open https://localhost:8080
# Default admin password:
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d

Step 2.5 — Vendor Gateway API CRDs

Before applying the App of Apps, vendor the Gateway API CRDs into git. Run from the repo root on city-hall:

curl -L https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml \
  -o cluster/platform/gateway-api/manifests/standard-install.yaml

curl -L https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/experimental-install.yaml \
  -o cluster/platform/gateway-api/manifests/experimental-install.yaml

git add cluster/platform/gateway-api/manifests/
git commit -m "feat(platform): vendor gateway-api CRDs v1.2.1"
git push

Wave Order

Wave	Application	Depends On
1	external-secrets	(none)
2	onepassword-connect	external-secrets
3	cert-manager	onepassword-connect (CF secret)
3	gateway-api-crds	(none — CRDs only)
4	cilium-config	gateway-api-crds (CRDs exist)
5	ingress-nginx	cilium-config (LB pool)
5	gateway (resource)	cilium-config (GatewayClass)
6	external-dns	ingress-nginx (IP to register)
10+	applications/*	platform complete

Upgrading ArgoCD

kubectl apply -n argocd \
  --server-side \
  --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/v<NEW_VERSION>/manifests/install.yaml

Update the version comment in cluster/argocd/install.yaml to match.

Troubleshooting

# Check sync status of all apps
kubectl get applications -n argocd

# Check why an app is out of sync
kubectl describe application <name> -n argocd

# Force a sync
kubectl patch application <name> -n argocd \
  --type merge \
  -p '{"operation": {"initiatedBy": {"username": "admin"}, "sync": {}}}'

# Check ESO secret sync
kubectl get externalsecret -A
kubectl describe externalsecret <name> -n <namespace>

4.3 KiB Raw Blame History