Files

Ryan Blundon 025e9a8f1c Added bootstrap folder and gateway controller

2026-05-18 18:07:35 -05:00

7.1 KiB

Raw Blame History

fastpass — Cluster Bootstrap

This directory contains resources and documentation for the manual bootstrap sequence required to bring up fastpass from a fresh Talos install.

Everything in this directory is applied once by hand before ArgoCD takes over. After Step 6 (app-of-apps), all further changes are made via Git.

Prerequisites

All 6 Talos nodes Ready (kubectl get nodes)
Cilium healthy (cilium status --wait)
IP pools applied (kubectl get ciliumloadbalancerippool)
city-hall has helm, kubectl, and talosctl installed
SSH keypair for ArgoCD → Gitea exists at /home/wed/.ssh/argocd_gitea
1Password credentials file at /home/wed/1password-credentials.json
1Password Connect token at /home/wed/connect-token (no trailing newline)

Step 1 — Install Gateway API CRDs

The Gateway API CRDs are not bundled with Cilium — they must be installed independently before Cilium can register the GatewayClass controller.

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml

Verify:

kubectl get crd gateways.gateway.networking.k8s.io

Step 2 — Create the Cilium GatewayClass

The Cilium operator manages this object but does not auto-create it.

kubectl apply -f cluster/bootstrap/gatewayclass.yaml

Verify (may take a few seconds):

kubectl get gatewayclass
# Expected: cilium   io.cilium/gateway-controller   True

Step 3 — Create the argocd namespace

Must be created before Helm — the Gitea repo secret targets this namespace.

kubectl apply -f cluster/argocd/namespace.yaml

Step 4 — Create the Gitea repo secret

kubectl create secret generic gitea-repo \
  --namespace argocd \
  --from-literal=type=git \
  --from-literal=url=git@gitea.mk-labs.cloud:rblundon/homelab.git \
  --from-file=sshPrivateKey=/home/wed/.ssh/argocd_gitea

kubectl label secret gitea-repo -n argocd \
  argocd.argoproj.io/secret-type=repository

Verify:

kubectl -n argocd get secret gitea-repo

Step 5 — Install ArgoCD via Helm

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

helm upgrade --install argocd argo/argo-cd \
  --namespace argocd \
  --create-namespace \
  --version 7.8.23 \
  --values cluster/argocd/values.yaml \
  --wait

Wait for all ArgoCD pods to be ready:

kubectl -n argocd get pods --watch

Step 6 — Add Gitea to ArgoCD known hosts

ssh-keyscan -t ed25519 gitea.mk-labs.cloud 2>/dev/null | \
  kubectl patch configmap argocd-ssh-known-hosts-cm -n argocd \
  --type merge \
  -p "{\"data\":{\"ssh_known_hosts\": \"$(ssh-keyscan -t ed25519 gitea.mk-labs.cloud 2>/dev/null)\"}}"

Step 7 — Get the initial admin password

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d; echo

Access the UI via port-forward (ingress not yet deployed):

kubectl port-forward svc/argocd-server -n argocd 8080:80 > /tmp/pf.log 2>&1 &

Browse to http://localhost:8080

Step 8 — Apply the app-of-apps

This is the final manual kubectl apply. It hands control to ArgoCD:

kubectl apply -n argocd -f cluster/argocd/apps-of-apps.yaml

From this point, all changes are made via Git. ArgoCD syncs cluster/platform/ and cluster/applications/ automatically.

Watch the sync:

kubectl -n argocd get applications --watch

Step 9 — Approve CSRs

kubectl get csr --no-headers | awk '{print $1}' | xargs kubectl certificate approve

Run periodically until no pending CSRs remain.

Step 10 — Bootstrap 1Password Connect secrets

Once onepassword-connect namespace exists (created by ArgoCD sync):

# Verify namespace exists
kubectl get namespace onepassword-connect

# Credentials secret — must be base64-encoded
kubectl create secret generic op-credentials \
  --namespace onepassword-connect \
  --from-literal=1password-credentials.json=$(base64 -w 0 /home/wed/1password-credentials.json)

# Token secret — file must have no trailing newline
kubectl create secret generic connect-token \
  --namespace onepassword-connect \
  --from-file=token=/home/wed/connect-token

Restart Connect to pick up secrets:

kubectl -n onepassword-connect rollout restart deployment onepassword-connect
kubectl -n onepassword-connect rollout status deployment onepassword-connect

Verify (should show only GET /health and GET /heartbeat 200s):

kubectl -n onepassword-connect logs deployment/onepassword-connect -c connect-api 2>&1 | tail -10

Expected platform sync order

Wave	App
0	apps-of-apps (root)
1	external-secrets
1	onepassword-connect
2	nfs-csi
3	cert-manager
4	ingress-nginx
5	external-dns

Troubleshooting

GatewayClass not ACCEPTED after applying: Restart the Cilium operator — it may have started before the CRDs were present:

kubectl rollout restart deployment/cilium-operator -n kube-system
kubectl get gatewayclass

ArgoCD can't reach Gitea:

kubectl -n argocd logs deployment/argocd-repo-server | grep -i "error\|ssh"

Check that gitea-repo secret exists and known hosts configmap has gitea.mk-labs.cloud.

App stuck in Unknown state:

kubectl -n argocd get application <name> -o yaml | grep -A10 "conditions"

ArgoCD pruning itself: The argocd namespace Applications must have prune: false. Never enable prune on any Application that manages the argocd namespace.

Helm valueFiles field: The correct field name is valueFiles (plural). valuesFile and valuesFiles are silently ignored — ArgoCD syncs green but values aren't applied.

1Password Connect base64 error: illegal base64 data at input byte 0 means credentials were stored as raw JSON. Delete and recreate:

kubectl delete secret op-credentials -n onepassword-connect
kubectl create secret generic op-credentials \
  --namespace onepassword-connect \
  --from-literal=1password-credentials.json=$(base64 -w 0 /home/wed/1password-credentials.json)
kubectl -n onepassword-connect rollout restart deployment onepassword-connect

1Password Connect invalid Authorization header: Token has a trailing newline. Recreate:

echo -n "your-token" > /home/wed/connect-token
kubectl delete secret connect-token -n onepassword-connect
kubectl create secret generic connect-token \
  --namespace onepassword-connect \
  --from-file=token=/home/wed/connect-token
kubectl -n onepassword-connect rollout restart deployment onepassword-connect

Post-bootstrap

Once ingress-nginx and cert-manager are synced:

ArgoCD UI: https://argocd.local.mk-labs.cloud
Change admin password: argocd account update-password
Authentik OIDC: uncomment oidc.config in cluster/argocd/values.yaml once the ArgoCD provider is configured in Authentik
Migrate gitea-repo secret to ESO once 1Password Connect is healthy

7.1 KiB Raw Blame History