# fastpass cluster rebuild runbook Root cause: Talos 1.13 defaults to `enforce: baseline` for Pod Security admission control. Cilium requires `NET_ADMIN`, `NET_RAW`, and `SYS_ADMIN` — all blocked under baseline. Fix: `enforce: privileged` in talconfig.yaml before generating machine configs. Everything else in this runbook follows from a clean wipe. All commands run from `city-hall` (10.1.71.38) in the `talos/talhelper/` directory unless otherwise noted. --- ## Phase 0 — pre-flight Confirm tools are present and on the right versions: ```bash talhelper --version # expect 3.1.10 talosctl version --client # expect v1.13.2 kubectl version --client # expect v1.36.x helm version # expect v3.x ``` Confirm the age key and sops config are in place: ```bash cat ~/.config/sops/age/keys.txt | head -1 # should show "# created: ..." cat /opt/git/homelab/.sops.yaml # should reference your age pubkey ``` --- ## Phase 1 — wipe and redeploy Talos nodes ### 1.1 Reset all six nodes Run from `talos/talhelper/`. If the nodes are currently in a broken state, reset them to wipe the install and reboot into maintenance mode: ```bash # Control plane nodes for node in 10.1.71.66 10.1.71.67 10.1.71.68; do talosctl reset \ --talosconfig clusterconfig/talosconfig \ --nodes $node \ --graceful=false \ --reboot \ --system-labels-to-wipe STATE \ --system-labels-to-wipe EPHEMERAL done # Worker nodes for node in 10.1.71.69 10.1.71.70 10.1.71.71; do talosctl reset \ --talosconfig clusterconfig/talosconfig \ --nodes $node \ --graceful=false \ --reboot \ --system-labels-to-wipe STATE \ --system-labels-to-wipe EPHEMERAL done ``` Wait ~2 minutes. Nodes boot into maintenance mode (no OS installed, just the Talos installer kernel running in RAM). You can verify via Proxmox console — the node should show the Talos maintenance screen. If nodes are completely unreachable (talosctl can't connect), boot them from the Talos ISO via Proxmox and they'll come up in maintenance mode automatically. ### 1.2 Regenerate machine configs The talconfig.yaml now has `enforce: privileged`. Regenerate everything: ```bash cd /opt/git/homelab/talos/talhelper talhelper genconfig ``` This writes fresh configs to `clusterconfig/`. Commit the regenerated configs before applying: ```bash cd /opt/git/homelab git add talos/ git commit -m "fix: set enforce:privileged for Cilium compatibility" git push ``` ### 1.3 Apply machine configs ```bash cd /opt/git/homelab/talos/talhelper talhelper gencommand apply --extra-flags="--insecure" | bash ``` Expected output for each node: `Applied configuration without a reboot` The nodes will install Talos to disk and reboot automatically. Watch progress on one control plane node: ```bash talosctl -n 10.1.71.66 --talosconfig clusterconfig/talosconfig dmesg --follow ``` Wait for all six nodes to finish installing and come back up (~3-5 min). ### 1.4 Bootstrap etcd Run bootstrap exactly once — on the first control plane node only: ```bash talhelper gencommand bootstrap | bash ``` This initializes etcd on `space-mountain`. The other two control plane nodes join automatically. Do not run bootstrap again — it will corrupt the cluster. ### 1.5 Get kubeconfig ```bash talhelper gencommand kubeconfig | bash # or directly: talosctl kubeconfig \ --talosconfig clusterconfig/talosconfig \ --nodes 10.1.71.66 \ ~/.kube/config ``` ### 1.6 Verify all 6 nodes are Ready ```bash kubectl get nodes -o wide --watch ``` Wait until all six show `Ready`. This takes 3-5 minutes after bootstrap. Do not proceed to Phase 2 until all six nodes are `Ready`. Expected output: ``` NAME STATUS ROLES AGE VERSION big-thunder-mountain Ready control-plane 5m v1.32.3 haunted-mansion Ready 4m v1.32.3 jungle-cruise Ready 4m v1.32.3 peter-pans-flight Ready 4m v1.32.3 space-mountain Ready control-plane 5m v1.32.3 splash-mountain Ready control-plane 5m v1.32.3 ``` ### 1.7 Approve all pending CSRs ```bash kubectl get csr --no-headers | grep Pending | awk '{print $1}' | xargs kubectl certificate approve ``` Nodes will show `NotReady` until Cilium is installed — that's expected. The condition clears after Phase 2. --- ## Phase 2 — install Cilium Cilium is Helm-only. It is never managed by ArgoCD. ### 2.1 Add the Helm repo ```bash helm repo add cilium https://helm.cilium.io/ helm repo update ``` ### 2.2 Install Cilium ```bash helm upgrade --install cilium cilium/cilium \ --version 1.17.3 \ --namespace kube-system \ --values /opt/git/homelab/talos/cilium/cilium-values.yaml \ --wait \ --timeout 5m ``` The `--wait` flag blocks until all Cilium pods are Running. If it times out, check what's happening: ```bash kubectl -n kube-system get pods -l app.kubernetes.io/name=cilium-agent kubectl -n kube-system describe pod ``` ### 2.3 Apply IP pools ```bash kubectl apply -f /opt/git/homelab/talos/cilium/ip-pools.yaml ``` ### 2.4 Verify Cilium health ```bash # Overall status cilium status --wait # All nodes should show cilium-agent as OK kubectl -n kube-system get pods -l app.kubernetes.io/name=cilium-agent # Confirm L2 announcement policy is active kubectl get ciliuml2announcementpolicy # Confirm IP pools are ready kubectl get ciliumloadbalancerippool ``` ### 2.5 Verify all nodes are Ready ```bash kubectl get nodes ``` All six must show `Ready` before proceeding. If any are still `NotReady` after Cilium is healthy, drain and uncordon to force a kubelet re-check: ```bash kubectl drain --ignore-daemonsets --delete-emptydir-data kubectl uncordon ``` --- ## Phase 3 — bootstrap ArgoCD ArgoCD is installed manually once. After that, everything else is GitOps. ### 3.1 Create the argocd namespace ```bash kubectl create namespace argocd ``` ### 3.2 Install ArgoCD ```bash kubectl apply -n argocd \ -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.2/manifests/install.yaml ``` Wait for ArgoCD to come up: ```bash kubectl -n argocd rollout status deployment argocd-server --timeout=3m ``` ### 3.3 Apply server config (insecure mode for ingress TLS termination) ArgoCD needs to run with `--insecure` so ingress-nginx handles TLS. This is in your repo at `cluster/argocd/argocd-cmd-params-cm.yaml`: ```bash kubectl apply -f /opt/git/homelab/cluster/argocd/argocd-cmd-params-cm.yaml kubectl -n argocd rollout restart deployment argocd-server kubectl -n argocd rollout status deployment argocd-server --timeout=2m ``` The ConfigMap should contain: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: argocd-cmd-params-cm namespace: argocd data: server.insecure: "true" ``` ### 3.4 Bootstrap 1Password Connect secret This is the only manual `kubectl apply` after ArgoCD install. Everything else comes from Git. ```bash # Create the secret from your 1Password Connect credentials file kubectl create namespace external-secrets kubectl create secret generic onepassword-connect-secret \ --namespace external-secrets \ --from-file=1password-credentials.json=/path/to/1password-credentials.json \ --from-literal=token= ``` Verify: ```bash kubectl -n external-secrets get secret onepassword-connect-secret ``` ### 3.5 Configure ArgoCD to access Gitea via SSH Add the Gitea SSH key to ArgoCD's known hosts and create the repo secret: ```bash # Get Gitea's SSH host key ssh-keyscan -t ed25519 mad-tea-party.local.mk-labs.cloud >> /tmp/gitea-known-hosts # Create the repo credentials secret kubectl apply -f - </dev/null)" ``` ### 3.6 Apply the root Application This is the app-of-apps entry point. It points ArgoCD at `cluster/platform/` and `cluster/applications/`: ```bash kubectl apply -f /opt/git/homelab/cluster/argocd/application.yaml ``` The argocd Application in that file must have `prune: false` to prevent ArgoCD from ever pruning its own namespace: ```yaml # cluster/argocd/application.yaml (key fields) spec: project: default source: repoURL: git@mad-tea-party.local.mk-labs.cloud:rblundon/homelab.git targetRevision: main path: cluster/platform directory: exclude: "application.yaml" # prevents self-reference loop destination: server: https://kubernetes.default.svc syncPolicy: automated: prune: false # NEVER prune argocd namespace selfHeal: true ``` ### 3.7 Watch the platform sync ```bash # Get initial admin password kubectl -n argocd get secret argocd-initial-admin-secret \ -o jsonpath='{.data.password}' | base64 -d; echo # Port-forward to access UI (before ingress is up) kubectl port-forward svc/argocd-server -n argocd 8080:443 # Or use CLI argocd login localhost:8080 --username admin --insecure # Watch sync status argocd app list argocd app get platform ``` Platform apps will sync in wave order (sync-wave annotation). Expected order: 1. Wave 0: cert-manager CRDs 2. Wave 1: cert-manager, external-secrets 3. Wave 2: ingress-nginx 4. Wave 3: external-dns 5. Wave 4: application workloads --- ## Phase 4 — verify platform ```bash # All platform pods running kubectl get pods -A | grep -v Running | grep -v Completed # Ingress controller has a LB IP from the pool kubectl -n ingress-nginx get svc ingress-nginx-controller # cert-manager is issuing certs kubectl get clusterissuers # External secrets operator is syncing kubectl -n external-secrets get pods # external-dns is running kubectl -n external-dns get pods ``` --- ## Known gotchas (learned the hard way) **ArgoCD self-prune:** The `argocd-config` Application must have `prune: false`. If prune is enabled and ArgoCD syncs itself, it will delete its own resources and take itself down. Recovery requires manual re-apply. **Multi-source self-reference loop:** If an Application's source path contains its own `application.yaml`, ArgoCD enters a sync loop. Fix: add `directory.exclude: "application.yaml"` to the source spec. **ArgoCD Helm valueFiles:** The correct field name is `valueFiles` (plural), not `valuesFile` or `valuesFiles`. Wrong field name causes silent failure — ArgoCD syncs green but ignores your values. **Cilium managed by Helm only:** Never add Cilium to ArgoCD. If ArgoCD manages Cilium and there's a sync conflict, ArgoCD can restart Cilium mid-cluster operation, breaking all networking. Helm is the correct tool here. **IP pool API version:** CiliumLoadBalancerIPPool must use `cilium.io/v2alpha1`. Wrong version creates the object but it never activates — LB IPs are assigned but not advertised. **Talos VIP and NIC name:** The VIP (10.1.71.65) is configured per-node in the `networkInterfaces` section. Each control plane node declares the VIP on `ens18`. Workers do not declare the VIP. If the NIC name changes on your Proxmox VMs, check with: `talosctl -n get addresses --talosconfig clusterconfig/talosconfig` **Bootstrap once only:** `talhelper gencommand bootstrap` initializes etcd. Running it a second time on an already-bootstrapped cluster corrupts etcd. If you accidentally run it twice, you need to wipe and start over.