Files

Ryan Blundon cdc78955b1 cilium, argo

2026-05-18 13:20:42 -05:00

12 KiB

Raw Blame History

fastpass cluster rebuild runbook

Root cause: Talos 1.13 defaults to enforce: baseline for Pod Security admission control. Cilium requires NET_ADMIN, NET_RAW, and SYS_ADMIN — all blocked under baseline. Fix: enforce: privileged in talconfig.yaml before generating machine configs. Everything else in this runbook follows from a clean wipe.

All commands run from city-hall (10.1.71.38) in the talos/talhelper/ directory unless otherwise noted.

Phase 0 — pre-flight

Confirm tools are present and on the right versions:

talhelper --version   # expect 3.1.10
talosctl version --client   # expect v1.13.2
kubectl version --client    # expect v1.36.x
helm version               # expect v3.x

Confirm the age key and sops config are in place:

cat ~/.config/sops/age/keys.txt | head -1   # should show "# created: ..."
cat /opt/git/homelab/.sops.yaml             # should reference your age pubkey

Phase 1 — wipe and redeploy Talos nodes

1.1 Reset all six nodes

Run from talos/talhelper/. If the nodes are currently in a broken state, reset them to wipe the install and reboot into maintenance mode:

# Control plane nodes
for node in 10.1.71.66 10.1.71.67 10.1.71.68; do
  talosctl reset \
    --talosconfig clusterconfig/talosconfig \
    --nodes $node \
    --graceful=false \
    --reboot \
    --system-labels-to-wipe STATE \
    --system-labels-to-wipe EPHEMERAL
done

# Worker nodes
for node in 10.1.71.69 10.1.71.70 10.1.71.71; do
  talosctl reset \
    --talosconfig clusterconfig/talosconfig \
    --nodes $node \
    --graceful=false \
    --reboot \
    --system-labels-to-wipe STATE \
    --system-labels-to-wipe EPHEMERAL
done

Wait ~2 minutes. Nodes boot into maintenance mode (no OS installed, just the Talos installer kernel running in RAM). You can verify via Proxmox console — the node should show the Talos maintenance screen.

If nodes are completely unreachable (talosctl can't connect), boot them from the Talos ISO via Proxmox and they'll come up in maintenance mode automatically.

1.2 Regenerate machine configs

The talconfig.yaml now has enforce: privileged. Regenerate everything:

cd /opt/git/homelab/talos/talhelper
talhelper genconfig

This writes fresh configs to clusterconfig/. Commit the regenerated configs before applying:

cd /opt/git/homelab
git add talos/
git commit -m "fix: set enforce:privileged for Cilium compatibility"
git push

1.3 Apply machine configs

cd /opt/git/homelab/talos/talhelper
talhelper gencommand apply --extra-flags="--insecure" | bash

Expected output for each node: Applied configuration without a reboot

The nodes will install Talos to disk and reboot automatically. Watch progress on one control plane node:

talosctl -n 10.1.71.66 --talosconfig clusterconfig/talosconfig dmesg --follow

Wait for all six nodes to finish installing and come back up (~3-5 min).

1.4 Bootstrap etcd

Run bootstrap exactly once — on the first control plane node only:

talhelper gencommand bootstrap | bash

This initializes etcd on space-mountain. The other two control plane nodes join automatically. Do not run bootstrap again — it will corrupt the cluster.

1.5 Get kubeconfig

talhelper gencommand kubeconfig | bash
# or directly:
talosctl kubeconfig \
  --talosconfig clusterconfig/talosconfig \
  --nodes 10.1.71.66 \
  ~/.kube/config

1.6 Verify all 6 nodes are Ready

kubectl get nodes -o wide --watch

Wait until all six show Ready. This takes 3-5 minutes after bootstrap. Do not proceed to Phase 2 until all six nodes are Ready.

Expected output:

NAME                   STATUS   ROLES           AGE   VERSION
big-thunder-mountain   Ready    control-plane   5m    v1.32.3
haunted-mansion        Ready    <none>           4m    v1.32.3
jungle-cruise          Ready    <none>           4m    v1.32.3
peter-pans-flight      Ready    <none>           4m    v1.32.3
space-mountain         Ready    control-plane   5m    v1.32.3
splash-mountain        Ready    control-plane   5m    v1.32.3

1.7 Approve all pending CSRs

kubectl get csr --no-headers | grep Pending | awk '{print $1}' | xargs kubectl certificate approve

Nodes will show NotReady until Cilium is installed — that's expected. The condition clears after Phase 2.

Phase 2 — install Cilium

Cilium is Helm-only. It is never managed by ArgoCD.

2.1 Add the Helm repo

helm repo add cilium https://helm.cilium.io/
helm repo update

2.2 Install Cilium

helm upgrade --install cilium cilium/cilium \
  --version 1.17.3 \
  --namespace kube-system \
  --values /opt/git/homelab/talos/cilium/cilium-values.yaml \
  --wait \
  --timeout 5m

The --wait flag blocks until all Cilium pods are Running. If it times out, check what's happening:

kubectl -n kube-system get pods -l app.kubernetes.io/name=cilium-agent
kubectl -n kube-system describe pod <cilium-agent-pod>

2.3 Apply IP pools

kubectl apply -f /opt/git/homelab/talos/cilium/ip-pools.yaml

2.4 Verify Cilium health

# Overall status
cilium status --wait

# All nodes should show cilium-agent as OK
kubectl -n kube-system get pods -l app.kubernetes.io/name=cilium-agent

# Confirm L2 announcement policy is active
kubectl get ciliuml2announcementpolicy

# Confirm IP pools are ready
kubectl get ciliumloadbalancerippool

2.5 Verify all nodes are Ready

kubectl get nodes

All six must show Ready before proceeding. If any are still NotReady after Cilium is healthy, drain and uncordon to force a kubelet re-check:

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
kubectl uncordon <node>

Phase 3 — bootstrap ArgoCD

ArgoCD is installed manually once. After that, everything else is GitOps.

3.1 Create the argocd namespace

kubectl create namespace argocd

3.2 Install ArgoCD

kubectl apply -n argocd \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.2/manifests/install.yaml

Wait for ArgoCD to come up:

kubectl -n argocd rollout status deployment argocd-server --timeout=3m

3.3 Apply server config (insecure mode for ingress TLS termination)

ArgoCD needs to run with --insecure so ingress-nginx handles TLS. This is in your repo at cluster/argocd/argocd-cmd-params-cm.yaml:

kubectl apply -f /opt/git/homelab/cluster/argocd/argocd-cmd-params-cm.yaml
kubectl -n argocd rollout restart deployment argocd-server
kubectl -n argocd rollout status deployment argocd-server --timeout=2m

The ConfigMap should contain:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
  namespace: argocd
data:
  server.insecure: "true"

3.4 Bootstrap 1Password Connect secret

This is the only manual kubectl apply after ArgoCD install. Everything else comes from Git.

# Create the secret from your 1Password Connect credentials file
kubectl create namespace external-secrets

kubectl create secret generic onepassword-connect-secret \
  --namespace external-secrets \
  --from-file=1password-credentials.json=/path/to/1password-credentials.json \
  --from-literal=token=<your-1password-connect-token>

Verify:

kubectl -n external-secrets get secret onepassword-connect-secret

3.5 Configure ArgoCD to access Gitea via SSH

Add the Gitea SSH key to ArgoCD's known hosts and create the repo secret:

# Get Gitea's SSH host key
ssh-keyscan -t ed25519 mad-tea-party.local.mk-labs.cloud >> /tmp/gitea-known-hosts

# Create the repo credentials secret
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: gitea-repo-creds
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: git@mad-tea-party.local.mk-labs.cloud:rblundon/homelab.git
  sshPrivateKey: |
$(cat ~/.ssh/argocd_gitea_ed25519 | sed 's/^/    /')
EOF

Add Gitea to ArgoCD known hosts:

argocd cert add-ssh mad-tea-party.local.mk-labs.cloud \
  --ssh-known-hosts-data "$(ssh-keyscan -t ed25519 mad-tea-party.local.mk-labs.cloud 2>/dev/null)"

3.6 Apply the root Application

This is the app-of-apps entry point. It points ArgoCD at cluster/platform/ and cluster/applications/:

kubectl apply -f /opt/git/homelab/cluster/argocd/application.yaml

The argocd Application in that file must have prune: false to prevent ArgoCD from ever pruning its own namespace:

# cluster/argocd/application.yaml  (key fields)
spec:
  project: default
  source:
    repoURL: git@mad-tea-party.local.mk-labs.cloud:rblundon/homelab.git
    targetRevision: main
    path: cluster/platform
    directory:
      exclude: "application.yaml"   # prevents self-reference loop
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: false    # NEVER prune argocd namespace
      selfHeal: true

3.7 Watch the platform sync

# Get initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d; echo

# Port-forward to access UI (before ingress is up)
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Or use CLI
argocd login localhost:8080 --username admin --insecure

# Watch sync status
argocd app list
argocd app get platform

Platform apps will sync in wave order (sync-wave annotation). Expected order:

Wave 0: cert-manager CRDs
Wave 1: cert-manager, external-secrets
Wave 2: ingress-nginx
Wave 3: external-dns
Wave 4: application workloads

Phase 4 — verify platform

# All platform pods running
kubectl get pods -A | grep -v Running | grep -v Completed

# Ingress controller has a LB IP from the pool
kubectl -n ingress-nginx get svc ingress-nginx-controller

# cert-manager is issuing certs
kubectl get clusterissuers

# External secrets operator is syncing
kubectl -n external-secrets get pods

# external-dns is running
kubectl -n external-dns get pods

Known gotchas (learned the hard way)

ArgoCD self-prune: The argocd-config Application must have prune: false. If prune is enabled and ArgoCD syncs itself, it will delete its own resources and take itself down. Recovery requires manual re-apply.

Multi-source self-reference loop: If an Application's source path contains its own application.yaml, ArgoCD enters a sync loop. Fix: add directory.exclude: "application.yaml" to the source spec.

ArgoCD Helm valueFiles: The correct field name is valueFiles (plural), not valuesFile or valuesFiles. Wrong field name causes silent failure — ArgoCD syncs green but ignores your values.

Cilium managed by Helm only: Never add Cilium to ArgoCD. If ArgoCD manages Cilium and there's a sync conflict, ArgoCD can restart Cilium mid-cluster operation, breaking all networking. Helm is the correct tool here.

IP pool API version: CiliumLoadBalancerIPPool must use cilium.io/v2alpha1. Wrong version creates the object but it never activates — LB IPs are assigned but not advertised.

Talos VIP and NIC name: The VIP (10.1.71.65) is configured per-node in the networkInterfaces section. Each control plane node declares the VIP on ens18. Workers do not declare the VIP. If the NIC name changes on your Proxmox VMs, check with: talosctl -n <ip> get addresses --talosconfig clusterconfig/talosconfig

Bootstrap once only: talhelper gencommand bootstrap initializes etcd. Running it a second time on an already-bootstrapped cluster corrupts etcd. If you accidentally run it twice, you need to wipe and start over.

12 KiB Raw Blame History