homelab/docs/fastpass-rebuild-runbook.md

# fastpass cluster rebuild runbook

Root cause: Talos 1.13 defaults to `enforce: baseline` for Pod Security admission
control. Cilium requires `NET_ADMIN`, `NET_RAW`, and `SYS_ADMIN` — all blocked
under baseline. Fix: `enforce: privileged` in talconfig.yaml before generating
machine configs. Everything else in this runbook follows from a clean wipe.

All commands run from `city-hall` (10.1.71.38) in the `talos/talhelper/`
directory unless otherwise noted.

---

## Phase 0 — pre-flight

Confirm tools are present and on the right versions:

```bash
talhelper --version   # expect 3.1.10
talosctl version --client   # expect v1.13.2
kubectl version --client    # expect v1.36.x
helm version               # expect v3.x
```

Confirm the age key and sops config are in place:

```bash
cat ~/.config/sops/age/keys.txt | head -1   # should show "# created: ..."
cat /opt/git/homelab/.sops.yaml             # should reference your age pubkey
```

---

## Phase 1 — wipe and redeploy Talos nodes

### 1.1 Reset all six nodes

Run from `talos/talhelper/`. If the nodes are currently in a broken state,
reset them to wipe the install and reboot into maintenance mode:

```bash
# Control plane nodes
for node in 10.1.71.66 10.1.71.67 10.1.71.68; do
  talosctl reset \
    --talosconfig clusterconfig/talosconfig \
    --nodes $node \
    --graceful=false \
    --reboot \
    --system-labels-to-wipe STATE \
    --system-labels-to-wipe EPHEMERAL
done

# Worker nodes
for node in 10.1.71.69 10.1.71.70 10.1.71.71; do
  talosctl reset \
    --talosconfig clusterconfig/talosconfig \
    --nodes $node \
    --graceful=false \
    --reboot \
    --system-labels-to-wipe STATE \
    --system-labels-to-wipe EPHEMERAL
done
```

Wait ~2 minutes. Nodes boot into maintenance mode (no OS installed, just the
Talos installer kernel running in RAM). You can verify via Proxmox console —
the node should show the Talos maintenance screen.

If nodes are completely unreachable (talosctl can't connect), boot them from
the Talos ISO via Proxmox and they'll come up in maintenance mode automatically.

### 1.2 Regenerate machine configs

The talconfig.yaml now has `enforce: privileged`. Regenerate everything:

```bash
cd /opt/git/homelab/talos/talhelper
talhelper genconfig
```

This writes fresh configs to `clusterconfig/`. Commit the regenerated configs
before applying:

```bash
cd /opt/git/homelab
git add talos/
git commit -m "fix: set enforce:privileged for Cilium compatibility"
git push
```

### 1.3 Apply machine configs

```bash
cd /opt/git/homelab/talos/talhelper
talhelper gencommand apply --extra-flags="--insecure" | bash
```

Expected output for each node: `Applied configuration without a reboot`

The nodes will install Talos to disk and reboot automatically. Watch progress
on one control plane node:

```bash
talosctl -n 10.1.71.66 --talosconfig clusterconfig/talosconfig dmesg --follow
```

Wait for all six nodes to finish installing and come back up (~3-5 min).

### 1.4 Bootstrap etcd

Run bootstrap exactly once — on the first control plane node only:

```bash
talhelper gencommand bootstrap | bash
```

This initializes etcd on `space-mountain`. The other two control plane nodes
join automatically. Do not run bootstrap again — it will corrupt the cluster.

### 1.5 Get kubeconfig

```bash
talhelper gencommand kubeconfig | bash
# or directly:
talosctl kubeconfig \
  --talosconfig clusterconfig/talosconfig \
  --nodes 10.1.71.66 \
  ~/.kube/config
```

### 1.6 Verify all 6 nodes are Ready

```bash
kubectl get nodes -o wide --watch
```

Wait until all six show `Ready`. This takes 3-5 minutes after bootstrap.
Do not proceed to Phase 2 until all six nodes are `Ready`.

Expected output:
```
NAME                   STATUS   ROLES           AGE   VERSION
big-thunder-mountain   Ready    control-plane   5m    v1.32.3
haunted-mansion        Ready    <none>           4m    v1.32.3
jungle-cruise          Ready    <none>           4m    v1.32.3
peter-pans-flight      Ready    <none>           4m    v1.32.3
space-mountain         Ready    control-plane   5m    v1.32.3
splash-mountain        Ready    control-plane   5m    v1.32.3
```

### 1.7 Approve all pending CSRs

```bash
kubectl get csr --no-headers | grep Pending | awk '{print $1}' | xargs kubectl certificate approve
```

Nodes will show `NotReady` until Cilium is installed — that's expected. The
condition clears after Phase 2.

---

## Phase 2 — install Cilium

Cilium is Helm-only. It is never managed by ArgoCD.

### 2.1 Add the Helm repo

```bash
helm repo add cilium https://helm.cilium.io/
helm repo update
```

### 2.2 Install Cilium

```bash
helm upgrade --install cilium cilium/cilium \
  --version 1.17.3 \
  --namespace kube-system \
  --values /opt/git/homelab/talos/cilium/cilium-values.yaml \
  --wait \
  --timeout 5m
```

The `--wait` flag blocks until all Cilium pods are Running. If it times out,
check what's happening:

```bash
kubectl -n kube-system get pods -l app.kubernetes.io/name=cilium-agent
kubectl -n kube-system describe pod <cilium-agent-pod>
```

### 2.3 Apply IP pools

```bash
kubectl apply -f /opt/git/homelab/talos/cilium/ip-pools.yaml
```

### 2.4 Verify Cilium health

```bash
# Overall status
cilium status --wait

# All nodes should show cilium-agent as OK
kubectl -n kube-system get pods -l app.kubernetes.io/name=cilium-agent

# Confirm L2 announcement policy is active
kubectl get ciliuml2announcementpolicy

# Confirm IP pools are ready
kubectl get ciliumloadbalancerippool
```

### 2.5 Verify all nodes are Ready

```bash
kubectl get nodes
```

All six must show `Ready` before proceeding. If any are still `NotReady` after
Cilium is healthy, drain and uncordon to force a kubelet re-check:

```bash
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
kubectl uncordon <node>
```

---

## Phase 3 — bootstrap ArgoCD

ArgoCD is installed manually once. After that, everything else is GitOps.

### 3.1 Create the argocd namespace

```bash
kubectl create namespace argocd
```

### 3.2 Install ArgoCD

```bash
kubectl apply -n argocd \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.2/manifests/install.yaml
```

Wait for ArgoCD to come up:

```bash
kubectl -n argocd rollout status deployment argocd-server --timeout=3m
```

### 3.3 Apply server config (insecure mode for ingress TLS termination)

ArgoCD needs to run with `--insecure` so ingress-nginx handles TLS. This is in
your repo at `cluster/argocd/argocd-cmd-params-cm.yaml`:

```bash
kubectl apply -f /opt/git/homelab/cluster/argocd/argocd-cmd-params-cm.yaml
kubectl -n argocd rollout restart deployment argocd-server
kubectl -n argocd rollout status deployment argocd-server --timeout=2m
```

The ConfigMap should contain:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
  namespace: argocd
data:
  server.insecure: "true"
```

### 3.4 Bootstrap 1Password Connect secret

This is the only manual `kubectl apply` after ArgoCD install. Everything else
comes from Git.

```bash
# Create the secret from your 1Password Connect credentials file
kubectl create namespace external-secrets

kubectl create secret generic onepassword-connect-secret \
  --namespace external-secrets \
  --from-file=1password-credentials.json=/path/to/1password-credentials.json \
  --from-literal=token=<your-1password-connect-token>
```

Verify:
```bash
kubectl -n external-secrets get secret onepassword-connect-secret
```

### 3.5 Configure ArgoCD to access Gitea via SSH

Add the Gitea SSH key to ArgoCD's known hosts and create the repo secret:

```bash
# Get Gitea's SSH host key
ssh-keyscan -t ed25519 mad-tea-party.local.mk-labs.cloud >> /tmp/gitea-known-hosts

# Create the repo credentials secret
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: gitea-repo-creds
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: git@mad-tea-party.local.mk-labs.cloud:rblundon/homelab.git
  sshPrivateKey: |
$(cat ~/.ssh/argocd_gitea_ed25519 | sed 's/^/    /')
EOF
```

Add Gitea to ArgoCD known hosts:
```bash
argocd cert add-ssh mad-tea-party.local.mk-labs.cloud \
  --ssh-known-hosts-data "$(ssh-keyscan -t ed25519 mad-tea-party.local.mk-labs.cloud 2>/dev/null)"
```

### 3.6 Apply the root Application

This is the app-of-apps entry point. It points ArgoCD at `cluster/platform/`
and `cluster/applications/`:

```bash
kubectl apply -f /opt/git/homelab/cluster/argocd/application.yaml
```

The argocd Application in that file must have `prune: false` to prevent ArgoCD
from ever pruning its own namespace:

```yaml
# cluster/argocd/application.yaml  (key fields)
spec:
  project: default
  source:
    repoURL: git@mad-tea-party.local.mk-labs.cloud:rblundon/homelab.git
    targetRevision: main
    path: cluster/platform
    directory:
      exclude: "application.yaml"   # prevents self-reference loop
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: false    # NEVER prune argocd namespace
      selfHeal: true
```

### 3.7 Watch the platform sync

```bash
# Get initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d; echo

# Port-forward to access UI (before ingress is up)
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Or use CLI
argocd login localhost:8080 --username admin --insecure

# Watch sync status
argocd app list
argocd app get platform
```

Platform apps will sync in wave order (sync-wave annotation). Expected order:
1. Wave 0: cert-manager CRDs
2. Wave 1: cert-manager, external-secrets
3. Wave 2: ingress-nginx
4. Wave 3: external-dns
5. Wave 4: application workloads

---

## Phase 4 — verify platform

```bash
# All platform pods running
kubectl get pods -A | grep -v Running | grep -v Completed

# Ingress controller has a LB IP from the pool
kubectl -n ingress-nginx get svc ingress-nginx-controller

# cert-manager is issuing certs
kubectl get clusterissuers

# External secrets operator is syncing
kubectl -n external-secrets get pods

# external-dns is running
kubectl -n external-dns get pods
```

---

## Known gotchas (learned the hard way)

**ArgoCD self-prune:** The `argocd-config` Application must have `prune: false`.
If prune is enabled and ArgoCD syncs itself, it will delete its own resources
and take itself down. Recovery requires manual re-apply.

**Multi-source self-reference loop:** If an Application's source path contains
its own `application.yaml`, ArgoCD enters a sync loop. Fix: add
`directory.exclude: "application.yaml"` to the source spec.

**ArgoCD Helm valueFiles:** The correct field name is `valueFiles` (plural), not
`valuesFile` or `valuesFiles`. Wrong field name causes silent failure — ArgoCD
syncs green but ignores your values.

**Cilium managed by Helm only:** Never add Cilium to ArgoCD. If ArgoCD manages
Cilium and there's a sync conflict, ArgoCD can restart Cilium mid-cluster
operation, breaking all networking. Helm is the correct tool here.

**IP pool API version:** CiliumLoadBalancerIPPool must use `cilium.io/v2alpha1`.
Wrong version creates the object but it never activates — LB IPs are assigned
but not advertised.

**Talos VIP and NIC name:** The VIP (10.1.71.65) is configured per-node in the
`networkInterfaces` section. Each control plane node declares the VIP on `ens18`.
Workers do not declare the VIP. If the NIC name changes on your Proxmox
VMs, check with: `talosctl -n <ip> get addresses --talosconfig clusterconfig/talosconfig`

**Bootstrap once only:** `talhelper gencommand bootstrap` initializes etcd.
Running it a second time on an already-bootstrapped cluster corrupts etcd.
If you accidentally run it twice, you need to wipe and start over.