rblundon/homelab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cilium, argo

Browse Source
This commit is contained in:
Ryan Blundon
2026-05-18 13:20:42 -05:00
parent 6f0c751f63
commit cdc78955b1
18 changed files with 821 additions and 136 deletions
Download Patch File Download Diff File Expand all files Collapse all files

104
docs/city-hall-setup.md
View File

@@ -59,6 +59,8 @@ Required for provisioning and managing the `fastpass` Talos Kubernetes cluster.
| `talosctl` | Talos cluster CLI | See below |
| `kubectl` | Kubernetes CLI | See below |
| `helm` | Kubernetes package manager | See below |
| `cilium` | Cilium CNI CLI (status, connectivity tests) | See below |
| `argocd` | ArgoCD CLI (app management, account operations) | See below |
| `age` | Encryption key generation (used by SOPS) | `sudo apt install age -y` |
| `sops` | Secret encryption for `talsecret.sops.yaml` | See below |
@@ -108,6 +110,37 @@ Verify:
helm version
```
#### Cilium CLI
```bash
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --remote-name-all \
https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
tar -xzvf cilium-linux-amd64.tar.gz
sudo mv cilium /usr/local/bin/
rm cilium-linux-amd64.tar.gz
```
Verify:
```bash
cilium version
```
#### ArgoCD CLI
```bash
ARGOCD_VERSION=$(curl -s https://api.github.com/repos/argoproj/argo-cd/releases/latest | grep tag_name | cut -d '"' -f 4)
curl -sSL -o /tmp/argocd \
https://github.com/argoproj/argo-cd/releases/download/${ARGOCD_VERSION}/argocd-linux-amd64
chmod +x /tmp/argocd
sudo mv /tmp/argocd /usr/local/bin/argocd
```
Verify:
```bash
argocd version --client
```
#### SOPS
```bash
@@ -274,48 +307,67 @@ talosctl --talosconfig clusterconfig/talosconfig \
--nodes 10.1.71.66 kubeconfig ~/.kube/config
```
### 8. Approve Worker CSRs
### 8. Approve CSRs
```bash
kubectl get csr
kubectl certificate approve $(kubectl get csr --no-headers | awk '{print $1}')
kubectl get csr --no-headers | awk '{print $1}' | xargs kubectl certificate approve
```
Run periodically until no pending CSRs remain.
### 9. Install Cilium (CNI)
Cilium is managed by Helm only — never via ArgoCD.
```bash
helm repo add cilium https://helm.cilium.io/
helm repo update
helm install cilium cilium/cilium \
helm upgrade --install cilium cilium/cilium \
--version 1.17.3 \
--namespace kube-system \
--set ipam.mode=kubernetes \
--set routingMode=tunnel \
--set tunnelProtocol=vxlan \
--set kubeProxyReplacement=true \
--set k8sServiceHost=10.1.71.65 \
--set k8sServicePort=6443 \
--set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
--set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
--set cgroup.autoMount.enabled=false \
--set cgroup.hostRoot=/sys/fs/cgroup
--values talos/cilium/cilium-values.yaml \
--wait \
--timeout 5m
```
Note: The explicit `securityContext.capabilities` flags are required for Talos — Talos
restricts privileged container capabilities by default. The `cgroup.autoMount.enabled=false`
and `cgroup.hostRoot` flags are required because Talos manages cgroups itself.
Apply IP pools:
### 10. Verify Cluster
```bash
kubectl apply -f talos/cilium/ip-pools.yaml
```
Verify:
```bash
cilium status --wait
kubectl get ciliumloadbalancerippool
kubectl get ciliuml2announcementpolicy
```
**Important:** The `cilium-values.yaml` contains required Talos-specific overrides:
- `securityContext.capabilities` — removes `SYS_MODULE` (blocked by Talos permanently)
- `cgroup.autoMount.enabled: false` — Talos manages cgroups itself
- `l2announcements.interface: ens18` — Proxmox VMs use `ens18`, not `eth0`
- `l2podAnnouncements.interface: ens18` — same reason
### 10. Bootstrap ArgoCD
See `cluster/argocd/BOOTSTRAP.md` for the full ArgoCD bootstrap sequence.
Summary:
1. `kubectl apply -f cluster/argocd/namespace.yaml`
2. Add Gitea SSH host key to ArgoCD known hosts
3. Create `gitea-repo` secret with SSH private key at `~/.ssh/argocd_gitea`
4. `helm upgrade --install argocd argo/argo-cd --values cluster/argocd/values.yaml`
5. `kubectl apply -n argocd -f cluster/argocd/apps-of-apps.yaml`
### 11. Verify Cluster
```bash
kubectl get nodes
kubectl get pods -n kube-system
kubectl get pods -A | grep -v Running | grep -v Completed
cilium status --wait
```
All nodes should show `Ready`. Cilium pods should be `Running`.
### 11. Post-Install HA Setup
See output of `terraform output` for the full HA registration commands.
Register VMs with Proxmox HA and add to node-affinity rules once the
cluster is stable and the ISO is no longer needed for boot.
All nodes should show `Ready`. All pods should be `Running` or `Completed`.

433
docs/fastpass-rebuild-runbook.md Normal file
View File

@@ -0,0 +1,433 @@
# fastpass cluster rebuild runbook
Root cause: Talos 1.13 defaults to `enforce: baseline` for Pod Security admission
control. Cilium requires `NET_ADMIN`, `NET_RAW`, and `SYS_ADMIN` — all blocked
under baseline. Fix: `enforce: privileged` in talconfig.yaml before generating
machine configs. Everything else in this runbook follows from a clean wipe.
All commands run from `city-hall` (10.1.71.38) in the `talos/talhelper/`
directory unless otherwise noted.
---
## Phase 0 — pre-flight
Confirm tools are present and on the right versions:
```bash
talhelper --version # expect 3.1.10
talosctl version --client # expect v1.13.2
kubectl version --client # expect v1.36.x
helm version # expect v3.x
```
Confirm the age key and sops config are in place:
```bash
cat ~/.config/sops/age/keys.txt | head -1 # should show "# created: ..."
cat /opt/git/homelab/.sops.yaml # should reference your age pubkey
```
---
## Phase 1 — wipe and redeploy Talos nodes
### 1.1 Reset all six nodes
Run from `talos/talhelper/`. If the nodes are currently in a broken state,
reset them to wipe the install and reboot into maintenance mode:
```bash
# Control plane nodes
for node in 10.1.71.66 10.1.71.67 10.1.71.68; do
talosctl reset \
--talosconfig clusterconfig/talosconfig \
--nodes $node \
--graceful=false \
--reboot \
--system-labels-to-wipe STATE \
--system-labels-to-wipe EPHEMERAL
done
# Worker nodes
for node in 10.1.71.69 10.1.71.70 10.1.71.71; do
talosctl reset \
--talosconfig clusterconfig/talosconfig \
--nodes $node \
--graceful=false \
--reboot \
--system-labels-to-wipe STATE \
--system-labels-to-wipe EPHEMERAL
done
```
Wait ~2 minutes. Nodes boot into maintenance mode (no OS installed, just the
Talos installer kernel running in RAM). You can verify via Proxmox console —
the node should show the Talos maintenance screen.
If nodes are completely unreachable (talosctl can't connect), boot them from
the Talos ISO via Proxmox and they'll come up in maintenance mode automatically.
### 1.2 Regenerate machine configs
The talconfig.yaml now has `enforce: privileged`. Regenerate everything:
```bash
cd /opt/git/homelab/talos/talhelper
talhelper genconfig
```
This writes fresh configs to `clusterconfig/`. Commit the regenerated configs
before applying:
```bash
cd /opt/git/homelab
git add talos/
git commit -m "fix: set enforce:privileged for Cilium compatibility"
git push
```
### 1.3 Apply machine configs
```bash
cd /opt/git/homelab/talos/talhelper
talhelper gencommand apply --extra-flags="--insecure" | bash
```
Expected output for each node: `Applied configuration without a reboot`
The nodes will install Talos to disk and reboot automatically. Watch progress
on one control plane node:
```bash
talosctl -n 10.1.71.66 --talosconfig clusterconfig/talosconfig dmesg --follow
```
Wait for all six nodes to finish installing and come back up (~3-5 min).
### 1.4 Bootstrap etcd
Run bootstrap exactly once — on the first control plane node only:
```bash
talhelper gencommand bootstrap | bash
```
This initializes etcd on `space-mountain`. The other two control plane nodes
join automatically. Do not run bootstrap again — it will corrupt the cluster.
### 1.5 Get kubeconfig
```bash
talhelper gencommand kubeconfig | bash
# or directly:
talosctl kubeconfig \
--talosconfig clusterconfig/talosconfig \
--nodes 10.1.71.66 \
~/.kube/config
```
### 1.6 Verify all 6 nodes are Ready
```bash
kubectl get nodes -o wide --watch
```
Wait until all six show `Ready`. This takes 3-5 minutes after bootstrap.
Do not proceed to Phase 2 until all six nodes are `Ready`.
Expected output:
```
NAME STATUS ROLES AGE VERSION
big-thunder-mountain Ready control-plane 5m v1.32.3
haunted-mansion Ready <none> 4m v1.32.3
jungle-cruise Ready <none> 4m v1.32.3
peter-pans-flight Ready <none> 4m v1.32.3
space-mountain Ready control-plane 5m v1.32.3
splash-mountain Ready control-plane 5m v1.32.3
```
### 1.7 Approve all pending CSRs
```bash
kubectl get csr --no-headers | grep Pending | awk '{print $1}' | xargs kubectl certificate approve
```
Nodes will show `NotReady` until Cilium is installed — that's expected. The
condition clears after Phase 2.
---
## Phase 2 — install Cilium
Cilium is Helm-only. It is never managed by ArgoCD.
### 2.1 Add the Helm repo
```bash
helm repo add cilium https://helm.cilium.io/
helm repo update
```
### 2.2 Install Cilium
```bash
helm upgrade --install cilium cilium/cilium \
--version 1.17.3 \
--namespace kube-system \
--values /opt/git/homelab/talos/cilium/cilium-values.yaml \
--wait \
--timeout 5m
```
The `--wait` flag blocks until all Cilium pods are Running. If it times out,
check what's happening:
```bash
kubectl -n kube-system get pods -l app.kubernetes.io/name=cilium-agent
kubectl -n kube-system describe pod <cilium-agent-pod>
```
### 2.3 Apply IP pools
```bash
kubectl apply -f /opt/git/homelab/talos/cilium/ip-pools.yaml
```
### 2.4 Verify Cilium health
```bash
# Overall status
cilium status --wait
# All nodes should show cilium-agent as OK
kubectl -n kube-system get pods -l app.kubernetes.io/name=cilium-agent
# Confirm L2 announcement policy is active
kubectl get ciliuml2announcementpolicy
# Confirm IP pools are ready
kubectl get ciliumloadbalancerippool
```
### 2.5 Verify all nodes are Ready
```bash
kubectl get nodes
```
All six must show `Ready` before proceeding. If any are still `NotReady` after
Cilium is healthy, drain and uncordon to force a kubelet re-check:
```bash
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
kubectl uncordon <node>
```
---
## Phase 3 — bootstrap ArgoCD
ArgoCD is installed manually once. After that, everything else is GitOps.
### 3.1 Create the argocd namespace
```bash
kubectl create namespace argocd
```
### 3.2 Install ArgoCD
```bash
kubectl apply -n argocd \
-f https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.2/manifests/install.yaml
```
Wait for ArgoCD to come up:
```bash
kubectl -n argocd rollout status deployment argocd-server --timeout=3m
```
### 3.3 Apply server config (insecure mode for ingress TLS termination)
ArgoCD needs to run with `--insecure` so ingress-nginx handles TLS. This is in
your repo at `cluster/argocd/argocd-cmd-params-cm.yaml`:
```bash
kubectl apply -f /opt/git/homelab/cluster/argocd/argocd-cmd-params-cm.yaml
kubectl -n argocd rollout restart deployment argocd-server
kubectl -n argocd rollout status deployment argocd-server --timeout=2m
```
The ConfigMap should contain:
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cmd-params-cm
namespace: argocd
data:
server.insecure: "true"
```
### 3.4 Bootstrap 1Password Connect secret
This is the only manual `kubectl apply` after ArgoCD install. Everything else
comes from Git.
```bash
# Create the secret from your 1Password Connect credentials file
kubectl create namespace external-secrets
kubectl create secret generic onepassword-connect-secret \
--namespace external-secrets \
--from-file=1password-credentials.json=/path/to/1password-credentials.json \
--from-literal=token=<your-1password-connect-token>
```
Verify:
```bash
kubectl -n external-secrets get secret onepassword-connect-secret
```
### 3.5 Configure ArgoCD to access Gitea via SSH
Add the Gitea SSH key to ArgoCD's known hosts and create the repo secret:
```bash
# Get Gitea's SSH host key
ssh-keyscan -t ed25519 mad-tea-party.local.mk-labs.cloud >> /tmp/gitea-known-hosts
# Create the repo credentials secret
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: gitea-repo-creds
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
stringData:
type: git
url: git@mad-tea-party.local.mk-labs.cloud:rblundon/homelab.git
sshPrivateKey: |
$(cat ~/.ssh/argocd_gitea_ed25519 | sed 's/^/ /')
EOF
```
Add Gitea to ArgoCD known hosts:
```bash
argocd cert add-ssh mad-tea-party.local.mk-labs.cloud \
--ssh-known-hosts-data "$(ssh-keyscan -t ed25519 mad-tea-party.local.mk-labs.cloud 2>/dev/null)"
```
### 3.6 Apply the root Application
This is the app-of-apps entry point. It points ArgoCD at `cluster/platform/`
and `cluster/applications/`:
```bash
kubectl apply -f /opt/git/homelab/cluster/argocd/application.yaml
```
The argocd Application in that file must have `prune: false` to prevent ArgoCD
from ever pruning its own namespace:
```yaml
# cluster/argocd/application.yaml (key fields)
spec:
project: default
source:
repoURL: git@mad-tea-party.local.mk-labs.cloud:rblundon/homelab.git
targetRevision: main
path: cluster/platform
directory:
exclude: "application.yaml" # prevents self-reference loop
destination:
server: https://kubernetes.default.svc
syncPolicy:
automated:
prune: false # NEVER prune argocd namespace
selfHeal: true
```
### 3.7 Watch the platform sync
```bash
# Get initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret \
-o jsonpath='{.data.password}' | base64 -d; echo
# Port-forward to access UI (before ingress is up)
kubectl port-forward svc/argocd-server -n argocd 8080:443
# Or use CLI
argocd login localhost:8080 --username admin --insecure
# Watch sync status
argocd app list
argocd app get platform
```
Platform apps will sync in wave order (sync-wave annotation). Expected order:
1. Wave 0: cert-manager CRDs
2. Wave 1: cert-manager, external-secrets
3. Wave 2: ingress-nginx
4. Wave 3: external-dns
5. Wave 4: application workloads
---
## Phase 4 — verify platform
```bash
# All platform pods running
kubectl get pods -A | grep -v Running | grep -v Completed
# Ingress controller has a LB IP from the pool
kubectl -n ingress-nginx get svc ingress-nginx-controller
# cert-manager is issuing certs
kubectl get clusterissuers
# External secrets operator is syncing
kubectl -n external-secrets get pods
# external-dns is running
kubectl -n external-dns get pods
```
---
## Known gotchas (learned the hard way)
**ArgoCD self-prune:** The `argocd-config` Application must have `prune: false`.
If prune is enabled and ArgoCD syncs itself, it will delete its own resources
and take itself down. Recovery requires manual re-apply.
**Multi-source self-reference loop:** If an Application's source path contains
its own `application.yaml`, ArgoCD enters a sync loop. Fix: add
`directory.exclude: "application.yaml"` to the source spec.
**ArgoCD Helm valueFiles:** The correct field name is `valueFiles` (plural), not
`valuesFile` or `valuesFiles`. Wrong field name causes silent failure — ArgoCD
syncs green but ignores your values.
**Cilium managed by Helm only:** Never add Cilium to ArgoCD. If ArgoCD manages
Cilium and there's a sync conflict, ArgoCD can restart Cilium mid-cluster
operation, breaking all networking. Helm is the correct tool here.
**IP pool API version:** CiliumLoadBalancerIPPool must use `cilium.io/v2alpha1`.
Wrong version creates the object but it never activates — LB IPs are assigned
but not advertised.
**Talos VIP and NIC name:** The VIP (10.1.71.65) is configured per-node in the
`networkInterfaces` section. Each control plane node declares the VIP on `ens18`.
Workers do not declare the VIP. If the NIC name changes on your Proxmox
VMs, check with: `talosctl -n <ip> get addresses --talosconfig clusterconfig/talosconfig`
**Bootstrap once only:** `talhelper gencommand bootstrap` initializes etcd.
Running it a second time on an already-bootstrapped cluster corrupts etcd.
If you accidentally run it twice, you need to wipe and start over.