feat(fastpass): Talos cluster provisioning and bootstrap
- Terraform: VM provisioning, Unifi DHCP, Technitium DNS - talhelper: cluster config for 6-node Talos cluster - Cilium 1.19.4 CNI with Talos-compatible security context - docs: city-hall setup guide and bootstrap runbook
This commit is contained in:
321
docs/city-hall-setup.md
Normal file
321
docs/city-hall-setup.md
Normal file
@@ -0,0 +1,321 @@
|
||||
# city-hall Setup Guide
|
||||
|
||||
`city-hall` (`10.1.71.38`) is the Ansible/Terraform control node and admin jump box
|
||||
for the `mk-labs` homelab. This document is the authoritative list of required software
|
||||
and configuration.
|
||||
|
||||
## System
|
||||
|
||||
- **OS**: Ubuntu 24.04
|
||||
- **User**: `wed` (passwordless sudo)
|
||||
- **SSH key**: `~/.ssh/ansible` (used by Ansible and Terraform SSH agent)
|
||||
- **Repo**: `/opt/git/homelab` (cloned from `git.mk-labs.cloud/rblundon/homelab`)
|
||||
|
||||
---
|
||||
|
||||
## Required Software
|
||||
|
||||
### Core Tools
|
||||
|
||||
| Tool | Purpose | Install |
|
||||
|---|---|---|
|
||||
| `git` | Source control | `sudo apt install git -y` |
|
||||
| `terraform` | Infrastructure provisioning | See below |
|
||||
| `ansible` | Configuration management | See below |
|
||||
| `packer` | VM template creation | See below |
|
||||
|
||||
#### Terraform
|
||||
|
||||
```bash
|
||||
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
|
||||
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
|
||||
sudo apt update && sudo apt install terraform -y
|
||||
```
|
||||
|
||||
#### Ansible
|
||||
|
||||
```bash
|
||||
sudo apt install pipx -y
|
||||
pipx install ansible
|
||||
pipx inject ansible netaddr
|
||||
ansible-galaxy collection install effectivelywild.technitium_dns
|
||||
```
|
||||
|
||||
#### Packer
|
||||
|
||||
```bash
|
||||
sudo apt update && sudo apt install packer -y
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Talos Cluster Tools
|
||||
|
||||
Required for provisioning and managing the `fastpass` Talos Kubernetes cluster.
|
||||
|
||||
| Tool | Purpose | Install |
|
||||
|---|---|---|
|
||||
| `talhelper` | Talos cluster config generation | See below |
|
||||
| `talosctl` | Talos cluster CLI | See below |
|
||||
| `kubectl` | Kubernetes CLI | See below |
|
||||
| `helm` | Kubernetes package manager | See below |
|
||||
| `age` | Encryption key generation (used by SOPS) | `sudo apt install age -y` |
|
||||
| `sops` | Secret encryption for `talsecret.sops.yaml` | See below |
|
||||
|
||||
#### talhelper
|
||||
|
||||
```bash
|
||||
curl -fsSL https://i.jpillora.com/budimanjojo/talhelper! | bash
|
||||
```
|
||||
|
||||
Verify:
|
||||
```bash
|
||||
talhelper --version
|
||||
```
|
||||
|
||||
#### talosctl
|
||||
|
||||
```bash
|
||||
curl -sL https://talos.dev/install | sh
|
||||
```
|
||||
|
||||
Verify:
|
||||
```bash
|
||||
talosctl version --client
|
||||
```
|
||||
|
||||
#### kubectl
|
||||
|
||||
```bash
|
||||
curl -LO "https://dl.k8s.io/release/$(curl -sL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
|
||||
chmod +x kubectl
|
||||
sudo mv kubectl /usr/local/bin/
|
||||
```
|
||||
|
||||
Verify:
|
||||
```bash
|
||||
kubectl version --client
|
||||
```
|
||||
|
||||
#### Helm
|
||||
|
||||
```bash
|
||||
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
|
||||
```
|
||||
|
||||
Verify:
|
||||
```bash
|
||||
helm version
|
||||
```
|
||||
|
||||
#### SOPS
|
||||
|
||||
```bash
|
||||
SOPS_VERSION=$(curl -s https://api.github.com/repos/getsops/sops/releases/latest | grep tag_name | cut -d '"' -f 4)
|
||||
curl -LO "https://github.com/getsops/sops/releases/download/${SOPS_VERSION}/sops-${SOPS_VERSION}.linux.amd64"
|
||||
chmod +x "sops-${SOPS_VERSION}.linux.amd64"
|
||||
sudo mv "sops-${SOPS_VERSION}.linux.amd64" /usr/local/bin/sops
|
||||
```
|
||||
|
||||
Verify:
|
||||
```bash
|
||||
sops --version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## SSH Agent
|
||||
|
||||
The SSH agent must be running and loaded with the Ansible key before running Terraform
|
||||
or Ansible. Add to `~/.bashrc`:
|
||||
|
||||
```bash
|
||||
eval $(ssh-agent) > /dev/null
|
||||
ssh-add ~/.ssh/ansible 2>/dev/null
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## SOPS / age Setup (first time only)
|
||||
|
||||
Required before running `talhelper genconfig` for the first time.
|
||||
|
||||
```bash
|
||||
# Generate an age key pair
|
||||
age-keygen -o ~/.config/sops/age/keys.txt
|
||||
|
||||
# Note the public key from the output, e.g.:
|
||||
# Public key: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
||||
|
||||
# Create a .sops.yaml at the repo root referencing your public key
|
||||
cat > /opt/git/homelab/.sops.yaml << EOF
|
||||
creation_rules:
|
||||
- path_regex: talos/talhelper/talsecret.sops.yaml
|
||||
age: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
||||
EOF
|
||||
```
|
||||
|
||||
Generate and encrypt the secrets file (talhelper looks for `talsecret.sops.yaml` by default):
|
||||
|
||||
```bash
|
||||
cd talos/talhelper
|
||||
talhelper gensecret > talsecret.sops.yaml
|
||||
sops -e -i talsecret.sops.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Terraform Notes
|
||||
|
||||
- Always run from the relevant module directory (e.g., `terraform/talos/cluster/`)
|
||||
- State files live in `states/` within each module — never committed to git
|
||||
- Sensitive vars passed via CLI, never in committed tfvars files
|
||||
- The `!` in the Proxmox API token requires single quotes to prevent bash history expansion:
|
||||
|
||||
```bash
|
||||
terraform apply \
|
||||
-var-file="tfvars/fastpass.tfvars" \
|
||||
-state="states/fastpass.tfstate" \
|
||||
-var='proxmox_api_token=terraform@pve!terraform-token=<secret>' \
|
||||
-var='unifi_password=<secret>' \
|
||||
-var='technitium_api_token=<secret>'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Ansible Notes
|
||||
|
||||
- Always run from the `ansible/` directory
|
||||
- Roles path: `ansible/playbooks/roles/`
|
||||
- SSH user: `wed`
|
||||
- Vault password file: `~/.vault_pass` (not committed)
|
||||
|
||||
---
|
||||
|
||||
## Talos / talhelper Notes
|
||||
|
||||
- Config lives at `talos/talhelper/` in the repo
|
||||
- Secrets file is named `talsecret.sops.yaml` (talhelper default) — SOPS-encrypted
|
||||
- Network interface on Proxmox VMs is `ens18` (not `eth0`)
|
||||
- `talosctl` config (`~/.talos/config`) is generated by talhelper — keep a backup
|
||||
- Cluster endpoint: `https://10.1.71.65:6443` (VIP)
|
||||
- Talos API endpoint: `https://10.1.71.65:50000`
|
||||
- Initial `apply-config` requires `--extra-flags="--insecure"` (maintenance mode only)
|
||||
- `bootstrap` only runs once — bootstraps etcd on `space-mountain`
|
||||
|
||||
---
|
||||
|
||||
## fastpass Cluster Bootstrap Runbook
|
||||
|
||||
Run from `talos/talhelper/` unless otherwise noted.
|
||||
|
||||
### 1. Provision VMs (Terraform)
|
||||
|
||||
```bash
|
||||
cd terraform/talos/cluster
|
||||
terraform apply \
|
||||
-var-file="tfvars/fastpass.tfvars" \
|
||||
-state="states/fastpass.tfstate" \
|
||||
-var='proxmox_api_token=terraform@pve!terraform-token=<secret>' \
|
||||
-var='unifi_password=<secret>' \
|
||||
-var='technitium_api_token=<secret>'
|
||||
```
|
||||
|
||||
### 2. Generate Talos Machine Configs
|
||||
|
||||
```bash
|
||||
cd talos/talhelper
|
||||
|
||||
# First time only — generate and encrypt secrets
|
||||
talhelper gensecret > talsecret.sops.yaml
|
||||
sops -e -i talsecret.sops.yaml
|
||||
|
||||
# Generate machine configs (uses talsecret.sops.yaml automatically)
|
||||
talhelper genconfig
|
||||
```
|
||||
|
||||
### 3. Verify Nodes are in Maintenance Mode
|
||||
|
||||
```bash
|
||||
talosctl -n 10.1.71.66 get discoveredvolumes \
|
||||
--talosconfig clusterconfig/talosconfig --insecure
|
||||
```
|
||||
|
||||
Confirm install disk is `/dev/sda`.
|
||||
|
||||
### 4. Apply Machine Configs
|
||||
|
||||
```bash
|
||||
talhelper gencommand apply --extra-flags="--insecure" | bash
|
||||
```
|
||||
|
||||
Nodes will install Talos to disk and reboot automatically.
|
||||
|
||||
### 5. Monitor Install
|
||||
|
||||
```bash
|
||||
talosctl -n 10.1.71.66 --talosconfig clusterconfig/talosconfig dashboard
|
||||
```
|
||||
|
||||
Wait until all nodes show `Running` state.
|
||||
|
||||
### 6. Bootstrap etcd (once only)
|
||||
|
||||
```bash
|
||||
talhelper gencommand bootstrap | bash
|
||||
```
|
||||
|
||||
Only run once — bootstraps etcd on `space-mountain`.
|
||||
|
||||
### 7. Get kubeconfig
|
||||
|
||||
```bash
|
||||
talosctl --talosconfig clusterconfig/talosconfig \
|
||||
--nodes 10.1.71.66 kubeconfig ~/.kube/config
|
||||
```
|
||||
|
||||
### 8. Approve Worker CSRs
|
||||
|
||||
```bash
|
||||
kubectl get csr
|
||||
kubectl certificate approve $(kubectl get csr --no-headers | awk '{print $1}')
|
||||
```
|
||||
|
||||
### 9. Install Cilium (CNI)
|
||||
|
||||
```bash
|
||||
helm repo add cilium https://helm.cilium.io/
|
||||
helm repo update
|
||||
|
||||
helm install cilium cilium/cilium \
|
||||
--namespace kube-system \
|
||||
--set ipam.mode=kubernetes \
|
||||
--set routingMode=tunnel \
|
||||
--set tunnelProtocol=vxlan \
|
||||
--set kubeProxyReplacement=true \
|
||||
--set k8sServiceHost=10.1.71.65 \
|
||||
--set k8sServicePort=6443 \
|
||||
--set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
|
||||
--set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
|
||||
--set cgroup.autoMount.enabled=false \
|
||||
--set cgroup.hostRoot=/sys/fs/cgroup
|
||||
```
|
||||
|
||||
Note: The explicit `securityContext.capabilities` flags are required for Talos — Talos
|
||||
restricts privileged container capabilities by default. The `cgroup.autoMount.enabled=false`
|
||||
and `cgroup.hostRoot` flags are required because Talos manages cgroups itself.
|
||||
|
||||
### 10. Verify Cluster
|
||||
|
||||
```bash
|
||||
kubectl get nodes
|
||||
kubectl get pods -n kube-system
|
||||
```
|
||||
|
||||
All nodes should show `Ready`. Cilium pods should be `Running`.
|
||||
|
||||
### 11. Post-Install HA Setup
|
||||
|
||||
See output of `terraform output` for the full HA registration commands.
|
||||
Register VMs with Proxmox HA and add to node-affinity rules once the
|
||||
cluster is stable and the ISO is no longer needed for boot.
|
||||
Reference in New Issue
Block a user