homelab/docs/city-hall-setup.md

# city-hall Setup Guide

`city-hall` (`10.1.71.38`) is the Ansible/Terraform control node and admin jump box
for the `mk-labs` homelab. This document is the authoritative list of required software
and configuration.

## System

- **OS**: Ubuntu 24.04
- **User**: `wed` (passwordless sudo)
- **SSH key**: `~/.ssh/ansible` (used by Ansible and Terraform SSH agent)
- **Repo**: `/opt/git/homelab` (cloned from `git.mk-labs.cloud/rblundon/homelab`)

---

## Required Software

### Core Tools

| Tool | Purpose | Install |
|---|---|---|
| `git` | Source control | `sudo apt install git -y` |
| `terraform` | Infrastructure provisioning | See below |
| `ansible` | Configuration management | See below |
| `packer` | VM template creation | See below |

#### Terraform

```bash
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform -y
```

#### Ansible

```bash
sudo apt install pipx -y
pipx install ansible
pipx inject ansible netaddr
ansible-galaxy collection install effectivelywild.technitium_dns
```

#### Packer

```bash
sudo apt update && sudo apt install packer -y
```

---

### Talos Cluster Tools

Required for provisioning and managing the `fastpass` Talos Kubernetes cluster.

| Tool | Purpose | Install |
|---|---|---|
| `talhelper` | Talos cluster config generation | See below |
| `talosctl` | Talos cluster CLI | See below |
| `kubectl` | Kubernetes CLI | See below |
| `helm` | Kubernetes package manager | See below |
| `age` | Encryption key generation (used by SOPS) | `sudo apt install age -y` |
| `sops` | Secret encryption for `talsecret.sops.yaml` | See below |

#### talhelper

```bash
curl -fsSL https://i.jpillora.com/budimanjojo/talhelper! | bash
```

Verify:
```bash
talhelper --version
```

#### talosctl

```bash
curl -sL https://talos.dev/install | sh
```

Verify:
```bash
talosctl version --client
```

#### kubectl

```bash
curl -LO "https://dl.k8s.io/release/$(curl -sL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin/
```

Verify:
```bash
kubectl version --client
```

#### Helm

```bash
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
```

Verify:
```bash
helm version
```

#### SOPS

```bash
SOPS_VERSION=$(curl -s https://api.github.com/repos/getsops/sops/releases/latest | grep tag_name | cut -d '"' -f 4)
curl -LO "https://github.com/getsops/sops/releases/download/${SOPS_VERSION}/sops-${SOPS_VERSION}.linux.amd64"
chmod +x "sops-${SOPS_VERSION}.linux.amd64"
sudo mv "sops-${SOPS_VERSION}.linux.amd64" /usr/local/bin/sops
```

Verify:
```bash
sops --version
```

---

## SSH Agent

The SSH agent must be running and loaded with the Ansible key before running Terraform
or Ansible. Add to `~/.bashrc`:

```bash
eval $(ssh-agent) > /dev/null
ssh-add ~/.ssh/ansible 2>/dev/null
```

---

## SOPS / age Setup (first time only)

Required before running `talhelper genconfig` for the first time.

```bash
# Generate an age key pair
age-keygen -o ~/.config/sops/age/keys.txt

# Note the public key from the output, e.g.:
# Public key: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# Create a .sops.yaml at the repo root referencing your public key
cat > /opt/git/homelab/.sops.yaml << EOF
creation_rules:
  - path_regex: talos/talhelper/talsecret.sops.yaml
    age: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
EOF
```

Generate and encrypt the secrets file (talhelper looks for `talsecret.sops.yaml` by default):

```bash
cd talos/talhelper
talhelper gensecret > talsecret.sops.yaml
sops -e -i talsecret.sops.yaml
```

---

## Terraform Notes

- Always run from the relevant module directory (e.g., `terraform/talos/cluster/`)
- State files live in `states/` within each module — never committed to git
- Sensitive vars passed via CLI, never in committed tfvars files
- The `!` in the Proxmox API token requires single quotes to prevent bash history expansion:

```bash
terraform apply \
  -var-file="tfvars/fastpass.tfvars" \
  -state="states/fastpass.tfstate" \
  -var='proxmox_api_token=terraform@pve!terraform-token=<secret>' \
  -var='unifi_password=<secret>' \
  -var='technitium_api_token=<secret>'
```

---

## Ansible Notes

- Always run from the `ansible/` directory
- Roles path: `ansible/playbooks/roles/`
- SSH user: `wed`
- Vault password file: `~/.vault_pass` (not committed)

---

## Talos / talhelper Notes

- Config lives at `talos/talhelper/` in the repo
- Secrets file is named `talsecret.sops.yaml` (talhelper default) — SOPS-encrypted
- Network interface on Proxmox VMs is `ens18` (not `eth0`)
- `talosctl` config (`~/.talos/config`) is generated by talhelper — keep a backup
- Cluster endpoint: `https://10.1.71.65:6443` (VIP)
- Talos API endpoint: `https://10.1.71.65:50000`
- Initial `apply-config` requires `--extra-flags="--insecure"` (maintenance mode only)
- `bootstrap` only runs once — bootstraps etcd on `space-mountain`

---

## fastpass Cluster Bootstrap Runbook

Run from `talos/talhelper/` unless otherwise noted.

### 1. Provision VMs (Terraform)

```bash
cd terraform/talos/cluster
terraform apply \
  -var-file="tfvars/fastpass.tfvars" \
  -state="states/fastpass.tfstate" \
  -var='proxmox_api_token=terraform@pve!terraform-token=<secret>' \
  -var='unifi_password=<secret>' \
  -var='technitium_api_token=<secret>'
```

### 2. Generate Talos Machine Configs

```bash
cd talos/talhelper

# First time only — generate and encrypt secrets
talhelper gensecret > talsecret.sops.yaml
sops -e -i talsecret.sops.yaml

# Generate machine configs (uses talsecret.sops.yaml automatically)
talhelper genconfig
```

### 3. Verify Nodes are in Maintenance Mode

```bash
talosctl -n 10.1.71.66 get discoveredvolumes \
  --talosconfig clusterconfig/talosconfig --insecure
```

Confirm install disk is `/dev/sda`.

### 4. Apply Machine Configs

```bash
talhelper gencommand apply --extra-flags="--insecure" | bash
```

Nodes will install Talos to disk and reboot automatically.

### 5. Monitor Install

```bash
talosctl -n 10.1.71.66 --talosconfig clusterconfig/talosconfig dashboard
```

Wait until all nodes show `Running` state.

### 6. Bootstrap etcd (once only)

```bash
talhelper gencommand bootstrap | bash
```

Only run once — bootstraps etcd on `space-mountain`.

### 7. Get kubeconfig

```bash
talosctl --talosconfig clusterconfig/talosconfig \
  --nodes 10.1.71.66 kubeconfig ~/.kube/config
```

### 8. Approve Worker CSRs

```bash
kubectl get csr
kubectl certificate approve $(kubectl get csr --no-headers | awk '{print $1}')
```

### 9. Install Cilium (CNI)

```bash
helm repo add cilium https://helm.cilium.io/
helm repo update

helm install cilium cilium/cilium \
  --namespace kube-system \
  --set ipam.mode=kubernetes \
  --set routingMode=tunnel \
  --set tunnelProtocol=vxlan \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=10.1.71.65 \
  --set k8sServicePort=6443 \
  --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
  --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
  --set cgroup.autoMount.enabled=false \
  --set cgroup.hostRoot=/sys/fs/cgroup
```

Note: The explicit `securityContext.capabilities` flags are required for Talos — Talos
restricts privileged container capabilities by default. The `cgroup.autoMount.enabled=false`
and `cgroup.hostRoot` flags are required because Talos manages cgroups itself.

### 10. Verify Cluster

```bash
kubectl get nodes
kubectl get pods -n kube-system
```

All nodes should show `Ready`. Cilium pods should be `Running`.

### 11. Post-Install HA Setup

See output of `terraform output` for the full HA registration commands.
Register VMs with Proxmox HA and add to node-affinity rules once the
cluster is stable and the ISO is no longer needed for boot.