homelab/docs/city-hall-setup.md

city-hall Setup Guide

city-hall (10.1.71.38) is the Ansible/Terraform control node and admin jump box for the mk-labs homelab. This document is the authoritative list of required software and configuration.

System

• OS: Ubuntu 24.04
• User: wed (passwordless sudo)
• SSH key: ~/.ssh/ansible (used by Ansible and Terraform SSH agent)
• Repo: /opt/git/homelab (cloned from git.mk-labs.cloud/rblundon/homelab)

---

Required Software

Core Tools

Tool	Purpose	Install
git	Source control	sudo apt install git -y
terraform	Infrastructure provisioning	See below
ansible	Configuration management	See below
packer	VM template creation	See below

Terraform

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform -y

Ansible

sudo apt install pipx -y
pipx install ansible
pipx inject ansible netaddr
ansible-galaxy collection install effectivelywild.technitium_dns

Packer

sudo apt update && sudo apt install packer -y

---

Talos Cluster Tools

Required for provisioning and managing the fastpass Talos Kubernetes cluster.

Tool	Purpose	Install
talhelper	Talos cluster config generation	See below
talosctl	Talos cluster CLI	See below
kubectl	Kubernetes CLI	See below
helm	Kubernetes package manager	See below
age	Encryption key generation (used by SOPS)	sudo apt install age -y
sops	Secret encryption for talsecret.sops.yaml	See below

talhelper

curl -fsSL https://i.jpillora.com/budimanjojo/talhelper! | bash

Verify:

talhelper --version

talosctl

curl -sL https://talos.dev/install | sh

Verify:

talosctl version --client

kubectl

curl -LO "https://dl.k8s.io/release/$(curl -sL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin/

Verify:

kubectl version --client

Helm

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Verify:

helm version

SOPS

SOPS_VERSION=$(curl -s https://api.github.com/repos/getsops/sops/releases/latest | grep tag_name | cut -d '"' -f 4)
curl -LO "https://github.com/getsops/sops/releases/download/${SOPS_VERSION}/sops-${SOPS_VERSION}.linux.amd64"
chmod +x "sops-${SOPS_VERSION}.linux.amd64"
sudo mv "sops-${SOPS_VERSION}.linux.amd64" /usr/local/bin/sops

Verify:

sops --version

---

SSH Agent

The SSH agent must be running and loaded with the Ansible key before running Terraform or Ansible. Add to ~/.bashrc:

eval $(ssh-agent) > /dev/null
ssh-add ~/.ssh/ansible 2>/dev/null

---

SOPS / age Setup (first time only)

Required before running talhelper genconfig for the first time.

# Generate an age key pair
age-keygen -o ~/.config/sops/age/keys.txt

# Note the public key from the output, e.g.:
# Public key: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# Create a .sops.yaml at the repo root referencing your public key
cat > /opt/git/homelab/.sops.yaml << EOF
creation_rules:
  - path_regex: talos/talhelper/talsecret.sops.yaml
    age: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
EOF

Generate and encrypt the secrets file (talhelper looks for talsecret.sops.yaml by default):

cd talos/talhelper
talhelper gensecret > talsecret.sops.yaml
sops -e -i talsecret.sops.yaml

---

Terraform Notes

• Always run from the relevant module directory (e.g., terraform/talos/cluster/)
• State files live in states/ within each module — never committed to git
• Sensitive vars passed via CLI, never in committed tfvars files
• The ! in the Proxmox API token requires single quotes to prevent bash history expansion:

terraform apply \
  -var-file="tfvars/fastpass.tfvars" \
  -state="states/fastpass.tfstate" \
  -var='proxmox_api_token=terraform@pve!terraform-token=<secret>' \
  -var='unifi_password=<secret>' \
  -var='technitium_api_token=<secret>'

---

Ansible Notes

• Always run from the ansible/ directory
• Roles path: ansible/playbooks/roles/
• SSH user: wed
• Vault password file: ~/.vault_pass (not committed)

---

Talos / talhelper Notes

• Config lives at talos/talhelper/ in the repo
• Secrets file is named talsecret.sops.yaml (talhelper default) — SOPS-encrypted
• Network interface on Proxmox VMs is ens18 (not eth0)
• talosctl config (~/.talos/config) is generated by talhelper — keep a backup
• Cluster endpoint: https://10.1.71.65:6443 (VIP)
• Talos API endpoint: https://10.1.71.65:50000
• Initial apply-config requires --extra-flags="--insecure" (maintenance mode only)
• bootstrap only runs once — bootstraps etcd on space-mountain

---

fastpass Cluster Bootstrap Runbook

Run from talos/talhelper/ unless otherwise noted.

1. Provision VMs (Terraform)

cd terraform/talos/cluster
terraform apply \
  -var-file="tfvars/fastpass.tfvars" \
  -state="states/fastpass.tfstate" \
  -var='proxmox_api_token=terraform@pve!terraform-token=<secret>' \
  -var='unifi_password=<secret>' \
  -var='technitium_api_token=<secret>'

2. Generate Talos Machine Configs

cd talos/talhelper

# First time only — generate and encrypt secrets
talhelper gensecret > talsecret.sops.yaml
sops -e -i talsecret.sops.yaml

# Generate machine configs (uses talsecret.sops.yaml automatically)
talhelper genconfig

3. Verify Nodes are in Maintenance Mode

talosctl -n 10.1.71.66 get discoveredvolumes \
  --talosconfig clusterconfig/talosconfig --insecure

Confirm install disk is /dev/sda.

4. Apply Machine Configs

talhelper gencommand apply --extra-flags="--insecure" | bash

Nodes will install Talos to disk and reboot automatically.

5. Monitor Install

talosctl -n 10.1.71.66 --talosconfig clusterconfig/talosconfig dashboard

Wait until all nodes show Running state.

6. Bootstrap etcd (once only)

talhelper gencommand bootstrap | bash

Only run once — bootstraps etcd on space-mountain.

7. Get kubeconfig

talosctl --talosconfig clusterconfig/talosconfig \
  --nodes 10.1.71.66 kubeconfig ~/.kube/config

8. Approve Worker CSRs

kubectl get csr
kubectl certificate approve $(kubectl get csr --no-headers | awk '{print $1}')

9. Install Cilium (CNI)

helm repo add cilium https://helm.cilium.io/
helm repo update

helm install cilium cilium/cilium \
  --namespace kube-system \
  --set ipam.mode=kubernetes \
  --set routingMode=tunnel \
  --set tunnelProtocol=vxlan \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=10.1.71.65 \
  --set k8sServicePort=6443 \
  --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
  --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
  --set cgroup.autoMount.enabled=false \
  --set cgroup.hostRoot=/sys/fs/cgroup

Note: The explicit securityContext.capabilities flags are required for Talos — Talos restricts privileged container capabilities by default. The cgroup.autoMount.enabled=false and cgroup.hostRoot flags are required because Talos manages cgroups itself.

10. Verify Cluster

kubectl get nodes
kubectl get pods -n kube-system

All nodes should show Ready. Cilium pods should be Running.

11. Post-Install HA Setup

See output of terraform output for the full HA registration commands. Register VMs with Proxmox HA and add to node-affinity rules once the cluster is stable and the ISO is no longer needed for boot.