homelab/docs/city-hall-setup.md

city-hall Setup Guide

city-hall (10.1.71.38) is the Ansible/Terraform control node and admin jump box for the mk-labs homelab. This document is the authoritative list of required software and configuration.

System

• OS: Ubuntu 24.04
• User: wed (passwordless sudo)
• SSH key: ~/.ssh/ansible (used by Ansible and Terraform SSH agent)
• Repo: /opt/git/homelab (cloned from git.mk-labs.cloud/rblundon/homelab)

---

Required Software

Core Tools

Tool	Purpose	Install
git	Source control	sudo apt install git -y
terraform	Infrastructure provisioning	See below
ansible	Configuration management	See below
packer	VM template creation	See below

Terraform

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform -y

Ansible

sudo apt install pipx -y
pipx install ansible
pipx inject ansible netaddr
ansible-galaxy collection install effectivelywild.technitium_dns

Packer

sudo apt update && sudo apt install packer -y

---

Talos Cluster Tools

Required for provisioning and managing the fastpass Talos Kubernetes cluster.

Tool	Purpose	Install
talhelper	Talos cluster config generation	See below
talosctl	Talos cluster CLI	See below
kubectl	Kubernetes CLI	See below
helm	Kubernetes package manager	See below
cilium	Cilium CNI CLI (status, connectivity tests)	See below
argocd	ArgoCD CLI (app management, account operations)	See below
age	Encryption key generation (used by SOPS)	sudo apt install age -y
sops	Secret encryption for talsecret.sops.yaml	See below

talhelper

curl -fsSL https://i.jpillora.com/budimanjojo/talhelper! | bash

Verify:

talhelper --version

talosctl

curl -sL https://talos.dev/install | sh

Verify:

talosctl version --client

kubectl

curl -LO "https://dl.k8s.io/release/$(curl -sL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin/

Verify:

kubectl version --client

Helm

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Verify:

helm version

Cilium CLI

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --remote-name-all \
  https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
tar -xzvf cilium-linux-amd64.tar.gz
sudo mv cilium /usr/local/bin/
rm cilium-linux-amd64.tar.gz

Verify:

cilium version

ArgoCD CLI

ARGOCD_VERSION=$(curl -s https://api.github.com/repos/argoproj/argo-cd/releases/latest | grep tag_name | cut -d '"' -f 4)
curl -sSL -o /tmp/argocd \
  https://github.com/argoproj/argo-cd/releases/download/${ARGOCD_VERSION}/argocd-linux-amd64
chmod +x /tmp/argocd
sudo mv /tmp/argocd /usr/local/bin/argocd

Verify:

argocd version --client

SOPS

SOPS_VERSION=$(curl -s https://api.github.com/repos/getsops/sops/releases/latest | grep tag_name | cut -d '"' -f 4)
curl -LO "https://github.com/getsops/sops/releases/download/${SOPS_VERSION}/sops-${SOPS_VERSION}.linux.amd64"
chmod +x "sops-${SOPS_VERSION}.linux.amd64"
sudo mv "sops-${SOPS_VERSION}.linux.amd64" /usr/local/bin/sops

Verify:

sops --version

---

SSH Agent

The SSH agent must be running and loaded with the Ansible key before running Terraform or Ansible. Add to ~/.bashrc:

eval $(ssh-agent) > /dev/null
ssh-add ~/.ssh/ansible 2>/dev/null

---

SOPS / age Setup (first time only)

Required before running talhelper genconfig for the first time.

# Generate an age key pair
age-keygen -o ~/.config/sops/age/keys.txt

# Note the public key from the output, e.g.:
# Public key: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# Create a .sops.yaml at the repo root referencing your public key
cat > /opt/git/homelab/.sops.yaml << EOF
creation_rules:
  - path_regex: talos/talhelper/talsecret.sops.yaml
    age: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
EOF

Generate and encrypt the secrets file (talhelper looks for talsecret.sops.yaml by default):

cd talos/talhelper
talhelper gensecret > talsecret.sops.yaml
sops -e -i talsecret.sops.yaml

---

Terraform Notes

• Always run from the relevant module directory (e.g., terraform/talos/cluster/)
• State files live in states/ within each module — never committed to git
• Sensitive vars passed via CLI, never in committed tfvars files
• The ! in the Proxmox API token requires single quotes to prevent bash history expansion:

terraform apply \
  -var-file="tfvars/fastpass.tfvars" \
  -state="states/fastpass.tfstate" \
  -var='proxmox_api_token=terraform@pve!terraform-token=<secret>' \
  -var='unifi_password=<secret>' \
  -var='technitium_api_token=<secret>'

---

Ansible Notes

• Always run from the ansible/ directory
• Roles path: ansible/playbooks/roles/
• SSH user: wed
• Vault password file: ~/.vault_pass (not committed)

---

Talos / talhelper Notes

• Config lives at talos/talhelper/ in the repo
• Secrets file is named talsecret.sops.yaml (talhelper default) — SOPS-encrypted
• Network interface on Proxmox VMs is ens18 (not eth0)
• talosctl config (~/.talos/config) is generated by talhelper — keep a backup
• Cluster endpoint: https://10.1.71.65:6443 (VIP)
• Talos API endpoint: https://10.1.71.65:50000
• Initial apply-config requires --extra-flags="--insecure" (maintenance mode only)
• bootstrap only runs once — bootstraps etcd on space-mountain

---

fastpass Cluster Bootstrap Runbook

Run from talos/talhelper/ unless otherwise noted.

1. Provision VMs (Terraform)

cd terraform/talos/cluster
terraform apply \
  -var-file="tfvars/fastpass.tfvars" \
  -state="states/fastpass.tfstate" \
  -var='proxmox_api_token=terraform@pve!terraform-token=<secret>' \
  -var='unifi_password=<secret>' \
  -var='technitium_api_token=<secret>'

2. Generate Talos Machine Configs

cd talos/talhelper

# First time only — generate and encrypt secrets
talhelper gensecret > talsecret.sops.yaml
sops -e -i talsecret.sops.yaml

# Generate machine configs (uses talsecret.sops.yaml automatically)
talhelper genconfig

3. Verify Nodes are in Maintenance Mode

talosctl -n 10.1.71.66 get discoveredvolumes \
  --talosconfig clusterconfig/talosconfig --insecure

Confirm install disk is /dev/sda.

4. Apply Machine Configs

talhelper gencommand apply --extra-flags="--insecure" | bash

Nodes will install Talos to disk and reboot automatically.

5. Monitor Install

talosctl -n 10.1.71.66 --talosconfig clusterconfig/talosconfig dashboard

Wait until all nodes show Running state.

6. Bootstrap etcd (once only)

talhelper gencommand bootstrap | bash

Only run once — bootstraps etcd on space-mountain.

7. Get kubeconfig

talosctl --talosconfig clusterconfig/talosconfig \
  --nodes 10.1.71.66 kubeconfig ~/.kube/config

8. Approve CSRs

kubectl get csr --no-headers | awk '{print $1}' | xargs kubectl certificate approve

Run periodically until no pending CSRs remain.

9. Install Cilium (CNI)

Cilium is managed by Helm only — never via ArgoCD.

helm repo add cilium https://helm.cilium.io/
helm repo update

helm upgrade --install cilium cilium/cilium \
  --version 1.17.3 \
  --namespace kube-system \
  --values talos/cilium/cilium-values.yaml \
  --wait \
  --timeout 5m

Apply IP pools:

kubectl apply -f talos/cilium/ip-pools.yaml

Verify:

cilium status --wait
kubectl get ciliumloadbalancerippool
kubectl get ciliuml2announcementpolicy

Important: The cilium-values.yaml contains required Talos-specific overrides:

• securityContext.capabilities — removes SYS_MODULE (blocked by Talos permanently)
• cgroup.autoMount.enabled: false — Talos manages cgroups itself
• l2announcements.interface: ens18 — Proxmox VMs use ens18, not eth0
• l2podAnnouncements.interface: ens18 — same reason

10. Bootstrap ArgoCD

See cluster/argocd/BOOTSTRAP.md for the full ArgoCD bootstrap sequence.

Summary:

1. kubectl apply -f cluster/argocd/namespace.yaml
2. Add Gitea SSH host key to ArgoCD known hosts
3. Create gitea-repo secret with SSH private key at ~/.ssh/argocd_gitea
4. helm upgrade --install argocd argo/argo-cd --values cluster/argocd/values.yaml
5. kubectl apply -n argocd -f cluster/argocd/apps-of-apps.yaml

11. Verify Cluster

kubectl get nodes
kubectl get pods -A | grep -v Running | grep -v Completed
cilium status --wait

All nodes should show Ready. All pods should be Running or Completed.