homelab/docs/city-hall-setup.md

# city-hall Setup Guide

`city-hall` (`10.1.71.38`) is the Ansible/Terraform control node and admin jump box
for the `mk-labs` homelab. This document is the authoritative list of required software
and configuration.

## System

- **OS**: Ubuntu 24.04
- **User**: `wed` (passwordless sudo)
- **SSH key**: `~/.ssh/ansible` (used by Ansible and Terraform SSH agent)
- **Repo**: `/opt/git/homelab` (cloned from `git.mk-labs.cloud/rblundon/homelab`)

---

## Required Software

### Core Tools

| Tool | Purpose | Install |
|---|---|---|
| `git` | Source control | `sudo apt install git -y` |
| `terraform` | Infrastructure provisioning | See below |
| `ansible` | Configuration management | See below |
| `packer` | VM template creation | See below |

#### Terraform

```bash
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform -y
```

#### Ansible

```bash
sudo apt install pipx -y
pipx install ansible
pipx inject ansible netaddr
ansible-galaxy collection install effectivelywild.technitium_dns
```

#### Packer

```bash
sudo apt update && sudo apt install packer -y
```

---

### Talos Cluster Tools

Required for provisioning and managing the `fastpass` Talos Kubernetes cluster.

| Tool | Purpose | Install |
|---|---|---|
| `talhelper` | Talos cluster config generation | See below |
| `talosctl` | Talos cluster CLI | See below |
| `kubectl` | Kubernetes CLI | See below |
| `helm` | Kubernetes package manager | See below |
| `cilium` | Cilium CNI CLI (status, connectivity tests) | See below |
| `argocd` | ArgoCD CLI (app management, account operations) | See below |
| `age` | Encryption key generation (used by SOPS) | `sudo apt install age -y` |
| `sops` | Secret encryption for `talsecret.sops.yaml` | See below |

#### talhelper

```bash
curl -fsSL https://i.jpillora.com/budimanjojo/talhelper! | bash
```

Verify:
```bash
talhelper --version
```

#### talosctl

```bash
curl -sL https://talos.dev/install | sh
```

Verify:
```bash
talosctl version --client
```

#### kubectl

```bash
curl -LO "https://dl.k8s.io/release/$(curl -sL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin/
```

Verify:
```bash
kubectl version --client
```

#### Helm

```bash
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
```

Verify:
```bash
helm version
```

#### Cilium CLI

```bash
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --remote-name-all \
  https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
tar -xzvf cilium-linux-amd64.tar.gz
sudo mv cilium /usr/local/bin/
rm cilium-linux-amd64.tar.gz
```

Verify:
```bash
cilium version
```

#### ArgoCD CLI

```bash
ARGOCD_VERSION=$(curl -s https://api.github.com/repos/argoproj/argo-cd/releases/latest | grep tag_name | cut -d '"' -f 4)
curl -sSL -o /tmp/argocd \
  https://github.com/argoproj/argo-cd/releases/download/${ARGOCD_VERSION}/argocd-linux-amd64
chmod +x /tmp/argocd
sudo mv /tmp/argocd /usr/local/bin/argocd
```

Verify:
```bash
argocd version --client
```

#### SOPS

```bash
SOPS_VERSION=$(curl -s https://api.github.com/repos/getsops/sops/releases/latest | grep tag_name | cut -d '"' -f 4)
curl -LO "https://github.com/getsops/sops/releases/download/${SOPS_VERSION}/sops-${SOPS_VERSION}.linux.amd64"
chmod +x "sops-${SOPS_VERSION}.linux.amd64"
sudo mv "sops-${SOPS_VERSION}.linux.amd64" /usr/local/bin/sops
```

Verify:
```bash
sops --version
```

---

## SSH Agent

The SSH agent must be running and loaded with the Ansible key before running Terraform
or Ansible. Add to `~/.bashrc`:

```bash
eval $(ssh-agent) > /dev/null
ssh-add ~/.ssh/ansible 2>/dev/null
```

---

## SOPS / age Setup (first time only)

Required before running `talhelper genconfig` for the first time.

```bash
# Generate an age key pair
age-keygen -o ~/.config/sops/age/keys.txt

# Note the public key from the output, e.g.:
# Public key: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# Create a .sops.yaml at the repo root referencing your public key
cat > /opt/git/homelab/.sops.yaml << EOF
creation_rules:
  - path_regex: talos/talhelper/talsecret.sops.yaml
    age: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
EOF
```

Generate and encrypt the secrets file (talhelper looks for `talsecret.sops.yaml` by default):

```bash
cd talos/talhelper
talhelper gensecret > talsecret.sops.yaml
sops -e -i talsecret.sops.yaml
```

---

## Terraform Notes

- Always run from the relevant module directory (e.g., `terraform/talos/cluster/`)
- State files live in `states/` within each module — never committed to git
- Sensitive vars passed via CLI, never in committed tfvars files
- The `!` in the Proxmox API token requires single quotes to prevent bash history expansion:

```bash
terraform apply \
  -var-file="tfvars/fastpass.tfvars" \
  -state="states/fastpass.tfstate" \
  -var='proxmox_api_token=terraform@pve!terraform-token=<secret>' \
  -var='unifi_password=<secret>' \
  -var='technitium_api_token=<secret>'
```

---

## Ansible Notes

- Always run from the `ansible/` directory
- Roles path: `ansible/playbooks/roles/`
- SSH user: `wed`
- Vault password file: `~/.vault_pass` (not committed)

---

## Talos / talhelper Notes

- Config lives at `talos/talhelper/` in the repo
- Secrets file is named `talsecret.sops.yaml` (talhelper default) — SOPS-encrypted
- Network interface on Proxmox VMs is `ens18` (not `eth0`)
- `talosctl` config (`~/.talos/config`) is generated by talhelper — keep a backup
- Cluster endpoint: `https://10.1.71.65:6443` (VIP)
- Talos API endpoint: `https://10.1.71.65:50000`
- Initial `apply-config` requires `--extra-flags="--insecure"` (maintenance mode only)
- `bootstrap` only runs once — bootstraps etcd on `space-mountain`

---

## fastpass Cluster Bootstrap Runbook

Run from `talos/talhelper/` unless otherwise noted.

### 1. Provision VMs (Terraform)

```bash
cd terraform/talos/cluster
terraform apply \
  -var-file="tfvars/fastpass.tfvars" \
  -state="states/fastpass.tfstate" \
  -var='proxmox_api_token=terraform@pve!terraform-token=<secret>' \
  -var='unifi_password=<secret>' \
  -var='technitium_api_token=<secret>'
```

### 2. Generate Talos Machine Configs

```bash
cd talos/talhelper

# First time only — generate and encrypt secrets
talhelper gensecret > talsecret.sops.yaml
sops -e -i talsecret.sops.yaml

# Generate machine configs (uses talsecret.sops.yaml automatically)
talhelper genconfig
```

### 3. Verify Nodes are in Maintenance Mode

```bash
talosctl -n 10.1.71.66 get discoveredvolumes \
  --talosconfig clusterconfig/talosconfig --insecure
```

Confirm install disk is `/dev/sda`.

### 4. Apply Machine Configs

```bash
talhelper gencommand apply --extra-flags="--insecure" | bash
```

Nodes will install Talos to disk and reboot automatically.

### 5. Monitor Install

```bash
talosctl -n 10.1.71.66 --talosconfig clusterconfig/talosconfig dashboard
```

Wait until all nodes show `Running` state.

### 6. Bootstrap etcd (once only)

```bash
talhelper gencommand bootstrap | bash
```

Only run once — bootstraps etcd on `space-mountain`.

### 7. Get kubeconfig

```bash
talosctl --talosconfig clusterconfig/talosconfig \
  --nodes 10.1.71.66 kubeconfig ~/.kube/config
```

### 8. Approve CSRs

```bash
kubectl get csr --no-headers | awk '{print $1}' | xargs kubectl certificate approve
```

Run periodically until no pending CSRs remain.

### 9. Install Cilium (CNI)

Cilium is managed by Helm only — never via ArgoCD.

```bash
helm repo add cilium https://helm.cilium.io/
helm repo update

helm upgrade --install cilium cilium/cilium \
  --version 1.17.3 \
  --namespace kube-system \
  --values talos/cilium/cilium-values.yaml \
  --wait \
  --timeout 5m
```

Apply IP pools:

```bash
kubectl apply -f talos/cilium/ip-pools.yaml
```

Verify:

```bash
cilium status --wait
kubectl get ciliumloadbalancerippool
kubectl get ciliuml2announcementpolicy
```

**Important:** The `cilium-values.yaml` contains required Talos-specific overrides:
- `securityContext.capabilities` — removes `SYS_MODULE` (blocked by Talos permanently)
- `cgroup.autoMount.enabled: false` — Talos manages cgroups itself
- `l2announcements.interface: ens18` — Proxmox VMs use `ens18`, not `eth0`
- `l2podAnnouncements.interface: ens18` — same reason

### 10. Bootstrap ArgoCD

See `cluster/argocd/BOOTSTRAP.md` for the full ArgoCD bootstrap sequence.

Summary:
1. `kubectl apply -f cluster/argocd/namespace.yaml`
2. Add Gitea SSH host key to ArgoCD known hosts
3. Create `gitea-repo` secret with SSH private key at `~/.ssh/argocd_gitea`
4. `helm upgrade --install argocd argo/argo-cd --values cluster/argocd/values.yaml`
5. `kubectl apply -n argocd -f cluster/argocd/apps-of-apps.yaml`

### 11. Verify Cluster

```bash
kubectl get nodes
kubectl get pods -A | grep -v Running | grep -v Completed
cilium status --wait
```

All nodes should show `Ready`. All pods should be `Running` or `Completed`.