homelab/cluster/bootstrap/README.md

# fastpass — Cluster Bootstrap

This directory contains resources and documentation for the **manual bootstrap
sequence** required to bring up `fastpass` from a fresh Talos install.

Everything in this directory is applied **once by hand** before ArgoCD takes
over. After Step 6 (app-of-apps), all further changes are made via Git.

---

## Prerequisites

- All 6 Talos nodes `Ready` (`kubectl get nodes`)
- Cilium healthy (`cilium status --wait`)
- IP pools applied (`kubectl get ciliumloadbalancerippool`)
- `city-hall` has `helm`, `kubectl`, and `talosctl` installed
- SSH keypair for ArgoCD → Gitea exists at `/home/wed/.ssh/argocd_gitea`
- 1Password credentials file at `/home/wed/1password-credentials.json`
- 1Password Connect token at `/home/wed/connect-token` (no trailing newline)

---

## Step 1 — Install Gateway API CRDs

The Gateway API CRDs are not bundled with Cilium — they must be installed
independently before Cilium can register the GatewayClass controller.

```bash
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml
```

Verify:

```bash
kubectl get crd gateways.gateway.networking.k8s.io
```

---

## Step 2 — Create the Cilium GatewayClass

The Cilium operator manages this object but does not auto-create it.

```bash
kubectl apply -f cluster/bootstrap/gatewayclass.yaml
```

Verify (may take a few seconds):

```bash
kubectl get gatewayclass
# Expected: cilium   io.cilium/gateway-controller   True
```

---

## Step 3 — Create the argocd namespace

Must be created before Helm — the Gitea repo secret targets this namespace.

```bash
kubectl apply -f cluster/argocd/namespace.yaml
```

---

## Step 4 — Create the Gitea repo secret

```bash
kubectl create secret generic gitea-repo \
  --namespace argocd \
  --from-literal=type=git \
  --from-literal=url=git@gitea.mk-labs.cloud:rblundon/homelab.git \
  --from-file=sshPrivateKey=/home/wed/.ssh/argocd_gitea

kubectl label secret gitea-repo -n argocd \
  argocd.argoproj.io/secret-type=repository
```

Verify:

```bash
kubectl -n argocd get secret gitea-repo
```

---

## Step 5 — Install ArgoCD via Helm

```bash
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

helm upgrade --install argocd argo/argo-cd \
  --namespace argocd \
  --create-namespace \
  --version 7.8.23 \
  --values cluster/argocd/values.yaml \
  --wait
```

Wait for all ArgoCD pods to be ready:

```bash
kubectl -n argocd get pods --watch
```

---

## Step 6 — Add Gitea to ArgoCD known hosts

```bash
ssh-keyscan -t ed25519 gitea.mk-labs.cloud 2>/dev/null | \
  kubectl patch configmap argocd-ssh-known-hosts-cm -n argocd \
  --type merge \
  -p "{\"data\":{\"ssh_known_hosts\": \"$(ssh-keyscan -t ed25519 gitea.mk-labs.cloud 2>/dev/null)\"}}"
```

---

## Step 7 — Get the initial admin password

```bash
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d; echo
```

Access the UI via port-forward (ingress not yet deployed):

```bash
kubectl port-forward svc/argocd-server -n argocd 8080:80 > /tmp/pf.log 2>&1 &
```

Browse to http://localhost:8080

---

## Step 8 — Apply the app-of-apps

This is the final manual `kubectl apply`. It hands control to ArgoCD:

```bash
kubectl apply -n argocd -f cluster/argocd/apps-of-apps.yaml
```

From this point, all changes are made via Git. ArgoCD syncs `cluster/platform/`
and `cluster/applications/` automatically.

Watch the sync:

```bash
kubectl -n argocd get applications --watch
```

---

## Step 9 — Approve CSRs

```bash
kubectl get csr --no-headers | awk '{print $1}' | xargs kubectl certificate approve
```

Run periodically until no pending CSRs remain.

---

## Step 10 — Bootstrap 1Password Connect secrets

Once `onepassword-connect` namespace exists (created by ArgoCD sync):

```bash
# Verify namespace exists
kubectl get namespace onepassword-connect

# Credentials secret — must be base64-encoded
kubectl create secret generic op-credentials \
  --namespace onepassword-connect \
  --from-literal=1password-credentials.json=$(base64 -w 0 /home/wed/1password-credentials.json)

# Token secret — file must have no trailing newline
kubectl create secret generic connect-token \
  --namespace onepassword-connect \
  --from-file=token=/home/wed/connect-token
```

Restart Connect to pick up secrets:

```bash
kubectl -n onepassword-connect rollout restart deployment onepassword-connect
kubectl -n onepassword-connect rollout status deployment onepassword-connect
```

Verify (should show only GET /health and GET /heartbeat 200s):

```bash
kubectl -n onepassword-connect logs deployment/onepassword-connect -c connect-api 2>&1 | tail -10
```

---

## Expected platform sync order

| Wave | App                  |
|------|----------------------|
| 0    | apps-of-apps (root)  |
| 1    | external-secrets     |
| 1    | onepassword-connect  |
| 2    | nfs-csi              |
| 3    | cert-manager         |
| 4    | ingress-nginx        |
| 5    | external-dns         |

---

## Troubleshooting

**GatewayClass not ACCEPTED after applying:**
Restart the Cilium operator — it may have started before the CRDs were present:
```bash
kubectl rollout restart deployment/cilium-operator -n kube-system
kubectl get gatewayclass
```

**ArgoCD can't reach Gitea:**
```bash
kubectl -n argocd logs deployment/argocd-repo-server | grep -i "error\|ssh"
```
Check that `gitea-repo` secret exists and known hosts configmap has `gitea.mk-labs.cloud`.

**App stuck in `Unknown` state:**
```bash
kubectl -n argocd get application <name> -o yaml | grep -A10 "conditions"
```

**ArgoCD pruning itself:**
The `argocd` namespace Applications must have `prune: false`. Never enable
prune on any Application that manages the `argocd` namespace.

**Helm valueFiles field:**
The correct field name is `valueFiles` (plural). `valuesFile` and `valuesFiles`
are silently ignored — ArgoCD syncs green but values aren't applied.

**1Password Connect base64 error:**
`illegal base64 data at input byte 0` means credentials were stored as raw JSON.
Delete and recreate:
```bash
kubectl delete secret op-credentials -n onepassword-connect
kubectl create secret generic op-credentials \
  --namespace onepassword-connect \
  --from-literal=1password-credentials.json=$(base64 -w 0 /home/wed/1password-credentials.json)
kubectl -n onepassword-connect rollout restart deployment onepassword-connect
```

**1Password Connect invalid Authorization header:**
Token has a trailing newline. Recreate:
```bash
echo -n "your-token" > /home/wed/connect-token
kubectl delete secret connect-token -n onepassword-connect
kubectl create secret generic connect-token \
  --namespace onepassword-connect \
  --from-file=token=/home/wed/connect-token
kubectl -n onepassword-connect rollout restart deployment onepassword-connect
```

---

## Post-bootstrap

Once ingress-nginx and cert-manager are synced:

- ArgoCD UI: https://argocd.local.mk-labs.cloud
- Change admin password: `argocd account update-password`
- Authentik OIDC: uncomment `oidc.config` in `cluster/argocd/values.yaml` once
  the ArgoCD provider is configured in Authentik
- Migrate `gitea-repo` secret to ESO once 1Password Connect is healthy

---

## Operational Runbook — Adding a New Application

When deploying a new application that needs HTTPS access via the Cilium Gateway,
follow this checklist:

### 1. Add the namespace to the ReferenceGrant

Edit `cluster/platform/gateway/referencegrant.yaml` and add a new `from` entry:

```yaml
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: <your-app-namespace>
```

This is **required** by the Gateway API spec — without it, the HTTPRoute will
fail to attach to the Gateway and traffic won't route. The ReferenceGrant must
live in the `gateway` namespace, so this file is the one place to update.

### 2. Create the application directory

```
cluster/applications/<app-name>/
├── application.yaml    ← ArgoCD Application, wave 10+
├── values.yaml         ← Helm values (if Helm-based)
├── externalsecret.yaml ← ESO resource if secrets needed
└── httproute.yaml      ← Certificate + HTTPRoute for Gateway access
```

### 3. HTTPRoute pattern

Always include all fields explicitly — Cilium defaults `group`, `kind`, `weight`,
and `matches` at reconcile time, causing ArgoCD out-of-sync if omitted.

Full HTTPRoute template:

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: <service>
  namespace: <app-namespace>
  annotations:
    external-dns.alpha.kubernetes.io/hostname: <service>.local.mk-labs.cloud
    external-dns.alpha.kubernetes.io/target: "10.1.71.90"
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: fastpass-gateway
      namespace: gateway
      sectionName: https
  hostnames:
    - <service>.local.mk-labs.cloud
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - group: ""
          kind: Service
          name: <service>
          port: <port>
          weight: 1
```

### 4. Per-service certificate

Each service gets its own cert via cert-manager. Add to `httproute.yaml`:

```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: <service>-tls
  namespace: <app-namespace>
spec:
  secretName: <service>-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - <service>.local.mk-labs.cloud
```

### 5. Push and verify

```bash
# Watch ArgoCD sync
kubectl -n argocd get applications --watch

# Verify cert issued
kubectl get certificate -n <app-namespace> -w

# Verify DNS record created
nslookup <service>.local.mk-labs.cloud 10.1.71.32

# Test
curl https://<service>.local.mk-labs.cloud
```