diff --git a/cluster/argocd/BOOTSTRAP.md b/cluster/bootstrap/README.md similarity index 53% rename from cluster/argocd/BOOTSTRAP.md rename to cluster/bootstrap/README.md index c502125..4a5c911 100644 --- a/cluster/argocd/BOOTSTRAP.md +++ b/cluster/bootstrap/README.md @@ -1,23 +1,62 @@ -# fastpass — ArgoCD Bootstrap +# fastpass — Cluster Bootstrap -This document covers the manual steps required to bootstrap ArgoCD on the -`fastpass` cluster. After these steps, all further changes are made via Git. +This directory contains resources and documentation for the **manual bootstrap +sequence** required to bring up `fastpass` from a fresh Talos install. + +Everything in this directory is applied **once by hand** before ArgoCD takes +over. After Step 6 (app-of-apps), all further changes are made via Git. + +--- + +## Prerequisites -**Prerequisites:** - All 6 Talos nodes `Ready` (`kubectl get nodes`) - Cilium healthy (`cilium status --wait`) - IP pools applied (`kubectl get ciliumloadbalancerippool`) - `city-hall` has `helm`, `kubectl`, and `talosctl` installed - SSH keypair for ArgoCD → Gitea exists at `/home/wed/.ssh/argocd_gitea` - 1Password credentials file at `/home/wed/1password-credentials.json` -- 1Password Connect token at `/home/wed/connect-token` (no trailing newline — use `echo -n`) +- 1Password Connect token at `/home/wed/connect-token` (no trailing newline) --- -## Step 1 — Create the argocd namespace +## Step 1 — Install Gateway API CRDs -Must be created first — the Gitea repo secret targets this namespace before -Helm creates it. +The Gateway API CRDs are not bundled with Cilium — they must be installed +independently before Cilium can register the GatewayClass controller. + +```bash +kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml +``` + +Verify: + +```bash +kubectl get crd gateways.gateway.networking.k8s.io +``` + +--- + +## Step 2 — Create the Cilium GatewayClass + +The Cilium operator manages this object but does not auto-create it. + +```bash +kubectl apply -f cluster/bootstrap/gatewayclass.yaml +``` + +Verify (may take a few seconds): + +```bash +kubectl get gatewayclass +# Expected: cilium io.cilium/gateway-controller True +``` + +--- + +## Step 3 — Create the argocd namespace + +Must be created before Helm — the Gitea repo secret targets this namespace. ```bash kubectl apply -f cluster/argocd/namespace.yaml @@ -25,7 +64,7 @@ kubectl apply -f cluster/argocd/namespace.yaml --- -## Step 2 — Create the Gitea repo secret +## Step 4 — Create the Gitea repo secret ```bash kubectl create secret generic gitea-repo \ @@ -39,13 +78,14 @@ kubectl label secret gitea-repo -n argocd \ ``` Verify: + ```bash kubectl -n argocd get secret gitea-repo ``` --- -## Step 3 — Install ArgoCD via Helm +## Step 5 — Install ArgoCD via Helm ```bash helm repo add argo https://argoproj.github.io/argo-helm @@ -67,9 +107,7 @@ kubectl -n argocd get pods --watch --- -## Step 4 — Add Gitea to ArgoCD known hosts - -The known hosts configmap is created by Helm — patch it after install: +## Step 6 — Add Gitea to ArgoCD known hosts ```bash ssh-keyscan -t ed25519 gitea.mk-labs.cloud 2>/dev/null | \ @@ -80,149 +118,131 @@ ssh-keyscan -t ed25519 gitea.mk-labs.cloud 2>/dev/null | \ --- -## Step 5 — Get the initial admin password +## Step 7 — Get the initial admin password ```bash kubectl -n argocd get secret argocd-initial-admin-secret \ -o jsonpath='{.data.password}' | base64 -d; echo ``` -Access the UI via port-forward (ingress-nginx isn't installed yet at this point): +Access the UI via port-forward (ingress not yet deployed): ```bash kubectl port-forward svc/argocd-server -n argocd 8080:80 > /tmp/pf.log 2>&1 & ``` -Browse to http://localhost:8080 — login with `admin` and the password above. +Browse to http://localhost:8080 --- -## Step 6 — Apply the app-of-apps +## Step 8 — Apply the app-of-apps -This is the second and final manual `kubectl apply`. It hands control to ArgoCD: +This is the final manual `kubectl apply`. It hands control to ArgoCD: ```bash kubectl apply -n argocd -f cluster/argocd/apps-of-apps.yaml ``` -From this point, all changes are made via Git. ArgoCD will begin syncing -`cluster/platform/` and `cluster/applications/` automatically. +From this point, all changes are made via Git. ArgoCD syncs `cluster/platform/` +and `cluster/applications/` automatically. Watch the sync: + ```bash kubectl -n argocd get applications --watch ``` --- -## Step 7 — Approve CSRs - -After nodes join and kubelet starts requesting certs, approve pending CSRs: +## Step 9 — Approve CSRs ```bash kubectl get csr --no-headers | awk '{print $1}' | xargs kubectl certificate approve ``` -Run this periodically until no pending CSRs remain. +Run periodically until no pending CSRs remain. --- -## Step 8 — Bootstrap 1Password Connect secrets +## Step 10 — Bootstrap 1Password Connect secrets -Once the `onepassword-connect` namespace exists (created by ArgoCD sync), -create the two secrets the Connect server requires. - -**Important notes:** -- The credentials file must be stored **base64-encoded** in the secret -- The token file must have **no trailing newline** — use `echo -n` when creating it -- Both secrets go in the `onepassword-connect` namespace, not `external-secrets` -- Secret names must match exactly: `op-credentials` and `connect-token` +Once `onepassword-connect` namespace exists (created by ArgoCD sync): ```bash -# Verify the namespace exists before proceeding +# Verify namespace exists kubectl get namespace onepassword-connect -# Create credentials secret — base64-encode the file contents +# Credentials secret — must be base64-encoded kubectl create secret generic op-credentials \ --namespace onepassword-connect \ --from-literal=1password-credentials.json=$(base64 -w 0 /home/wed/1password-credentials.json) -# Create token secret — token file must have no trailing newline +# Token secret — file must have no trailing newline kubectl create secret generic connect-token \ --namespace onepassword-connect \ --from-file=token=/home/wed/connect-token ``` -Restart the Connect deployment to pick up the secrets: +Restart Connect to pick up secrets: ```bash kubectl -n onepassword-connect rollout restart deployment onepassword-connect kubectl -n onepassword-connect rollout status deployment onepassword-connect ``` -Verify Connect is healthy: +Verify (should show only GET /health and GET /heartbeat 200s): ```bash kubectl -n onepassword-connect logs deployment/onepassword-connect -c connect-api 2>&1 | tail -10 ``` -You should see only `GET /health` and `GET /heartbeat` 200 responses — no errors. - --- ## Expected platform sync order -Apps sync in wave order via `argocd.argoproj.io/sync-wave` annotations: - -| Wave | App | -|------|-----| -| 0 | apps-of-apps (root) | -| 1 | cert-manager CRDs | -| 2 | cert-manager, external-secrets | -| 3 | ingress-nginx | -| 4 | external-dns | -| 5+ | application workloads | +| Wave | App | +|------|----------------------| +| 0 | apps-of-apps (root) | +| 1 | external-secrets | +| 1 | onepassword-connect | +| 2 | nfs-csi | +| 3 | cert-manager | +| 4 | ingress-nginx | +| 5 | external-dns | --- ## Troubleshooting +**GatewayClass not ACCEPTED after applying:** +Restart the Cilium operator — it may have started before the CRDs were present: +```bash +kubectl rollout restart deployment/cilium-operator -n kube-system +kubectl get gatewayclass +``` + **ArgoCD can't reach Gitea:** ```bash kubectl -n argocd logs deployment/argocd-repo-server | grep -i "error\|ssh" ``` -Check that the `gitea-repo` secret exists and the known hosts configmap has -`gitea.mk-labs.cloud`. +Check that `gitea-repo` secret exists and known hosts configmap has `gitea.mk-labs.cloud`. **App stuck in `Unknown` state:** ```bash kubectl -n argocd get application -o yaml | grep -A10 "conditions" ``` -**Self-reference loop (app syncing itself):** -Ensure `apps-of-apps.yaml` uses `include: "*/application.yaml"` with -`recurse: true` — this prevents ArgoCD from picking up its own manifest. - **ArgoCD pruning itself:** The `argocd` namespace Applications must have `prune: false`. Never enable prune on any Application that manages the `argocd` namespace. **Helm valueFiles field:** The correct field name is `valueFiles` (plural). `valuesFile` and `valuesFiles` -are silently ignored — ArgoCD syncs green but your values aren't applied. - -**argocd-cm configmap missing:** -If the UI shows `configmap "argocd-cm" not found`, recreate it: -```bash -kubectl create configmap argocd-cm \ - --namespace argocd \ - --from-literal=url=https://argocd.local.mk-labs.cloud -kubectl -n argocd rollout restart deployment argocd-server -``` +are silently ignored — ArgoCD syncs green but values aren't applied. **1Password Connect base64 error:** -`illegal base64 data at input byte 0` means the credentials secret was created -with raw JSON instead of base64-encoded content. Delete and recreate: +`illegal base64 data at input byte 0` means credentials were stored as raw JSON. +Delete and recreate: ```bash kubectl delete secret op-credentials -n onepassword-connect kubectl create secret generic op-credentials \ @@ -232,8 +252,7 @@ kubectl -n onepassword-connect rollout restart deployment onepassword-connect ``` **1Password Connect invalid Authorization header:** -`invalid header field value for "Authorization"` means the token has a trailing -newline. Recreate the token file and secret: +Token has a trailing newline. Recreate: ```bash echo -n "your-token" > /home/wed/connect-token kubectl delete secret connect-token -n onepassword-connect @@ -248,8 +267,9 @@ kubectl -n onepassword-connect rollout restart deployment onepassword-connect ## Post-bootstrap Once ingress-nginx and cert-manager are synced: -- ArgoCD UI available at https://argocd.local.mk-labs.cloud -- Change the admin password: `argocd account update-password` -- Authentik OIDC: uncomment the `oidc.config` block in `values.yaml` once + +- ArgoCD UI: https://argocd.local.mk-labs.cloud +- Change admin password: `argocd account update-password` +- Authentik OIDC: uncomment `oidc.config` in `cluster/argocd/values.yaml` once the ArgoCD provider is configured in Authentik -- Migrate `gitea-repo` secret to ESO once 1Password Connect is running +- Migrate `gitea-repo` secret to ESO once 1Password Connect is healthy diff --git a/cluster/bootstrap/gatewayclass.yaml b/cluster/bootstrap/gatewayclass.yaml new file mode 100644 index 0000000..ebe6cb1 --- /dev/null +++ b/cluster/bootstrap/gatewayclass.yaml @@ -0,0 +1,15 @@ +# ------------------------------------------------------------------------------ +# Cilium GatewayClass +# +# Applied once during cluster bootstrap — before ArgoCD takes over. +# The Cilium operator manages this object once created but does NOT +# auto-create it. Must exist before any Gateway or HTTPRoute resources. +# +# Apply: kubectl apply -f cluster/bootstrap/gatewayclass.yaml +# ------------------------------------------------------------------------------ +apiVersion: gateway.networking.k8s.io/v1 +kind: GatewayClass +metadata: + name: cilium +spec: + controllerName: io.cilium/gateway-controller