cilium, argo

2026-05-18 13:20:42 -05:00
parent 6f0c751f63
commit cdc78955b1
18 changed files with 821 additions and 136 deletions
--- a/cluster/argocd/BOOTSTRAP.md
+++ b/cluster/argocd/BOOTSTRAP.md
@@ -1,136 +1,180 @@
-# fastpass GitOps Bootstrap
+# fastpass — ArgoCD Bootstrap

-This document is the complete procedure to go from a freshly provisioned Talos
-cluster to a fully GitOps-managed platform. After completing these steps, all
-future changes are made via git — never `kubectl apply` again.
+This document covers the manual steps required to bootstrap ArgoCD on the
+`fastpass` cluster. After these steps, all further changes are made via Git.

-## Prerequisites
+**Prerequisites:**
+- All 6 Talos nodes `Ready` (`kubectl get nodes`)
+- Cilium healthy (`cilium status --wait`)
+- IP pools applied (`kubectl get ciliumloadbalancerippool`)
+- `city-hall` has `helm`, `kubectl`, and `talosctl` installed
+- SSH keypair for ArgoCD → Gitea exists at `/home/wed/.ssh/argocd_gitea`

- Talos cluster running (`space-mountain`, `big-thunder-mountain`, `splash-mountain` CP nodes)
- Cilium installed as CNI (done at Talos bootstrap via talhelper)
- `kubectl` configured and pointing at `fastpass`
- `kubeconfig` on `city-hall` at `~/.kube/config`
- 1Password Connect credentials JSON downloaded from your 1Password account
-  (Developer Tools > Connect Servers > New Server)
- 1Password Connect token generated for the `mk-labs` vault
+---

-## Step 1 — Install ArgoCD
-
-This is the only `kubectl apply` you will ever run against this cluster.
+## Step 1 — Create the argocd namespace

 ```bash
-kubectl create namespace argocd
-
-kubectl apply -n argocd \
-  --server-side \
-  --force-conflicts \
-  -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.2/manifests/install.yaml
-
-# Wait for ArgoCD to be healthy
-kubectl wait --for=condition=available deployment/argocd-server \
-  -n argocd --timeout=120s
+kubectl apply -f cluster/argocd/namespace.yaml
 ```

-## Step 2 — Bootstrap the 1Password Connect secret
+---

-This is the only secret you will ever manage manually. Everything else flows
-through ESO from here.
+## Step 2 — Add Gitea to ArgoCD known hosts
+
+ArgoCD refuses SSH connections to unknown hosts. Pre-populate the known hosts
+configmap before ArgoCD starts:

 ```bash
-# Create the namespace first
-kubectl create namespace onepassword-connect
-
-# Apply the credentials file (downloaded from 1Password)
-kubectl create secret generic op-credentials \
-  --from-file=1password-credentials.json=/path/to/1password-credentials.json \
-  -n onepassword-connect
-
-# Apply the Connect API token
-kubectl create secret generic connect-token \
-  --from-literal=token=<your-connect-token> \
-  -n onepassword-connect
+ssh-keyscan -t ed25519 gitea.mk-labs.cloud | \
+  kubectl create configmap argocd-ssh-known-hosts-cm \
+  --namespace argocd \
+  --from-file=ssh_known_hosts=/dev/stdin \
+  --dry-run=client -o yaml | kubectl apply -f -
 ```

-## Step 3 — Apply the App of Apps
+---

-This is the second and final manual apply. From this point on, ArgoCD manages
-everything, including itself.
+## Step 3 — Create the Gitea repo secret
+
+This is the only credential that must be manually created. Once ESO is running,
+secrets will be managed via 1Password. For now, create it directly:
+
+```bash
+kubectl create secret generic gitea-repo \
+  --namespace argocd \
+  --from-literal=type=git \
+  --from-literal=url=git@gitea.mk-labs.cloud:rblundon/homelab.git \
+  --from-file=sshPrivateKey=/home/wed/.ssh/argocd_gitea
+
+kubectl label secret gitea-repo -n argocd \
+  argocd.argoproj.io/secret-type=repository
+```
+
+Verify:
+```bash
+kubectl -n argocd get secret gitea-repo
+```
+
+---
+
+## Step 4 — Install ArgoCD via Helm
+
+```bash
+helm repo add argo https://argoproj.github.io/argo-helm
+helm repo update
+
+helm upgrade --install argocd argo/argo-cd \
+  --namespace argocd \
+  --create-namespace \
+  --version 7.8.23 \
+  --values cluster/argocd/values.yaml \
+  --wait
+```
+
+Wait for all ArgoCD pods to be ready:
+
+```bash
+kubectl -n argocd get pods --watch
+```
+
+---
+
+## Step 5 — Get the initial admin password
+
+```bash
+kubectl -n argocd get secret argocd-initial-admin-secret \
+  -o jsonpath='{.data.password}' | base64 -d; echo
+```
+
+Access the UI via port-forward (ingress-nginx isn't installed yet at this point):
+
+```bash
+kubectl port-forward svc/argocd-server -n argocd 8080:80
+```
+
+Browse to http://localhost:8080 — login with `admin` and the password above.
+
+---
+
+## Step 6 — Apply the app-of-apps
+
+This is the second and final manual `kubectl apply`. It hands control to ArgoCD:

 ```bash
 kubectl apply -n argocd -f cluster/argocd/apps-of-apps.yaml
 ```

-## Step 4 — Watch it go
+From this point, all changes are made via Git. ArgoCD will begin syncing
+`cluster/platform/` and `cluster/applications/` automatically.

+Watch the sync:
 ```bash
-# Watch all Applications come up in wave order
-kubectl get applications -n argocd -w
-
-# Or use the ArgoCD UI
-kubectl port-forward svc/argocd-server -n argocd 8080:443
-# Then open https://localhost:8080
-# Default admin password:
-kubectl -n argocd get secret argocd-initial-admin-secret \
-  -o jsonpath="{.data.password}" | base64 -d
+kubectl -n argocd get applications --watch
 ```

-## Step 2.5 — Vendor Gateway API CRDs
+---

-Before applying the App of Apps, vendor the Gateway API CRDs into git.
-Run from the repo root on city-hall:
+## Step 7 — Approve CSRs
+
+After nodes join and kubelet starts requesting certs, approve pending CSRs:

 ```bash
-curl -L https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml \
-  -o cluster/platform/gateway-api/manifests/standard-install.yaml
-
-curl -L https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/experimental-install.yaml \
-  -o cluster/platform/gateway-api/manifests/experimental-install.yaml
-
-git add cluster/platform/gateway-api/manifests/
-git commit -m "feat(platform): vendor gateway-api CRDs v1.2.1"
-git push
+kubectl get csr --no-headers | awk '{print $1}' | xargs kubectl certificate approve
 ```

-## Wave Order
+Run this periodically until no pending CSRs remain.

-| Wave | Application         | Depends On                      |
-|------|---------------------|---------------------------------|
-| 1    | external-secrets    | (none)                          |
-| 2    | onepassword-connect | external-secrets                |
-| 3    | cert-manager        | onepassword-connect (CF secret) |
-| 3    | gateway-api-crds    | (none — CRDs only)              |
-| 4    | cilium-config       | gateway-api-crds (CRDs exist)   |
-| 5    | ingress-nginx       | cilium-config (LB pool)         |
-| 5    | gateway (resource)  | cilium-config (GatewayClass)    |
-| 6    | external-dns        | ingress-nginx (IP to register)  |
-| 10+  | applications/*      | platform complete               |
+---

-## Upgrading ArgoCD
+## Expected platform sync order

-```bash
-kubectl apply -n argocd \
-  --server-side \
-  --force-conflicts \
-  -f https://raw.githubusercontent.com/argoproj/argo-cd/v<NEW_VERSION>/manifests/install.yaml
-```
+Apps sync in wave order via `argocd.argoproj.io/sync-wave` annotations:

-Update the version comment in `cluster/argocd/install.yaml` to match.
+| Wave | App |
+|------|-----|
+| 0 | apps-of-apps (root) |
+| 1 | cert-manager CRDs |
+| 2 | cert-manager, external-secrets |
+| 3 | ingress-nginx |
+| 4 | external-dns |
+| 5+ | application workloads |
+
+---

 ## Troubleshooting

+**ArgoCD can't reach Gitea:**
 ```bash
-# Check sync status of all apps
-kubectl get applications -n argocd
-
-# Check why an app is out of sync
-kubectl describe application <name> -n argocd
-
-# Force a sync
-kubectl patch application <name> -n argocd \
-  --type merge \
-  -p '{"operation": {"initiatedBy": {"username": "admin"}, "sync": {}}}'
-
-# Check ESO secret sync
-kubectl get externalsecret -A
-kubectl describe externalsecret <name> -n <namespace>
+kubectl -n argocd logs deployment/argocd-repo-server | grep -i "error\|ssh"
 ```
+Check that the `gitea-repo` secret exists and the known hosts configmap has
+`gitea.mk-labs.cloud`.
+
+**App stuck in `Unknown` state:**
+```bash
+kubectl -n argocd get application <name> -o yaml | grep -A10 "conditions"
+```
+
+**self-reference loop (app syncing itself):**
+Ensure `apps-of-apps.yaml` uses `include: "*/application.yaml"` with
+`recurse: true` — this prevents ArgoCD from picking up its own manifest.
+
+**ArgoCD pruning itself:**
+The `argocd` namespace Applications must have `prune: false`. Never enable
+prune on any Application that manages the `argocd` namespace.
+
+**Helm valueFiles field:**
+The correct field name is `valueFiles` (plural). `valuesFile` and `valuesFiles`
+are silently ignored — ArgoCD syncs green but your values aren't applied.
+
+---
+
+## Post-bootstrap
+
+Once ingress-nginx and cert-manager are synced:
+- ArgoCD UI available at https://argocd.local.mk-labs.cloud
+- Change the admin password: `argocd account update-password`
+- Authentik OIDC: uncomment the `oidc.config` block in `values.yaml` once
+  the ArgoCD provider is configured in Authentik
+- Migrate `gitea-repo` secret to ESO once 1Password Connect is running
--- a/cluster/argocd/values.yaml
+++ b/cluster/argocd/values.yaml
@@ -0,0 +1,110 @@
+# ─── ArgoCD Helm values — fastpass cluster ───────────────────────────────────
+# Chart:   argo/argo-cd
+# Version: 7.8.23
+#
+# Install:
+#   helm repo add argo https://argoproj.github.io/argo-helm
+#   helm repo update
+#   helm upgrade --install argocd argo/argo-cd \
+#     --namespace argocd \
+#     --create-namespace \
+#     --version 7.8.23 \
+#     --values cluster/argocd/values.yaml \
+#     --wait
+#
+# Then apply the root app-of-apps (second and final manual step):
+#   kubectl apply -n argocd -f cluster/argocd/apps-of-apps.yaml
+# ─────────────────────────────────────────────────────────────────────────────
+
+# ─── Global ──────────────────────────────────────────────────────────────────
+global:
+  domain: argocd.local.mk-labs.cloud
+
+# ─── ArgoCD server ───────────────────────────────────────────────────────────
+server:
+  # Run insecure — TLS is terminated at ingress-nginx, not at ArgoCD itself.
+  # Without this, ArgoCD redirects to HTTPS internally and breaks behind ingress.
+  extraArgs:
+    - --insecure
+
+  ingress:
+    enabled: true
+    ingressClassName: nginx
+    hostname: argocd.local.mk-labs.cloud
+    tls: true
+    annotations:
+      cert-manager.io/cluster-issuer: letsencrypt-production
+      nginx.ingress.kubernetes.io/ssl-passthrough: "false"
+      nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
+
+# ─── ConfigMap — core ArgoCD settings ────────────────────────────────────────
+configs:
+  params:
+    # Matches --insecure above — tells the server not to handle TLS
+    server.insecure: "true"
+
+  cm:
+    # Gitea as the source repo — HTTPS with token auth (set in secret below)
+    # SSH alternative is in the repositories section if preferred
+    url: https://argocd.local.mk-labs.cloud
+
+    # Authentik OIDC SSO
+    # Uncomment once Authentik is configured with an ArgoCD OIDC provider
+    # oidc.config: |
+    #   name: Authentik
+    #   issuer: https://authentik.local.mk-labs.cloud/application/o/argocd/
+    #   clientID: <client-id-from-authentik>
+    #   clientSecret: $oidc.authentik.clientSecret
+    #   requestedScopes:
+    #     - openid
+    #     - profile
+    #     - email
+    #     - groups
+
+    # Resource health checks — ignore progressing Cilium CRDs
+    resource.customizations.health.cilium.io_CiliumLoadBalancerIPPool: |
+      hs = {}
+      hs.status = "Healthy"
+      hs.message = "IP pool ready"
+      return hs
+
+  # RBAC — admin gets full access, SSO groups mapped after Authentik integration
+  rbac:
+    policy.default: role:readonly
+    policy.csv: |
+      g, argocd-admins, role:admin
+
+  # Repository credentials — Gitea SSH
+  # Private key is injected via ESO once external-secrets is bootstrapped.
+  # For initial bootstrap, create the secret manually:
+  #   kubectl create secret generic gitea-repo \
+  #     --namespace argocd \
+  #     --from-literal=type=git \
+  #     --from-literal=url=git@gitea.mk-labs.cloud:rblundon/homelab.git \
+  #     --from-file=sshPrivateKey=/path/to/argocd_gitea_ed25519 \
+  #   kubectl label secret gitea-repo -n argocd \
+  #     argocd.argoproj.io/secret-type=repository
+  repositories:
+    homelab-repo:
+      url: git@gitea.mk-labs.cloud:rblundon/homelab.git
+      name: homelab
+      type: git
+      insecure: "false"
+
+# ─── Redis ───────────────────────────────────────────────────────────────────
+redis:
+  enabled: true
+
+# ─── Notifications ───────────────────────────────────────────────────────────
+# Disabled for now — wire into n8n via webhook later
+notifications:
+  enabled: false
+
+# ─── ApplicationSet controller ───────────────────────────────────────────────
+applicationSet:
+  enabled: true
+
+# ─── Dex (built-in OIDC) ─────────────────────────────────────────────────────
+# Disabled — using Authentik as the OIDC provider
+dex:
+  enabled: false