Initial argo deployment

cluster/argocd/BOOTSTRAP.md

@@ -0,0 +1,136 @@
+# fastpass GitOps Bootstrap
+
+This document is the complete procedure to go from a freshly provisioned Talos
+cluster to a fully GitOps-managed platform. After completing these steps, all
+future changes are made via git — never `kubectl apply` again.
+
+## Prerequisites
+
+- Talos cluster running (`space-mountain`, `big-thunder-mountain`, `splash-mountain` CP nodes)
+- Cilium installed as CNI (done at Talos bootstrap via talhelper)
+- `kubectl` configured and pointing at `fastpass`
+- `kubeconfig` on `city-hall` at `~/.kube/config`
+- 1Password Connect credentials JSON downloaded from your 1Password account
+  (Developer Tools > Connect Servers > New Server)
+- 1Password Connect token generated for the `mk-labs` vault
+
+## Step 1 — Install ArgoCD
+
+This is the only `kubectl apply` you will ever run against this cluster.
+
+```bash
+kubectl create namespace argocd
+
+kubectl apply -n argocd \
+  --server-side \
+  --force-conflicts \
+  -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.2/manifests/install.yaml
+
+# Wait for ArgoCD to be healthy
+kubectl wait --for=condition=available deployment/argocd-server \
+  -n argocd --timeout=120s
+```
+
+## Step 2 — Bootstrap the 1Password Connect secret
+
+This is the only secret you will ever manage manually. Everything else flows
+through ESO from here.
+
+```bash
+# Create the namespace first
+kubectl create namespace onepassword-connect
+
+# Apply the credentials file (downloaded from 1Password)
+kubectl create secret generic op-credentials \
+  --from-file=1password-credentials.json=/path/to/1password-credentials.json \
+  -n onepassword-connect
+
+# Apply the Connect API token
+kubectl create secret generic connect-token \
+  --from-literal=token=<your-connect-token> \
+  -n onepassword-connect
+```
+
+## Step 3 — Apply the App of Apps
+
+This is the second and final manual apply. From this point on, ArgoCD manages
+everything, including itself.
+
+```bash
+kubectl apply -n argocd -f cluster/argocd/apps-of-apps.yaml
+```
+
+## Step 4 — Watch it go
+
+```bash
+# Watch all Applications come up in wave order
+kubectl get applications -n argocd -w
+
+# Or use the ArgoCD UI
+kubectl port-forward svc/argocd-server -n argocd 8080:443
+# Then open https://localhost:8080
+# Default admin password:
+kubectl -n argocd get secret argocd-initial-admin-secret \
+  -o jsonpath="{.data.password}" | base64 -d
+```
+
+## Step 2.5 — Vendor Gateway API CRDs
+
+Before applying the App of Apps, vendor the Gateway API CRDs into git.
+Run from the repo root on city-hall:
+
+```bash
+curl -L https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml \
+  -o cluster/platform/gateway-api/manifests/standard-install.yaml
+
+curl -L https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/experimental-install.yaml \
+  -o cluster/platform/gateway-api/manifests/experimental-install.yaml
+
+git add cluster/platform/gateway-api/manifests/
+git commit -m "feat(platform): vendor gateway-api CRDs v1.2.1"
+git push
+```
+
+## Wave Order
+
+| Wave | Application         | Depends On                      |
+|------|---------------------|---------------------------------|
+| 1    | external-secrets    | (none)                          |
+| 2    | onepassword-connect | external-secrets                |
+| 3    | cert-manager        | onepassword-connect (CF secret) |
+| 3    | gateway-api-crds    | (none — CRDs only)              |
+| 4    | cilium-config       | gateway-api-crds (CRDs exist)   |
+| 5    | ingress-nginx       | cilium-config (LB pool)         |
+| 5    | gateway (resource)  | cilium-config (GatewayClass)    |
+| 6    | external-dns        | ingress-nginx (IP to register)  |
+| 10+  | applications/*      | platform complete               |
+
+## Upgrading ArgoCD
+
+```bash
+kubectl apply -n argocd \
+  --server-side \
+  --force-conflicts \
+  -f https://raw.githubusercontent.com/argoproj/argo-cd/v<NEW_VERSION>/manifests/install.yaml
+```
+
+Update the version comment in `cluster/argocd/install.yaml` to match.
+
+## Troubleshooting
+
+```bash
+# Check sync status of all apps
+kubectl get applications -n argocd
+
+# Check why an app is out of sync
+kubectl describe application <name> -n argocd
+
+# Force a sync
+kubectl patch application <name> -n argocd \
+  --type merge \
+  -p '{"operation": {"initiatedBy": {"username": "admin"}, "sync": {}}}'
+
+# Check ESO secret sync
+kubectl get externalsecret -A
+kubectl describe externalsecret <name> -n <namespace>
+```

cluster/argocd/apps-of-apps.yaml

@@ -0,0 +1,45 @@
+# ------------------------------------------------------------------------------
+# App of Apps — Root
+#
+# This is applied ONCE after ArgoCD is bootstrapped. It is the second and
+# final manual kubectl apply:
+#
+#   kubectl apply -n argocd -f cluster/argocd/apps-of-apps.yaml
+#
+# From this point forward, all changes are made via git. ArgoCD self-manages.
+#
+# This Application watches cluster/platform/ and cluster/applications/ and
+# creates child Applications for everything it finds there.
+# ------------------------------------------------------------------------------
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: apps-of-apps
+  namespace: argocd
+  annotations:
+    # Wave 0 — this is the root, everything else comes after
+    argocd.argoproj.io/sync-wave: "0"
+spec:
+  project: default
+  source:
+    repoURL: https://gitea.mk-labs.cloud/rblundon/homelab.git
+    targetRevision: main
+    path: cluster
+    directory:
+      recurse: true
+      include: "*/application.yaml"
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: argocd
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+      - ServerSideApply=true
+  ignoreDifferences:
+    - group: argoproj.io
+      kind: Application
+      jsonPointers:
+        - /spec/syncPolicy/automated

cluster/argocd/install.yaml

@@ -0,0 +1,21 @@
+# ------------------------------------------------------------------------------
+# ArgoCD Install — v3.4.2
+#
+# This is NOT applied by ArgoCD itself (chicken-and-egg).
+# It is the ONE manual bootstrap step:
+#
+#   kubectl create namespace argocd
+#   kubectl apply -n argocd --server-side --force-conflicts -f install.yaml
+#
+# After this, everything else is driven by apps-of-apps.yaml.
+#
+# To upgrade ArgoCD: update the version below and re-apply.
+# --server-side --force-conflicts is required due to CRD size limits.
+#
+# Source: https://github.com/argoproj/argo-cd/releases/tag/v3.4.2
+# ------------------------------------------------------------------------------
+# kubectl apply -n argocd --server-side --force-conflicts -f \
+#   https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.2/manifests/install.yaml
+#
+# We reference the upstream manifest directly rather than vendoring it.
+# Pin upgrades by changing the version tag above and re-running the command.

cluster/argocd/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: argocd

cluster/platform/argocd/application.yaml

@@ -0,0 +1,44 @@
+# ------------------------------------------------------------------------------
+# Platform: ArgoCD Configuration (post-bootstrap)
+# Wave 2 — after ESO (wave 1), needs 1Password for Authentik client secret
+#
+# This is NOT the ArgoCD install (that's a manual bootstrap step).
+# This Application manages ArgoCD's own configuration declaratively:
+#   - argocd-cm       (URL, Dex/OIDC config, repo credentials)
+#   - argocd-rbac-cm  (RBAC policies mapped from Authentik groups)
+#   - argocd-secret   (Authentik client secret via ESO)
+#
+# ArgoCD managing its own config is intentional and works cleanly —
+# it's the "self-managed ArgoCD" pattern.
+# ------------------------------------------------------------------------------
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: argocd-config
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "2"
+spec:
+  project: default
+  source:
+    repoURL: https://gitea.mk-labs.cloud/rblundon/homelab.git
+    targetRevision: main
+    path: cluster/platform/argocd
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: argocd
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - ServerSideApply=true
+  ignoreDifferences:
+    # ArgoCD mutates its own secret — ignore those fields
+    - group: ""
+      kind: Secret
+      name: argocd-secret
+      jsonPointers:
+        - /data/admin.password
+        - /data/admin.passwordMtime
+        - /data/server.secretkey

cluster/platform/argocd/argocd-cm.yaml

@@ -0,0 +1,69 @@
+# ------------------------------------------------------------------------------
+# argocd-cm — ArgoCD Core Configuration
+#
+# Configures:
+#   - ArgoCD public URL (used in Dex redirect URIs)
+#   - Dex connector to Authentik OIDC
+#   - Gitea repo credentials reference
+#
+# Authentik side setup (manual — do this in guest-relations before bootstrapping):
+#   1. Admin UI → Applications → Create New Application
+#   2. Provider type: OAuth2/OpenID Connect
+#   3. Name: argocd
+#   4. Authorization flow: default-provider-authorization-explicit-consent
+#   5. Client type: Confidential
+#   6. Strict redirect URIs:
+#        https://argocd.local.mk-labs.cloud/api/dex/callback
+#        http://localhost:8085/auth/callback        (ArgoCD CLI)
+#   7. Scopes: openid, profile, email, groups
+#   8. Note the Client ID and Client Secret → store in 1Password as:
+#        Item: "argocd-oidc"
+#        Fields: client-id, client-secret
+#   9. Create two groups in Authentik:
+#        - argocd-admins  (add your user)
+#        - argocd-viewers (read-only users)
+# ------------------------------------------------------------------------------
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: argocd-cm
+  namespace: argocd
+data:
+  # Public URL — must match the HTTPRoute hostname
+  url: https://argocd.local.mk-labs.cloud
+
+  # Dex connector to Authentik
+  # Using Dex (not native OIDC) to preserve ArgoCD CLI functionality
+  dex.config: |
+    connectors:
+      - type: oidc
+        id: authentik
+        name: Authentik
+        config:
+          # Authentik OIDC issuer — slug must match the Application slug in Authentik
+          # Slug is auto-generated from the app name, usually lowercase with hyphens
+          issuer: https://authentik.local.mk-labs.cloud/application/o/argocd/
+          clientID: $dex.authentik.clientID
+          clientSecret: $dex.authentik.clientSecret
+          redirectURI: https://argocd.local.mk-labs.cloud/api/dex/callback
+          scopes:
+            - openid
+            - profile
+            - email
+            - groups
+          insecureEnableGroups: true
+          groupsKey: groups
+          userIDKey: sub
+          userNameKey: preferred_username
+          emailKey: email
+
+  # Gitea repo — ArgoCD needs credentials to pull from your private repo
+  # Credentials are stored as a Repository secret (see repository-secret.yaml)
+  # This tells ArgoCD to use SSH for the repo
+  repositories: |
+    - url: git@gitea.mk-labs.cloud:rblundon/homelab.git
+      type: git
+      name: homelab
+
+  # Resource tracking via annotation (recommended over label tracking)
+  application.resourceTrackingMethod: annotation

cluster/platform/argocd/argocd-rbac-cm.yaml

@@ -0,0 +1,30 @@
+# ------------------------------------------------------------------------------
+# argocd-rbac-cm — RBAC Policy
+#
+# Maps Authentik groups to ArgoCD roles.
+# Groups must exist in Authentik and be assigned to users there.
+#
+# Roles:
+#   role:admin    — full access, all operations
+#   role:readonly — read-only, can view but not sync/modify
+#
+# Single-user homelab: your user should be in argocd-admins in Authentik.
+# Add argocd-viewers later when you have guests/collaborators.
+# ------------------------------------------------------------------------------
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: argocd-rbac-cm
+  namespace: argocd
+data:
+  # Default policy for any authenticated user not in a mapped group
+  # readonly is safe — they can see but not touch
+  policy.default: role:readonly
+
+  policy.csv: |
+    # Authentik group → ArgoCD role mappings
+    g, argocd-admins, role:admin
+    g, argocd-viewers, role:readonly
+
+  # Tell ArgoCD which OIDC claims to use for group mapping
+  scopes: '[groups, email]'

cluster/platform/argocd/argocd-server-patch.yaml

@@ -0,0 +1,23 @@
+# ------------------------------------------------------------------------------
+# ArgoCD Server — Insecure Mode Patch
+#
+# When TLS is terminated at the Gateway (not at argocd-server itself),
+# argocd-server must run in --insecure mode. Without this, it tries to
+# do double TLS and the Gateway connection fails.
+#
+# This patches the argocd-server Deployment to add the --insecure flag.
+# Applied as part of the argocd-config Application (same path/namespace).
+# ------------------------------------------------------------------------------
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: argocd-server
+  namespace: argocd
+spec:
+  template:
+    spec:
+      containers:
+        - name: argocd-server
+          command:
+            - argocd-server
+            - --insecure

cluster/platform/argocd/certificate.yaml

@@ -0,0 +1,41 @@
+# ------------------------------------------------------------------------------
+# Certificate — ArgoCD TLS
+# Issued by cert-manager via letsencrypt-prod + Cloudflare DNS-01
+#
+# The secret produced (argocd-tls) is referenced by the Gateway listener
+# in cluster/platform/gateway-api/gateway.yaml via ReferenceGrant.
+#
+# Per the thematic naming convention: argocd.local.mk-labs.cloud is the
+# service DNS name. No VM hostname alias needed here.
+# ------------------------------------------------------------------------------
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: argocd-tls
+  namespace: argocd
+spec:
+  secretName: argocd-tls
+  issuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer
+  dnsNames:
+    - argocd.local.mk-labs.cloud
+
+---
+# ReferenceGrant — allows the Gateway in gateway-system to reference
+# the TLS secret in the argocd namespace
+# Without this, cross-namespace secret references are denied
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: ReferenceGrant
+metadata:
+  name: argocd-tls-grant
+  namespace: argocd
+spec:
+  from:
+    - group: gateway.networking.k8s.io
+      kind: Gateway
+      namespace: gateway-system
+  to:
+    - group: ""
+      kind: Secret
+      name: argocd-tls

cluster/platform/argocd/externalsecret.yaml

@@ -0,0 +1,49 @@
+# ------------------------------------------------------------------------------
+# ExternalSecret — ArgoCD Secrets
+#
+# Pulls two secrets from 1Password:
+#   1. Authentik OIDC client ID + secret (for Dex connector)
+#   2. Gitea personal access token (for repo access)
+#
+# In 1Password (mk-labs vault), create:
+#   Item: "argocd-oidc"
+#     Field: client-id     → Authentik OAuth2 client ID
+#     Field: client-secret → Authentik OAuth2 client secret
+#
+#   Item: "gitea-argocd-token"
+#     Field: token → Gitea personal access token with repo read permissions
+#     (Create in Gitea: Settings → Applications → Generate Token)
+#
+# These materialize into argocd-secret, which ArgoCD reads natively.
+# The $dex.authentik.clientID / $dex.authentik.clientSecret references
+# in argocd-cm.yaml are resolved by ArgoCD from this secret automatically.
+# ------------------------------------------------------------------------------
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: argocd-secrets
+  namespace: argocd
+spec:
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: onepassword-connect
+  refreshInterval: "1h"
+  target:
+    # Must be named argocd-secret — ArgoCD reads this specific secret name
+    name: argocd-secret
+    creationPolicy: Merge    # Merge into existing argocd-secret, don't replace it
+  data:
+    # Authentik OIDC credentials — referenced as $dex.authentik.* in argocd-cm
+    - secretKey: dex.authentik.clientID
+      remoteRef:
+        key: argocd-oidc
+        property: client-id
+    - secretKey: dex.authentik.clientSecret
+      remoteRef:
+        key: argocd-oidc
+        property: client-secret
+    # Gitea token — used by the Repository credential below
+    - secretKey: sshPrivateKey
+      remoteRef:
+        key: gitea-argocd-ssh
+        property: private-key

cluster/platform/argocd/httproute.yaml

@@ -0,0 +1,54 @@
+# ------------------------------------------------------------------------------
+# HTTPRoute — ArgoCD
+# Routes argocd.local.mk-labs.cloud → argocd-server service
+#
+# This is the first real Gateway API HTTPRoute in the cluster — ArgoCD
+# dogfoods the Gateway it manages. Satisfying.
+#
+# TLS is terminated at the Gateway. cert-manager issues the certificate
+# via the Certificate resource below. The Gateway listener references it.
+#
+# ArgoCD server runs in insecure mode (--insecure) when TLS is terminated
+# upstream — this is correct and expected with a Gateway/proxy in front.
+# ------------------------------------------------------------------------------
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: argocd
+  namespace: argocd
+spec:
+  parentRefs:
+    - name: fastpass-gateway
+      namespace: gateway-system
+      sectionName: https
+  hostnames:
+    - argocd.local.mk-labs.cloud
+  rules:
+    - matches:
+        - path:
+            type: PathPrefix
+            value: /
+      backendRefs:
+        - name: argocd-server
+          port: 80    # argocd-server listens on 80 in insecure mode
+
+---
+# HTTP → HTTPS redirect
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: argocd-http-redirect
+  namespace: argocd
+spec:
+  parentRefs:
+    - name: fastpass-gateway
+      namespace: gateway-system
+      sectionName: http
+  hostnames:
+    - argocd.local.mk-labs.cloud
+  rules:
+    - filters:
+        - type: RequestRedirect
+          requestRedirect:
+            scheme: https
+            statusCode: 301

cluster/platform/argocd/repository-secret.yaml

@@ -0,0 +1,26 @@
+# ------------------------------------------------------------------------------
+# Repository Credential Secret — Gitea homelab repo (SSH)
+#
+# ArgoCD discovers repository credentials via secrets with the label
+# argocd.argoproj.io/secret-type: repository
+#
+# The SSH private key is injected by ESO from 1Password.
+# In 1Password (mk-labs vault):
+#   Item: "gitea-argocd-ssh"
+#   Field: private-key → full contents of the ed25519 private key PEM block
+# ------------------------------------------------------------------------------
+apiVersion: v1
+kind: Secret
+metadata:
+  name: repo-homelab-gitea
+  namespace: argocd
+  labels:
+    argocd.argoproj.io/secret-type: repository
+stringData:
+  type: git
+  url: git@gitea.mk-labs.cloud:rblundon/homelab.git
+  sshPrivateKey: |
+    -----BEGIN OPENSSH PRIVATE KEY-----
+    # This value is overwritten by ESO — placeholder only
+    # See externalsecret.yaml
+    -----END OPENSSH PRIVATE KEY-----

cluster/platform/cert-manager/application.yaml

@@ -0,0 +1,29 @@
+# ------------------------------------------------------------------------------
+# Platform: cert-manager
+# Wave 3 — depends on ESO (wave 1) for Cloudflare token secret
+# ------------------------------------------------------------------------------
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: cert-manager
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "3"
+spec:
+  project: default
+  source:
+    repoURL: https://charts.jetstack.io
+    chart: cert-manager
+    targetRevision: v1.17.2
+    helm:
+      valuesFile: values.yaml
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: cert-manager
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+      - ServerSideApply=true

cluster/platform/cert-manager/clusterissuer.yaml

@@ -0,0 +1,49 @@
+# ------------------------------------------------------------------------------
+# cert-manager ClusterIssuer — Let's Encrypt via Cloudflare DNS-01
+#
+# The cloudflare-api-token secret is populated by ESO from 1Password.
+# See externalsecret.yaml for the ESO resource that creates it.
+# ------------------------------------------------------------------------------
+
+# Staging issuer — use this first to avoid LE rate limits during testing
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-staging
+spec:
+  acme:
+    server: https://acme-staging-v02.api.letsencrypt.org/directory
+    email: ryan.blundon@protonmail.com
+    privateKeySecretRef:
+      name: letsencrypt-staging-key
+    solvers:
+      - dns01:
+          cloudflare:
+            apiTokenSecretRef:
+              name: cloudflare-api-token
+              key: api-token
+        selector:
+          dnsZones:
+            - mk-labs.cloud
+
+---
+# Production issuer — switch to this once staging certs are working
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-prod
+spec:
+  acme:
+    server: https://acme-v02.api.letsencrypt.org/directory
+    email: ryan.blundon@protonmail.com
+    privateKeySecretRef:
+      name: letsencrypt-prod-key
+    solvers:
+      - dns01:
+          cloudflare:
+            apiTokenSecretRef:
+              name: cloudflare-api-token
+              key: api-token
+        selector:
+          dnsZones:
+            - mk-labs.cloud

cluster/platform/cert-manager/externalsecret.yaml

@@ -0,0 +1,26 @@
+# ------------------------------------------------------------------------------
+# ExternalSecret — Cloudflare API Token
+# Pulled from 1Password via Connect Server, materialized as a K8s secret
+# in the cert-manager namespace for use by the ClusterIssuer.
+#
+# In 1Password, store the token as an item named "cloudflare" with a field
+# named "api-token" in the "mk-labs" vault.
+# ------------------------------------------------------------------------------
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: cloudflare-api-token
+  namespace: cert-manager
+spec:
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: onepassword-connect
+  refreshInterval: "1h"
+  target:
+    name: cloudflare-api-token
+    creationPolicy: Owner
+  data:
+    - secretKey: api-token
+      remoteRef:
+        key: cloudflare          # 1Password item name
+        property: api-token      # 1Password field name

cluster/platform/cert-manager/values.yaml

@@ -0,0 +1,23 @@
+# ------------------------------------------------------------------------------
+# cert-manager — Helm Values
+# Chart: https://charts.jetstack.io
+# ------------------------------------------------------------------------------
+
+# Install CRDs via Helm (recommended — keeps CRDs in sync with chart version)
+crds:
+  enabled: true
+
+# Replicas — 1 is fine for homelab
+replicaCount: 1
+
+# Use Cloudflare DNS-01 recursive nameservers (same pattern as Traefik/lightning-lane)
+# Only use 1.1.1.1 for DNS-01 challenge resolution — keeps it off internal DNS
+extraArgs:
+  - --dns01-recursive-nameservers-only
+  - --dns01-recursive-nameservers=1.1.1.1:53
+
+# Prometheus metrics for cinderella-castle
+prometheus:
+  enabled: true
+  servicemonitor:
+    enabled: false   # enable once Prometheus operator is in-cluster

cluster/platform/cilium-config/application.yaml

@@ -0,0 +1,48 @@
+# ------------------------------------------------------------------------------
+# Platform: Cilium — Helm upgrade + LB IPAM + Gateway API
+# Wave 4 — no dependencies, Cilium is pre-installed via Talos bootstrap
+#
+# This Application does two things:
+#   1. Manages Cilium itself via Helm (enables Gateway API, keeps config in git)
+#   2. Applies LB pool and L2 announcement manifests
+#
+# On first sync, ArgoCD will upgrade Cilium in-place with --reuse-values
+# plus our additions (gatewayAPI.enabled, l2announcements, etc.).
+# Cilium operator and daemonset will rolling-restart automatically.
+#
+# IP Allocation:
+#   10.1.71.80       — ingress-nginx (reserved, not from pool)
+#   10.1.71.81       — Cilium Gateway (reserved, not from pool)
+#   10.1.71.82-.89   — ingress-nginx LB pool (future additional controllers)
+#   10.1.71.90-.99   — Cilium Gateway LB pool
+# ------------------------------------------------------------------------------
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: cilium-config
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "4"
+spec:
+  project: default
+  sources:
+    # Source 1: Cilium Helm chart — manages Cilium itself
+    - repoURL: https://helm.cilium.io
+      chart: cilium
+      targetRevision: 1.17.4
+      helm:
+        valuesFiles:
+          - $values/cluster/platform/cilium-config/values.yaml
+    # Source 2: Our repo — provides the values file and manifests
+    - repoURL: https://gitea.mk-labs.cloud/rblundon/homelab.git
+      targetRevision: main
+      ref: values
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: kube-system
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - ServerSideApply=true

cluster/platform/cilium-config/manifests/lb-pool.yaml

@@ -0,0 +1,66 @@
+# ------------------------------------------------------------------------------
+# Cilium LB IPAM — IP Address Pools
+#
+# Pool 1: ingress-nginx pool  — 10.1.71.80 - 10.1.71.89
+# Pool 2: cilium-gateway pool — 10.1.71.90 - 10.1.71.99
+#
+# Pools are separated so ingress-nginx and Gateway API services never
+# compete for the same IPs. Each pool uses a serviceSelector to ensure
+# IPs are only assigned to the right controller.
+# ------------------------------------------------------------------------------
+
+# Pool for ingress-nginx LoadBalancer service
+apiVersion: cilium.io/v2alpha1
+kind: CiliumLoadBalancerIPPool
+metadata:
+  name: ingress-nginx-pool
+  namespace: kube-system
+spec:
+  blocks:
+    - start: 10.1.71.80
+      stop: 10.1.71.89
+  serviceSelector:
+    matchLabels:
+      app.kubernetes.io/name: ingress-nginx
+
+---
+# Pool for Cilium Gateway API — only assigned to Gateway-owned services
+# Cilium automatically labels Gateway services with io.cilium.gateway/owning-gateway
+apiVersion: cilium.io/v2alpha1
+kind: CiliumLoadBalancerIPPool
+metadata:
+  name: cilium-gateway-pool
+  namespace: kube-system
+spec:
+  blocks:
+    - start: 10.1.71.90
+      stop: 10.1.71.99
+  serviceSelector:
+    matchExpressions:
+      - key: io.cilium.gateway/owning-gateway
+        operator: Exists
+
+---
+# L2 Announcement Policy — ARP on server VLAN for both pools
+apiVersion: cilium.io/v2alpha1
+kind: CiliumL2AnnouncementPolicy
+metadata:
+  name: fastpass-l2-policy
+  namespace: kube-system
+spec:
+  loadBalancerIPs: true
+  # All nodes participate — Cilium handles failover automatically
+  interfaces:
+    - ^eth[0-9]+
+
+---
+# GatewayClass — tells Kubernetes that Cilium handles Gateway resources
+# This is the entry point for all Gateway API routing
+apiVersion: gateway.networking.k8s.io/v1
+kind: GatewayClass
+metadata:
+  name: cilium
+spec:
+  controllerName: io.cilium/gateway-controller
+  description: Cilium Gateway API implementation (eBPF-native)
+

cluster/platform/cilium-config/values.yaml

@@ -0,0 +1,57 @@
+# ------------------------------------------------------------------------------
+# Cilium Helm Values — fastpass cluster
+# Chart: https://helm.cilium.io
+# Version: 1.17.4
+#
+# These values are applied as a Helm upgrade over the bootstrap install.
+# --reuse-values is implicit via ArgoCD's Helm source — existing values
+# from the Talos bootstrap are preserved; we only override what's listed here.
+#
+# Talos bootstrap values (already set, do not need repeating):
+#   kubeProxyReplacement: true
+#   k8sServiceHost: 10.1.71.65
+#   k8sServicePort: 6443
+#   ipam.mode: kubernetes
+#   tunnel: vxlan
+# ------------------------------------------------------------------------------
+
+# Enable Gateway API support — this is the key addition
+# Cilium operator will register as a GatewayClass controller automatically
+gatewayAPI:
+  enabled: true
+
+# L2 announcements — ARP-based LB IP advertisement on the server VLAN
+l2announcements:
+  enabled: true
+
+# Rate limiting for L2 announcements — important with many services
+# Tune up if you see "rate limit exceeded" in cilium-operator logs
+k8sClientRateLimit:
+  qps: 20
+  burst: 40
+
+# ExternalIPs advertisement
+externalIPs:
+  enabled: true
+
+# Hubble observability — worth enabling for visibility into cluster traffic
+hubble:
+  enabled: true
+  relay:
+    enabled: true
+  ui:
+    enabled: true
+  metrics:
+    enabled:
+      - dns
+      - drop
+      - tcp
+      - flow
+      - port-distribution
+      - httpV2
+
+# Prometheus metrics
+prometheus:
+  enabled: true
+  serviceMonitor:
+    enabled: false  # enable once Prometheus operator is in-cluster

cluster/platform/external-dns/application.yaml

@@ -0,0 +1,30 @@
+# ------------------------------------------------------------------------------
+# Platform: external-dns
+# Wave 6 — depends on ingress-nginx (wave 5) to have an IP to register
+# Configured to sync with Technitium DNS on monorail (10.1.71.32)
+# ------------------------------------------------------------------------------
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: external-dns
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "6"
+spec:
+  project: default
+  source:
+    repoURL: https://kubernetes-sigs.github.io/external-dns
+    chart: external-dns
+    targetRevision: 1.15.1
+    helm:
+      valuesFile: values.yaml
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: external-dns
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+      - ServerSideApply=true

cluster/platform/external-dns/externalsecret.yaml

@@ -0,0 +1,29 @@
+# ------------------------------------------------------------------------------
+# ExternalSecret — Technitium TSIG Secret
+# Pulled from 1Password, materialized in the external-dns namespace.
+#
+# In 1Password, store as item "technitium-tsig" with field "tsig-secret"
+# in the "mk-labs" vault.
+#
+# To generate a TSIG key for Technitium:
+#   tsig-keygen -a hmac-sha256 external-dns
+# Add the key to Technitium under Settings > DNS Settings > TSIG Keys
+# ------------------------------------------------------------------------------
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: technitium-tsig-secret
+  namespace: external-dns
+spec:
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: onepassword-connect
+  refreshInterval: "1h"
+  target:
+    name: technitium-tsig-secret
+    creationPolicy: Owner
+  data:
+    - secretKey: tsig-secret
+      remoteRef:
+        key: technitium-tsig       # 1Password item name
+        property: tsig-secret      # 1Password field name

cluster/platform/external-dns/values.yaml

@@ -0,0 +1,58 @@
+# ------------------------------------------------------------------------------
+# external-dns — Helm Values
+# Chart: https://kubernetes-sigs.github.io/external-dns
+#
+# Provider: Technitium DNS on monorail (10.1.71.32)
+# Technitium uses the RFC2136 (DNS UPDATE) provider in external-dns.
+# Zone: local.mk-labs.cloud
+# ------------------------------------------------------------------------------
+
+provider:
+  name: rfc2136
+
+env:
+  # Technitium API endpoint — RFC2136 compatible
+  - name: EXTERNAL_DNS_RFC2136_HOST
+    value: "10.1.71.32"
+  - name: EXTERNAL_DNS_RFC2136_PORT
+    value: "53"
+  - name: EXTERNAL_DNS_RFC2136_ZONE
+    value: "local.mk-labs.cloud"
+  - name: EXTERNAL_DNS_RFC2136_TSIG_AXFR
+    value: "true"
+  # TSIG secret key name — must match what's configured in Technitium
+  - name: EXTERNAL_DNS_RFC2136_TSIG_SECRET_ALG
+    value: "hmac-sha256"
+  - name: EXTERNAL_DNS_RFC2136_TSIG_KEYNAME
+    value: "external-dns"
+  - name: EXTERNAL_DNS_RFC2136_TSIG_SECRET
+    valueFrom:
+      secretKeyRef:
+        name: technitium-tsig-secret
+        key: tsig-secret
+
+# Only manage records in this zone
+domainFilters:
+  - local.mk-labs.cloud
+
+# Only process Ingress resources (and Services with the right annotation)
+sources:
+  - ingress
+  - service
+
+# Annotation to opt services into DNS management:
+# external-dns.alpha.kubernetes.io/hostname: myservice.local.mk-labs.cloud
+annotationFilter: "external-dns.alpha.kubernetes.io/hostname"
+
+# Log level — info in normal ops, debug when troubleshooting
+logLevel: info
+
+# Interval between sync runs
+interval: "1m"
+
+# Policy: sync = create AND delete records. upsert-only = never delete.
+policy: sync
+
+# Registry to track which records external-dns owns
+registry: txt
+txtOwnerId: fastpass

cluster/platform/external-secrets/application.yaml

@@ -0,0 +1,29 @@
+# ------------------------------------------------------------------------------
+# Platform: External Secrets Operator
+# Wave 1 — must be fully healthy before 1Password Connect (wave 2)
+# ------------------------------------------------------------------------------
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: external-secrets
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "1"
+spec:
+  project: default
+  source:
+    repoURL: https://charts.external-secrets.io
+    chart: external-secrets
+    targetRevision: 0.14.4
+    helm:
+      valuesFile: values.yaml
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: external-secrets
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+      - ServerSideApply=true

cluster/platform/external-secrets/values.yaml

@@ -0,0 +1,23 @@
+# ------------------------------------------------------------------------------
+# External Secrets Operator — Helm Values
+# Chart: https://charts.external-secrets.io
+# ------------------------------------------------------------------------------
+
+# Install CRDs as part of the chart (recommended for GitOps)
+installCRDs: true
+
+# Replica count — 1 is fine for homelab
+replicaCount: 1
+
+# Enable Prometheus metrics for cinderella-castle monitoring
+metrics:
+  service:
+    enabled: true
+
+# Webhook for secret validation
+webhook:
+  replicaCount: 1
+
+# Cert controller manages webhook certs
+certController:
+  replicaCount: 1

cluster/platform/gateway-api/application.yaml

@@ -0,0 +1,40 @@
+# ------------------------------------------------------------------------------
+# Platform: Gateway API CRDs
+# Wave 3.5 — must exist before cilium-config (wave 4) enables gatewayAPI
+#
+# The Gateway API CRDs are maintained separately from any implementation.
+# We install the standard channel CRDs from the upstream sig-network repo.
+# Cilium requires these CRDs to exist before gatewayAPI.enabled=true works.
+#
+# Standard channel includes: GatewayClass, Gateway, HTTPRoute, GRPCRoute,
+#                            ReferenceGrant
+# Experimental channel adds: TCPRoute, TLSRoute, UDPRoute (added below)
+# ------------------------------------------------------------------------------
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: gateway-api-crds
+  namespace: argocd
+  annotations:
+    # Wave 3.5 — after cert-manager (3), before cilium-config (4)
+    # ArgoCD sync waves are integers, so we use 3 here and rely on
+    # sync ordering within the wave. Add a sync-wave of 3 to cert-manager
+    # and this will naturally sequence correctly.
+    argocd.argoproj.io/sync-wave: "3"
+spec:
+  project: default
+  source:
+    repoURL: https://gitea.mk-labs.cloud/rblundon/homelab.git
+    targetRevision: main
+    path: cluster/platform/gateway-api/manifests
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: gateway-api
+  syncPolicy:
+    automated:
+      prune: false   # Never prune CRDs — too dangerous
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+      - ServerSideApply=true
+      - Replace=true   # CRDs require Replace not Patch for updates

cluster/platform/gateway-api/gateway.yaml

@@ -0,0 +1,57 @@
+# ------------------------------------------------------------------------------
+# Gateway — fastpass-gateway
+# Wave 5.5 — after cilium-config (4) and ingress-nginx (5)
+#
+# This is the central Gateway that all HTTPRoute resources attach to.
+# Cilium operator creates a LoadBalancer service for this Gateway automatically,
+# which will receive an IP from the cilium-gateway-pool (10.1.71.90-.99).
+#
+# Listeners:
+#   - HTTP  (port 80)  — redirect to HTTPS or direct for internal services
+#   - HTTPS (port 443) — TLS termination via cert-manager certificates
+#   - TCP   (port configurable) — for non-HTTP services (future use)
+#
+# Usage: Applications create HTTPRoute resources that reference this Gateway:
+#
+#   spec:
+#     parentRefs:
+#       - name: fastpass-gateway
+#         namespace: gateway-system
+#
+# ------------------------------------------------------------------------------
+apiVersion: gateway.networking.k8s.io/v1
+kind: Gateway
+metadata:
+  name: fastpass-gateway
+  namespace: gateway-system
+  annotations:
+    # Request a specific IP from the cilium-gateway-pool
+    io.cilium/lb-ipam-ips: "10.1.71.90"
+spec:
+  gatewayClassName: cilium
+  listeners:
+    # HTTP listener — accepts from all namespaces
+    - name: http
+      protocol: HTTP
+      port: 80
+      allowedRoutes:
+        namespaces:
+          from: All
+
+    # HTTPS listener — TLS termination
+    # Each service provides its own cert via cert-manager Certificate resource
+    # and a ReferenceGrant allowing the Gateway to read it cross-namespace
+    - name: https
+      protocol: HTTPS
+      port: 443
+      allowedRoutes:
+        namespaces:
+          from: All
+      tls:
+        mode: Terminate
+        certificateRefs:
+          # ArgoCD TLS cert — first service on the Gateway
+          # Add additional certs here as services are added, or use
+          # a wildcard cert (*.local.mk-labs.cloud) once the pattern is proven
+          - name: argocd-tls
+            namespace: argocd

cluster/platform/gateway-api/manifests/README.md

@@ -0,0 +1,40 @@
+# ------------------------------------------------------------------------------
+# Gateway API CRDs — Standard + Experimental Channel
+# Version: v1.2.1 (latest stable as of May 2026)
+#
+# We reference upstream CRD manifests directly. ArgoCD will apply them
+# from the upstream GitHub URL via the source config in application.yaml.
+#
+# To upgrade Gateway API CRDs:
+#   1. Update the version tag in the URLs below
+#   2. Commit and push — ArgoCD applies automatically
+#
+# Standard channel CRDs (stable, production-ready):
+#   - GatewayClass
+#   - Gateway
+#   - HTTPRoute
+#   - GRPCRoute
+#   - ReferenceGrant
+#
+# Experimental channel additions (used for TCPRoute for non-HTTP services):
+#   - TLSRoute
+#   - TCPRoute
+#
+# Source: https://github.com/kubernetes-sigs/gateway-api/releases/tag/v1.2.1
+# ------------------------------------------------------------------------------
+
+# NOTE: ArgoCD applies these from the upstream URL. This file documents
+# the versions in use. The actual application.yaml points ArgoCD at this
+# directory, so place the CRD manifests here by running:
+#
+#   curl -L https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml \
+#     -o cluster/platform/gateway-api/manifests/standard-install.yaml
+#
+#   curl -L https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/experimental-install.yaml \
+#     -o cluster/platform/gateway-api/manifests/experimental-install.yaml
+#
+# Commit both files. ArgoCD will apply them on next sync.
+#
+# The commands above should be run from city-hall or your workstation
+# and the resulting files committed to git. We vendor them rather than
+# fetching at runtime so the exact CRD versions are pinned in git.

cluster/platform/gateway-api/namespace.yaml

@@ -0,0 +1,7 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: gateway-system
+  labels:
+    # Allow all namespaces to attach Routes to Gateways in this namespace
+    kubernetes.io/metadata.name: gateway-system

cluster/platform/ingress-nginx/application.yaml

@@ -0,0 +1,29 @@
+# ------------------------------------------------------------------------------
+# Platform: ingress-nginx
+# Wave 5 — depends on Cilium LB pool (wave 4) to get a LoadBalancer IP
+# ------------------------------------------------------------------------------
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: ingress-nginx
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "5"
+spec:
+  project: default
+  source:
+    repoURL: https://kubernetes.github.io/ingress-nginx
+    chart: ingress-nginx
+    targetRevision: 4.12.1
+    helm:
+      valuesFile: values.yaml
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: ingress-nginx
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+      - ServerSideApply=true

cluster/platform/ingress-nginx/values.yaml

@@ -0,0 +1,43 @@
+# ------------------------------------------------------------------------------
+# ingress-nginx — Helm Values
+# Chart: https://kubernetes.github.io/ingress-nginx
+# ------------------------------------------------------------------------------
+
+controller:
+  # Request a specific IP from the Cilium LB pool
+  # 10.1.71.80 — first address in the pool, reserved for ingress
+  service:
+    loadBalancerIP: 10.1.71.80
+    annotations:
+      # Tell Cilium which pool to use (optional if only one pool exists)
+      io.cilium/lb-ipam-ips: "10.1.71.80"
+
+  # IngressClass name — referenced by all Ingress resources
+  ingressClassResource:
+    name: nginx
+    enabled: true
+    default: true
+
+  # Pass real client IPs through to backends
+  config:
+    use-forwarded-headers: "true"
+    compute-full-forwarded-for: "true"
+    use-proxy-protocol: "false"
+
+  # Prometheus metrics
+  metrics:
+    enabled: true
+    serviceMonitor:
+      enabled: false  # enable once Prometheus operator is in-cluster
+
+  # 2 replicas for basic resilience across worker nodes
+  replicaCount: 2
+
+  # Spread across worker nodes
+  topologySpreadConstraints:
+    - maxSkew: 1
+      topologyKey: kubernetes.io/hostname
+      whenUnsatisfiable: DoNotSchedule
+      labelSelector:
+        matchLabels:
+          app.kubernetes.io/name: ingress-nginx

cluster/platform/onepassword-connect/application.yaml

@@ -0,0 +1,41 @@
+# ------------------------------------------------------------------------------
+# Platform: 1Password Connect Server
+# Wave 2 — depends on ESO (wave 1) being healthy
+#
+# IMPORTANT — Bootstrap secret:
+# Before this Application can sync successfully, the 1Password credentials
+# file must be applied manually ONE TIME:
+#
+#   kubectl create namespace onepassword-connect
+#   kubectl create secret generic op-credentials \
+#     --from-file=1password-credentials.json \
+#     -n onepassword-connect
+#
+# This is the ONLY secret that cannot flow through ESO (it IS the ESO backend).
+# Store the credentials file securely in 1Password itself as a backup.
+# ------------------------------------------------------------------------------
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: onepassword-connect
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "2"
+spec:
+  project: default
+  source:
+    repoURL: https://1password.github.io/connect-helm-charts
+    chart: connect
+    targetRevision: 1.16.0
+    helm:
+      valuesFile: values.yaml
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: onepassword-connect
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+      - ServerSideApply=true

cluster/platform/onepassword-connect/cluster-secret-store.yaml

@@ -0,0 +1,32 @@
+# ------------------------------------------------------------------------------
+# ClusterSecretStore — 1Password Connect
+#
+# This is the bridge between ESO and 1Password Connect. Once this exists,
+# any ExternalSecret in any namespace can reference:
+#   secretStoreRef:
+#     kind: ClusterSecretStore
+#     name: onepassword-connect
+#
+# The connect-token secret must exist in the onepassword-connect namespace.
+# Generate a token in 1Password: Developer Tools > Connect Servers > <your server>
+# Then apply once:
+#   kubectl create secret generic connect-token \
+#     --from-literal=token=<your-token> \
+#     -n onepassword-connect
+# ------------------------------------------------------------------------------
+apiVersion: external-secrets.io/v1beta1
+kind: ClusterSecretStore
+metadata:
+  name: onepassword-connect
+spec:
+  provider:
+    onepassword:
+      connectHost: http://onepassword-connect.onepassword-connect.svc.cluster.local:8080
+      vaults:
+        mk-labs: 1          # vault name in 1Password : priority
+      auth:
+        secretRef:
+          connectTokenSecretRef:
+            name: connect-token
+            key: token
+            namespace: onepassword-connect

cluster/platform/onepassword-connect/values.yaml

@@ -0,0 +1,24 @@
+# ------------------------------------------------------------------------------
+# 1Password Connect Server — Helm Values
+# Chart: https://1password.github.io/connect-helm-charts
+# ------------------------------------------------------------------------------
+
+connect:
+  # Reference the manually-bootstrapped credentials secret
+  credentialsName: op-credentials
+
+  # Connect server API token — pulled from the credentials secret
+  # The token is generated in your 1Password account under
+  # Developer Tools > Connect Servers
+  serviceType: ClusterIP
+
+  # Operator manages automatic secret injection (optional, disable if not using)
+  operator:
+    create: false
+
+# ------------------------------------------------------------------------------
+# ClusterSecretStore — tells ESO to use this Connect server as its backend.
+# Applied as a raw manifest alongside the Helm chart via ArgoCD's multi-source
+# (or manually after Connect is healthy — whichever you prefer).
+# See: cluster-secret-store.yaml
+# ------------------------------------------------------------------------------