rblundon/homelab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
4ed64ab91cda2f95b0001f83b870be3f247ad336
homelab/cluster/bootstrap
History

README.md

fastpass — Cluster Bootstrap

This directory contains resources and documentation for the manual bootstrap sequence required to bring up fastpass from a fresh Talos install.

Everything in this directory is applied once by hand before ArgoCD takes over. After Step 6 (app-of-apps), all further changes are made via Git.

Prerequisites

  • All 6 Talos nodes Ready (kubectl get nodes)
  • Cilium healthy (cilium status --wait)
  • IP pools applied (kubectl get ciliumloadbalancerippool)
  • city-hall has helm, kubectl, and talosctl installed
  • SSH keypair for ArgoCD → Gitea exists at /home/wed/.ssh/argocd_gitea
  • 1Password credentials file at /home/wed/1password-credentials.json
  • 1Password Connect token at /home/wed/connect-token (no trailing newline)

Step 1 — Install Gateway API CRDs

The Gateway API CRDs are not bundled with Cilium — they must be installed independently before Cilium can register the GatewayClass controller.

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml

Verify:

kubectl get crd gateways.gateway.networking.k8s.io

Step 2 — Create the Cilium GatewayClass

The Cilium operator manages this object but does not auto-create it.

kubectl apply -f cluster/bootstrap/gatewayclass.yaml

Verify (may take a few seconds):

kubectl get gatewayclass
# Expected: cilium   io.cilium/gateway-controller   True

Step 3 — Create the argocd namespace

Must be created before Helm — the Gitea repo secret targets this namespace.

kubectl apply -f cluster/argocd/namespace.yaml

Step 4 — Create the Gitea repo secret

kubectl create secret generic gitea-repo \
  --namespace argocd \
  --from-literal=type=git \
  --from-literal=url=git@gitea.mk-labs.cloud:rblundon/homelab.git \
  --from-file=sshPrivateKey=/home/wed/.ssh/argocd_gitea

kubectl label secret gitea-repo -n argocd \
  argocd.argoproj.io/secret-type=repository

Verify:

kubectl -n argocd get secret gitea-repo

Step 5 — Install ArgoCD via Helm

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

helm upgrade --install argocd argo/argo-cd \
  --namespace argocd \
  --create-namespace \
  --version 7.8.23 \
  --values cluster/argocd/values.yaml \
  --wait

Wait for all ArgoCD pods to be ready:

kubectl -n argocd get pods --watch

Step 6 — Add Gitea to ArgoCD known hosts

ssh-keyscan -t ed25519 gitea.mk-labs.cloud 2>/dev/null | \
  kubectl patch configmap argocd-ssh-known-hosts-cm -n argocd \
  --type merge \
  -p "{\"data\":{\"ssh_known_hosts\": \"$(ssh-keyscan -t ed25519 gitea.mk-labs.cloud 2>/dev/null)\"}}"

Step 7 — Get the initial admin password

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d; echo

Access the UI via port-forward (ingress not yet deployed):

kubectl port-forward svc/argocd-server -n argocd 8080:80 > /tmp/pf.log 2>&1 &

Browse to http://localhost:8080

Step 8 — Apply the app-of-apps

This is the final manual kubectl apply. It hands control to ArgoCD:

kubectl apply -n argocd -f cluster/argocd/apps-of-apps.yaml

From this point, all changes are made via Git. ArgoCD syncs cluster/platform/ and cluster/applications/ automatically.

Watch the sync:

kubectl -n argocd get applications --watch

Step 9 — Approve CSRs

kubectl get csr --no-headers | awk '{print $1}' | xargs kubectl certificate approve

Run periodically until no pending CSRs remain.

Step 10 — Bootstrap 1Password Connect secrets

Once onepassword-connect namespace exists (created by ArgoCD sync):

# Verify namespace exists
kubectl get namespace onepassword-connect

# Credentials secret — must be base64-encoded
kubectl create secret generic op-credentials \
  --namespace onepassword-connect \
  --from-literal=1password-credentials.json=$(base64 -w 0 /home/wed/1password-credentials.json)

# Token secret — file must have no trailing newline
kubectl create secret generic connect-token \
  --namespace onepassword-connect \
  --from-file=token=/home/wed/connect-token

Restart Connect to pick up secrets:

kubectl -n onepassword-connect rollout restart deployment onepassword-connect
kubectl -n onepassword-connect rollout status deployment onepassword-connect

Verify (should show only GET /health and GET /heartbeat 200s):

kubectl -n onepassword-connect logs deployment/onepassword-connect -c connect-api 2>&1 | tail -10

Expected platform sync order

Wave App
0 apps-of-apps (root)
1 external-secrets
1 onepassword-connect
2 nfs-csi
3 cert-manager
4 ingress-nginx
5 external-dns

Troubleshooting

GatewayClass not ACCEPTED after applying: Restart the Cilium operator — it may have started before the CRDs were present:

kubectl rollout restart deployment/cilium-operator -n kube-system
kubectl get gatewayclass

ArgoCD can't reach Gitea:

kubectl -n argocd logs deployment/argocd-repo-server | grep -i "error\|ssh"

Check that gitea-repo secret exists and known hosts configmap has gitea.mk-labs.cloud.

App stuck in Unknown state:

kubectl -n argocd get application <name> -o yaml | grep -A10 "conditions"

ArgoCD pruning itself: The argocd namespace Applications must have prune: false. Never enable prune on any Application that manages the argocd namespace.

Helm valueFiles field: The correct field name is valueFiles (plural). valuesFile and valuesFiles are silently ignored — ArgoCD syncs green but values aren't applied.

1Password Connect base64 error: illegal base64 data at input byte 0 means credentials were stored as raw JSON. Delete and recreate:

kubectl delete secret op-credentials -n onepassword-connect
kubectl create secret generic op-credentials \
  --namespace onepassword-connect \
  --from-literal=1password-credentials.json=$(base64 -w 0 /home/wed/1password-credentials.json)
kubectl -n onepassword-connect rollout restart deployment onepassword-connect

1Password Connect invalid Authorization header: Token has a trailing newline. Recreate:

echo -n "your-token" > /home/wed/connect-token
kubectl delete secret connect-token -n onepassword-connect
kubectl create secret generic connect-token \
  --namespace onepassword-connect \
  --from-file=token=/home/wed/connect-token
kubectl -n onepassword-connect rollout restart deployment onepassword-connect

Post-bootstrap

Once ingress-nginx and cert-manager are synced:

  • ArgoCD UI: https://argocd.local.mk-labs.cloud
  • Change admin password: argocd account update-password
  • Authentik OIDC: uncomment oidc.config in cluster/argocd/values.yaml once the ArgoCD provider is configured in Authentik
  • Migrate gitea-repo secret to ESO once 1Password Connect is healthy

Operational Runbook — Adding a New Application

When deploying a new application that needs HTTPS access via the Cilium Gateway, follow this checklist:

1. Add the namespace to the ReferenceGrant

Edit cluster/platform/gateway/referencegrant.yaml and add a new from entry:

    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: <your-app-namespace>

This is required by the Gateway API spec — without it, the HTTPRoute will fail to attach to the Gateway and traffic won't route. The ReferenceGrant must live in the gateway namespace, so this file is the one place to update.

2. Create the application directory

cluster/applications/<app-name>/
├── application.yaml    ← ArgoCD Application, wave 10+
├── values.yaml         ← Helm values (if Helm-based)
├── externalsecret.yaml ← ESO resource if secrets needed
└── httproute.yaml      ← Certificate + HTTPRoute for Gateway access

3. HTTPRoute pattern

Always include all fields explicitly — Cilium defaults group, kind, weight, and matches at reconcile time, causing ArgoCD out-of-sync if omitted.

Full HTTPRoute template:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: <service>
  namespace: <app-namespace>
  annotations:
    external-dns.alpha.kubernetes.io/hostname: <service>.local.mk-labs.cloud
    external-dns.alpha.kubernetes.io/target: "10.1.71.90"
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: fastpass-gateway
      namespace: gateway
      sectionName: https
  hostnames:
    - <service>.local.mk-labs.cloud
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - group: ""
          kind: Service
          name: <service>
          port: <port>
          weight: 1

4. Per-service certificate

Each service gets its own cert via cert-manager. Add to httproute.yaml:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: <service>-tls
  namespace: <app-namespace>
spec:
  secretName: <service>-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - <service>.local.mk-labs.cloud

5. Push and verify

# Watch ArgoCD sync
kubectl -n argocd get applications --watch

# Verify cert issued
kubectl get certificate -n <app-namespace> -w

# Verify DNS record created
nslookup <service>.local.mk-labs.cloud 10.1.71.32

# Test
curl https://<service>.local.mk-labs.cloud
Reference in New Issue View Git Blame Copy Permalink