homelab/cluster/argocd/BOOTSTRAP.md

fastpass — ArgoCD Bootstrap

This document covers the manual steps required to bootstrap ArgoCD on the fastpass cluster. After these steps, all further changes are made via Git.

Prerequisites:

• All 6 Talos nodes Ready (kubectl get nodes)
• Cilium healthy (cilium status --wait)
• IP pools applied (kubectl get ciliumloadbalancerippool)
• city-hall has helm, kubectl, and talosctl installed
• SSH keypair for ArgoCD → Gitea exists at /home/wed/.ssh/argocd_gitea
• 1Password credentials file at /home/wed/1password-credentials.json
• 1Password Connect token at /home/wed/connect-token (no trailing newline — use echo -n)

---

Step 1 — Create the argocd namespace

Must be created first — the Gitea repo secret targets this namespace before Helm creates it.

kubectl apply -f cluster/argocd/namespace.yaml

---

Step 2 — Create the Gitea repo secret

kubectl create secret generic gitea-repo \
  --namespace argocd \
  --from-literal=type=git \
  --from-literal=url=git@gitea.mk-labs.cloud:rblundon/homelab.git \
  --from-file=sshPrivateKey=/home/wed/.ssh/argocd_gitea

kubectl label secret gitea-repo -n argocd \
  argocd.argoproj.io/secret-type=repository

Verify:

kubectl -n argocd get secret gitea-repo

---

Step 3 — Install ArgoCD via Helm

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

helm upgrade --install argocd argo/argo-cd \
  --namespace argocd \
  --create-namespace \
  --version 7.8.23 \
  --values cluster/argocd/values.yaml \
  --wait

Wait for all ArgoCD pods to be ready:

kubectl -n argocd get pods --watch

---

Step 4 — Add Gitea to ArgoCD known hosts

The known hosts configmap is created by Helm — patch it after install:

ssh-keyscan -t ed25519 gitea.mk-labs.cloud 2>/dev/null | \
  kubectl patch configmap argocd-ssh-known-hosts-cm -n argocd \
  --type merge \
  -p "{\"data\":{\"ssh_known_hosts\": \"$(ssh-keyscan -t ed25519 gitea.mk-labs.cloud 2>/dev/null)\"}}"

---

Step 5 — Get the initial admin password

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d; echo

Access the UI via port-forward (ingress-nginx isn't installed yet at this point):

kubectl port-forward svc/argocd-server -n argocd 8080:80 > /tmp/pf.log 2>&1 &

Browse to http://localhost:8080 — login with admin and the password above.

---

Step 6 — Apply the app-of-apps

This is the second and final manual kubectl apply. It hands control to ArgoCD:

kubectl apply -n argocd -f cluster/argocd/apps-of-apps.yaml

From this point, all changes are made via Git. ArgoCD will begin syncing cluster/platform/ and cluster/applications/ automatically.

Watch the sync:

kubectl -n argocd get applications --watch

---

Step 7 — Approve CSRs

After nodes join and kubelet starts requesting certs, approve pending CSRs:

kubectl get csr --no-headers | awk '{print $1}' | xargs kubectl certificate approve

Run this periodically until no pending CSRs remain.

---

Step 8 — Bootstrap 1Password Connect secrets

Once the onepassword-connect namespace exists (created by ArgoCD sync), create the two secrets the Connect server requires.

Important notes:

• The credentials file must be stored base64-encoded in the secret
• The token file must have no trailing newline — use echo -n when creating it
• Both secrets go in the onepassword-connect namespace, not external-secrets
• Secret names must match exactly: op-credentials and connect-token

# Verify the namespace exists before proceeding
kubectl get namespace onepassword-connect

# Create credentials secret — base64-encode the file contents
kubectl create secret generic op-credentials \
  --namespace onepassword-connect \
  --from-literal=1password-credentials.json=$(base64 -w 0 /home/wed/1password-credentials.json)

# Create token secret — token file must have no trailing newline
kubectl create secret generic connect-token \
  --namespace onepassword-connect \
  --from-file=token=/home/wed/connect-token

Restart the Connect deployment to pick up the secrets:

kubectl -n onepassword-connect rollout restart deployment onepassword-connect
kubectl -n onepassword-connect rollout status deployment onepassword-connect

Verify Connect is healthy:

kubectl -n onepassword-connect logs deployment/onepassword-connect -c connect-api 2>&1 | tail -10

You should see only GET /health and GET /heartbeat 200 responses — no errors.

---

Expected platform sync order

Apps sync in wave order via argocd.argoproj.io/sync-wave annotations:

Wave	App
0	apps-of-apps (root)
1	cert-manager CRDs
2	cert-manager, external-secrets
3	ingress-nginx
4	external-dns
5+	application workloads

---

Troubleshooting

ArgoCD can't reach Gitea:

kubectl -n argocd logs deployment/argocd-repo-server | grep -i "error\|ssh"

Check that the gitea-repo secret exists and the known hosts configmap has gitea.mk-labs.cloud.

App stuck in Unknown state:

kubectl -n argocd get application <name> -o yaml | grep -A10 "conditions"

Self-reference loop (app syncing itself): Ensure apps-of-apps.yaml uses include: "*/application.yaml" with recurse: true — this prevents ArgoCD from picking up its own manifest.

ArgoCD pruning itself: The argocd namespace Applications must have prune: false. Never enable prune on any Application that manages the argocd namespace.

Helm valueFiles field: The correct field name is valueFiles (plural). valuesFile and valuesFiles are silently ignored — ArgoCD syncs green but your values aren't applied.

argocd-cm configmap missing: If the UI shows configmap "argocd-cm" not found, recreate it:

kubectl create configmap argocd-cm \
  --namespace argocd \
  --from-literal=url=https://argocd.local.mk-labs.cloud
kubectl -n argocd rollout restart deployment argocd-server

1Password Connect base64 error: illegal base64 data at input byte 0 means the credentials secret was created with raw JSON instead of base64-encoded content. Delete and recreate:

kubectl delete secret op-credentials -n onepassword-connect
kubectl create secret generic op-credentials \
  --namespace onepassword-connect \
  --from-literal=1password-credentials.json=$(base64 -w 0 /home/wed/1password-credentials.json)
kubectl -n onepassword-connect rollout restart deployment onepassword-connect

1Password Connect invalid Authorization header: invalid header field value for "Authorization" means the token has a trailing newline. Recreate the token file and secret:

echo -n "your-token" > /home/wed/connect-token
kubectl delete secret connect-token -n onepassword-connect
kubectl create secret generic connect-token \
  --namespace onepassword-connect \
  --from-file=token=/home/wed/connect-token
kubectl -n onepassword-connect rollout restart deployment onepassword-connect

---

Post-bootstrap

Once ingress-nginx and cert-manager are synced:

• ArgoCD UI available at https://argocd.local.mk-labs.cloud
• Change the admin password: argocd account update-password
• Authentik OIDC: uncomment the oidc.config block in values.yaml once the ArgoCD provider is configured in Authentik
• Migrate gitea-repo secret to ESO once 1Password Connect is running