repo cleanup

cluster/argocd/BOOTSTRAP.md

@@ -9,36 +9,23 @@ This document covers the manual steps required to bootstrap ArgoCD on the
 - IP pools applied (`kubectl get ciliumloadbalancerippool`)
 - `city-hall` has `helm`, `kubectl`, and `talosctl` installed
 - SSH keypair for ArgoCD → Gitea exists at `/home/wed/.ssh/argocd_gitea`
+- 1Password credentials file at `/home/wed/1password-credentials.json`
+- 1Password Connect token at `/home/wed/connect-token` (no trailing newline — use `echo -n`)
 
 ---
 
 ## Step 1 — Create the argocd namespace
 
+Must be created first — the Gitea repo secret targets this namespace before
+Helm creates it.
+
 ```bash
 kubectl apply -f cluster/argocd/namespace.yaml
 ```
 
 ---
 
-## Step 2 — Add Gitea to ArgoCD known hosts
-
-ArgoCD refuses SSH connections to unknown hosts. Pre-populate the known hosts
-configmap before ArgoCD starts:
-
-```bash
-ssh-keyscan -t ed25519 gitea.mk-labs.cloud | \
-  kubectl create configmap argocd-ssh-known-hosts-cm \
-  --namespace argocd \
-  --from-file=ssh_known_hosts=/dev/stdin \
-  --dry-run=client -o yaml | kubectl apply -f -
-```
-
----
-
-## Step 3 — Create the Gitea repo secret
-
-This is the only credential that must be manually created. Once ESO is running,
-secrets will be managed via 1Password. For now, create it directly:
+## Step 2 — Create the Gitea repo secret
 
 ```bash
 kubectl create secret generic gitea-repo \
@@ -58,7 +45,7 @@ kubectl -n argocd get secret gitea-repo
 
 ---
 
-## Step 4 — Install ArgoCD via Helm
+## Step 3 — Install ArgoCD via Helm
 
 ```bash
 helm repo add argo https://argoproj.github.io/argo-helm
@@ -80,6 +67,19 @@ kubectl -n argocd get pods --watch
 
 ---
 
+## Step 4 — Add Gitea to ArgoCD known hosts
+
+The known hosts configmap is created by Helm — patch it after install:
+
+```bash
+ssh-keyscan -t ed25519 gitea.mk-labs.cloud 2>/dev/null | \
+  kubectl patch configmap argocd-ssh-known-hosts-cm -n argocd \
+  --type merge \
+  -p "{\"data\":{\"ssh_known_hosts\": \"$(ssh-keyscan -t ed25519 gitea.mk-labs.cloud 2>/dev/null)\"}}"
+```
+
+---
+
 ## Step 5 — Get the initial admin password
 
 ```bash
@@ -90,7 +90,7 @@ kubectl -n argocd get secret argocd-initial-admin-secret \
 Access the UI via port-forward (ingress-nginx isn't installed yet at this point):
 
 ```bash
-kubectl port-forward svc/argocd-server -n argocd 8080:80
+kubectl port-forward svc/argocd-server -n argocd 8080:80 > /tmp/pf.log 2>&1 &
 ```
 
 Browse to http://localhost:8080 — login with `admin` and the password above.
@@ -127,6 +127,49 @@ Run this periodically until no pending CSRs remain.
 
 ---
 
+## Step 8 — Bootstrap 1Password Connect secrets
+
+Once the `onepassword-connect` namespace exists (created by ArgoCD sync),
+create the two secrets the Connect server requires.
+
+**Important notes:**
+- The credentials file must be stored **base64-encoded** in the secret
+- The token file must have **no trailing newline** — use `echo -n` when creating it
+- Both secrets go in the `onepassword-connect` namespace, not `external-secrets`
+- Secret names must match exactly: `op-credentials` and `connect-token`
+
+```bash
+# Verify the namespace exists before proceeding
+kubectl get namespace onepassword-connect
+
+# Create credentials secret — base64-encode the file contents
+kubectl create secret generic op-credentials \
+  --namespace onepassword-connect \
+  --from-literal=1password-credentials.json=$(base64 -w 0 /home/wed/1password-credentials.json)
+
+# Create token secret — token file must have no trailing newline
+kubectl create secret generic connect-token \
+  --namespace onepassword-connect \
+  --from-file=token=/home/wed/connect-token
+```
+
+Restart the Connect deployment to pick up the secrets:
+
+```bash
+kubectl -n onepassword-connect rollout restart deployment onepassword-connect
+kubectl -n onepassword-connect rollout status deployment onepassword-connect
+```
+
+Verify Connect is healthy:
+
+```bash
+kubectl -n onepassword-connect logs deployment/onepassword-connect -c connect-api 2>&1 | tail -10
+```
+
+You should see only `GET /health` and `GET /heartbeat` 200 responses — no errors.
+
+---
+
 ## Expected platform sync order
 
 Apps sync in wave order via `argocd.argoproj.io/sync-wave` annotations:
@@ -156,7 +199,7 @@ Check that the `gitea-repo` secret exists and the known hosts configmap has
 kubectl -n argocd get application <name> -o yaml | grep -A10 "conditions"
 ```
 
-**self-reference loop (app syncing itself):**
+**Self-reference loop (app syncing itself):**
 Ensure `apps-of-apps.yaml` uses `include: "*/application.yaml"` with
 `recurse: true` — this prevents ArgoCD from picking up its own manifest.
 
@@ -168,6 +211,38 @@ prune on any Application that manages the `argocd` namespace.
 The correct field name is `valueFiles` (plural). `valuesFile` and `valuesFiles`
 are silently ignored — ArgoCD syncs green but your values aren't applied.
 
+**argocd-cm configmap missing:**
+If the UI shows `configmap "argocd-cm" not found`, recreate it:
+```bash
+kubectl create configmap argocd-cm \
+  --namespace argocd \
+  --from-literal=url=https://argocd.local.mk-labs.cloud
+kubectl -n argocd rollout restart deployment argocd-server
+```
+
+**1Password Connect base64 error:**
+`illegal base64 data at input byte 0` means the credentials secret was created
+with raw JSON instead of base64-encoded content. Delete and recreate:
+```bash
+kubectl delete secret op-credentials -n onepassword-connect
+kubectl create secret generic op-credentials \
+  --namespace onepassword-connect \
+  --from-literal=1password-credentials.json=$(base64 -w 0 /home/wed/1password-credentials.json)
+kubectl -n onepassword-connect rollout restart deployment onepassword-connect
+```
+
+**1Password Connect invalid Authorization header:**
+`invalid header field value for "Authorization"` means the token has a trailing
+newline. Recreate the token file and secret:
+```bash
+echo -n "your-token" > /home/wed/connect-token
+kubectl delete secret connect-token -n onepassword-connect
+kubectl create secret generic connect-token \
+  --namespace onepassword-connect \
+  --from-file=token=/home/wed/connect-token
+kubectl -n onepassword-connect rollout restart deployment onepassword-connect
+```
+
 ---
 
 ## Post-bootstrap

cluster/argocd/values.yaml

@@ -33,7 +33,7 @@ server:
     hostname: argocd.local.mk-labs.cloud
     tls: true
     annotations:
-      cert-manager.io/cluster-issuer: letsencrypt-production
+      cert-manager.io/cluster-issuer: letsencrypt-prod
       nginx.ingress.kubernetes.io/ssl-passthrough: "false"
       nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
 
@@ -68,6 +68,17 @@ configs:
       hs.message = "IP pool ready"
       return hs
 
+    # Ignore ESO-injected default fields on ExternalSecret resources.
+    # ESO mutates these fields after sync causing false OutOfSync state.
+    resource.customizations.ignoreDifferences.external-secrets.io_ExternalSecret: |
+      jqPathExpressions:
+        - '.spec.data[]?.remoteRef?.conversionStrategy'
+        - '.spec.data[]?.remoteRef?.decodingStrategy'
+        - '.spec.data[]?.remoteRef?.metadataPolicy'
+        - '.spec.dataFrom[]?.extract?.conversionStrategy'
+        - '.spec.dataFrom[]?.extract?.decodingStrategy'
+        - '.spec.dataFrom[]?.extract?.metadataPolicy'
+  
   # RBAC — admin gets full access, SSO groups mapped after Authentik integration
   rbac:
     policy.default: role:readonly