256 lines
7.4 KiB
Markdown
256 lines
7.4 KiB
Markdown
# fastpass — ArgoCD Bootstrap
|
|
|
|
This document covers the manual steps required to bootstrap ArgoCD on the
|
|
`fastpass` cluster. After these steps, all further changes are made via Git.
|
|
|
|
**Prerequisites:**
|
|
- All 6 Talos nodes `Ready` (`kubectl get nodes`)
|
|
- Cilium healthy (`cilium status --wait`)
|
|
- IP pools applied (`kubectl get ciliumloadbalancerippool`)
|
|
- `city-hall` has `helm`, `kubectl`, and `talosctl` installed
|
|
- SSH keypair for ArgoCD → Gitea exists at `/home/wed/.ssh/argocd_gitea`
|
|
- 1Password credentials file at `/home/wed/1password-credentials.json`
|
|
- 1Password Connect token at `/home/wed/connect-token` (no trailing newline — use `echo -n`)
|
|
|
|
---
|
|
|
|
## Step 1 — Create the argocd namespace
|
|
|
|
Must be created first — the Gitea repo secret targets this namespace before
|
|
Helm creates it.
|
|
|
|
```bash
|
|
kubectl apply -f cluster/argocd/namespace.yaml
|
|
```
|
|
|
|
---
|
|
|
|
## Step 2 — Create the Gitea repo secret
|
|
|
|
```bash
|
|
kubectl create secret generic gitea-repo \
|
|
--namespace argocd \
|
|
--from-literal=type=git \
|
|
--from-literal=url=git@gitea.mk-labs.cloud:rblundon/homelab.git \
|
|
--from-file=sshPrivateKey=/home/wed/.ssh/argocd_gitea
|
|
|
|
kubectl label secret gitea-repo -n argocd \
|
|
argocd.argoproj.io/secret-type=repository
|
|
```
|
|
|
|
Verify:
|
|
```bash
|
|
kubectl -n argocd get secret gitea-repo
|
|
```
|
|
|
|
---
|
|
|
|
## Step 3 — Install ArgoCD via Helm
|
|
|
|
```bash
|
|
helm repo add argo https://argoproj.github.io/argo-helm
|
|
helm repo update
|
|
|
|
helm upgrade --install argocd argo/argo-cd \
|
|
--namespace argocd \
|
|
--create-namespace \
|
|
--version 7.8.23 \
|
|
--values cluster/argocd/values.yaml \
|
|
--wait
|
|
```
|
|
|
|
Wait for all ArgoCD pods to be ready:
|
|
|
|
```bash
|
|
kubectl -n argocd get pods --watch
|
|
```
|
|
|
|
---
|
|
|
|
## Step 4 — Add Gitea to ArgoCD known hosts
|
|
|
|
The known hosts configmap is created by Helm — patch it after install:
|
|
|
|
```bash
|
|
ssh-keyscan -t ed25519 gitea.mk-labs.cloud 2>/dev/null | \
|
|
kubectl patch configmap argocd-ssh-known-hosts-cm -n argocd \
|
|
--type merge \
|
|
-p "{\"data\":{\"ssh_known_hosts\": \"$(ssh-keyscan -t ed25519 gitea.mk-labs.cloud 2>/dev/null)\"}}"
|
|
```
|
|
|
|
---
|
|
|
|
## Step 5 — Get the initial admin password
|
|
|
|
```bash
|
|
kubectl -n argocd get secret argocd-initial-admin-secret \
|
|
-o jsonpath='{.data.password}' | base64 -d; echo
|
|
```
|
|
|
|
Access the UI via port-forward (ingress-nginx isn't installed yet at this point):
|
|
|
|
```bash
|
|
kubectl port-forward svc/argocd-server -n argocd 8080:80 > /tmp/pf.log 2>&1 &
|
|
```
|
|
|
|
Browse to http://localhost:8080 — login with `admin` and the password above.
|
|
|
|
---
|
|
|
|
## Step 6 — Apply the app-of-apps
|
|
|
|
This is the second and final manual `kubectl apply`. It hands control to ArgoCD:
|
|
|
|
```bash
|
|
kubectl apply -n argocd -f cluster/argocd/apps-of-apps.yaml
|
|
```
|
|
|
|
From this point, all changes are made via Git. ArgoCD will begin syncing
|
|
`cluster/platform/` and `cluster/applications/` automatically.
|
|
|
|
Watch the sync:
|
|
```bash
|
|
kubectl -n argocd get applications --watch
|
|
```
|
|
|
|
---
|
|
|
|
## Step 7 — Approve CSRs
|
|
|
|
After nodes join and kubelet starts requesting certs, approve pending CSRs:
|
|
|
|
```bash
|
|
kubectl get csr --no-headers | awk '{print $1}' | xargs kubectl certificate approve
|
|
```
|
|
|
|
Run this periodically until no pending CSRs remain.
|
|
|
|
---
|
|
|
|
## Step 8 — Bootstrap 1Password Connect secrets
|
|
|
|
Once the `onepassword-connect` namespace exists (created by ArgoCD sync),
|
|
create the two secrets the Connect server requires.
|
|
|
|
**Important notes:**
|
|
- The credentials file must be stored **base64-encoded** in the secret
|
|
- The token file must have **no trailing newline** — use `echo -n` when creating it
|
|
- Both secrets go in the `onepassword-connect` namespace, not `external-secrets`
|
|
- Secret names must match exactly: `op-credentials` and `connect-token`
|
|
|
|
```bash
|
|
# Verify the namespace exists before proceeding
|
|
kubectl get namespace onepassword-connect
|
|
|
|
# Create credentials secret — base64-encode the file contents
|
|
kubectl create secret generic op-credentials \
|
|
--namespace onepassword-connect \
|
|
--from-literal=1password-credentials.json=$(base64 -w 0 /home/wed/1password-credentials.json)
|
|
|
|
# Create token secret — token file must have no trailing newline
|
|
kubectl create secret generic connect-token \
|
|
--namespace onepassword-connect \
|
|
--from-file=token=/home/wed/connect-token
|
|
```
|
|
|
|
Restart the Connect deployment to pick up the secrets:
|
|
|
|
```bash
|
|
kubectl -n onepassword-connect rollout restart deployment onepassword-connect
|
|
kubectl -n onepassword-connect rollout status deployment onepassword-connect
|
|
```
|
|
|
|
Verify Connect is healthy:
|
|
|
|
```bash
|
|
kubectl -n onepassword-connect logs deployment/onepassword-connect -c connect-api 2>&1 | tail -10
|
|
```
|
|
|
|
You should see only `GET /health` and `GET /heartbeat` 200 responses — no errors.
|
|
|
|
---
|
|
|
|
## Expected platform sync order
|
|
|
|
Apps sync in wave order via `argocd.argoproj.io/sync-wave` annotations:
|
|
|
|
| Wave | App |
|
|
|------|-----|
|
|
| 0 | apps-of-apps (root) |
|
|
| 1 | cert-manager CRDs |
|
|
| 2 | cert-manager, external-secrets |
|
|
| 3 | ingress-nginx |
|
|
| 4 | external-dns |
|
|
| 5+ | application workloads |
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
**ArgoCD can't reach Gitea:**
|
|
```bash
|
|
kubectl -n argocd logs deployment/argocd-repo-server | grep -i "error\|ssh"
|
|
```
|
|
Check that the `gitea-repo` secret exists and the known hosts configmap has
|
|
`gitea.mk-labs.cloud`.
|
|
|
|
**App stuck in `Unknown` state:**
|
|
```bash
|
|
kubectl -n argocd get application <name> -o yaml | grep -A10 "conditions"
|
|
```
|
|
|
|
**Self-reference loop (app syncing itself):**
|
|
Ensure `apps-of-apps.yaml` uses `include: "*/application.yaml"` with
|
|
`recurse: true` — this prevents ArgoCD from picking up its own manifest.
|
|
|
|
**ArgoCD pruning itself:**
|
|
The `argocd` namespace Applications must have `prune: false`. Never enable
|
|
prune on any Application that manages the `argocd` namespace.
|
|
|
|
**Helm valueFiles field:**
|
|
The correct field name is `valueFiles` (plural). `valuesFile` and `valuesFiles`
|
|
are silently ignored — ArgoCD syncs green but your values aren't applied.
|
|
|
|
**argocd-cm configmap missing:**
|
|
If the UI shows `configmap "argocd-cm" not found`, recreate it:
|
|
```bash
|
|
kubectl create configmap argocd-cm \
|
|
--namespace argocd \
|
|
--from-literal=url=https://argocd.local.mk-labs.cloud
|
|
kubectl -n argocd rollout restart deployment argocd-server
|
|
```
|
|
|
|
**1Password Connect base64 error:**
|
|
`illegal base64 data at input byte 0` means the credentials secret was created
|
|
with raw JSON instead of base64-encoded content. Delete and recreate:
|
|
```bash
|
|
kubectl delete secret op-credentials -n onepassword-connect
|
|
kubectl create secret generic op-credentials \
|
|
--namespace onepassword-connect \
|
|
--from-literal=1password-credentials.json=$(base64 -w 0 /home/wed/1password-credentials.json)
|
|
kubectl -n onepassword-connect rollout restart deployment onepassword-connect
|
|
```
|
|
|
|
**1Password Connect invalid Authorization header:**
|
|
`invalid header field value for "Authorization"` means the token has a trailing
|
|
newline. Recreate the token file and secret:
|
|
```bash
|
|
echo -n "your-token" > /home/wed/connect-token
|
|
kubectl delete secret connect-token -n onepassword-connect
|
|
kubectl create secret generic connect-token \
|
|
--namespace onepassword-connect \
|
|
--from-file=token=/home/wed/connect-token
|
|
kubectl -n onepassword-connect rollout restart deployment onepassword-connect
|
|
```
|
|
|
|
---
|
|
|
|
## Post-bootstrap
|
|
|
|
Once ingress-nginx and cert-manager are synced:
|
|
- ArgoCD UI available at https://argocd.local.mk-labs.cloud
|
|
- Change the admin password: `argocd account update-password`
|
|
- Authentik OIDC: uncomment the `oidc.config` block in `values.yaml` once
|
|
the ArgoCD provider is configured in Authentik
|
|
- Migrate `gitea-repo` secret to ESO once 1Password Connect is running
|