cilium, argo
This commit is contained in:
@@ -1,136 +1,180 @@
|
||||
# fastpass GitOps Bootstrap
|
||||
# fastpass — ArgoCD Bootstrap
|
||||
|
||||
This document is the complete procedure to go from a freshly provisioned Talos
|
||||
cluster to a fully GitOps-managed platform. After completing these steps, all
|
||||
future changes are made via git — never `kubectl apply` again.
|
||||
This document covers the manual steps required to bootstrap ArgoCD on the
|
||||
`fastpass` cluster. After these steps, all further changes are made via Git.
|
||||
|
||||
## Prerequisites
|
||||
**Prerequisites:**
|
||||
- All 6 Talos nodes `Ready` (`kubectl get nodes`)
|
||||
- Cilium healthy (`cilium status --wait`)
|
||||
- IP pools applied (`kubectl get ciliumloadbalancerippool`)
|
||||
- `city-hall` has `helm`, `kubectl`, and `talosctl` installed
|
||||
- SSH keypair for ArgoCD → Gitea exists at `/home/wed/.ssh/argocd_gitea`
|
||||
|
||||
- Talos cluster running (`space-mountain`, `big-thunder-mountain`, `splash-mountain` CP nodes)
|
||||
- Cilium installed as CNI (done at Talos bootstrap via talhelper)
|
||||
- `kubectl` configured and pointing at `fastpass`
|
||||
- `kubeconfig` on `city-hall` at `~/.kube/config`
|
||||
- 1Password Connect credentials JSON downloaded from your 1Password account
|
||||
(Developer Tools > Connect Servers > New Server)
|
||||
- 1Password Connect token generated for the `mk-labs` vault
|
||||
---
|
||||
|
||||
## Step 1 — Install ArgoCD
|
||||
|
||||
This is the only `kubectl apply` you will ever run against this cluster.
|
||||
## Step 1 — Create the argocd namespace
|
||||
|
||||
```bash
|
||||
kubectl create namespace argocd
|
||||
|
||||
kubectl apply -n argocd \
|
||||
--server-side \
|
||||
--force-conflicts \
|
||||
-f https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.2/manifests/install.yaml
|
||||
|
||||
# Wait for ArgoCD to be healthy
|
||||
kubectl wait --for=condition=available deployment/argocd-server \
|
||||
-n argocd --timeout=120s
|
||||
kubectl apply -f cluster/argocd/namespace.yaml
|
||||
```
|
||||
|
||||
## Step 2 — Bootstrap the 1Password Connect secret
|
||||
---
|
||||
|
||||
This is the only secret you will ever manage manually. Everything else flows
|
||||
through ESO from here.
|
||||
## Step 2 — Add Gitea to ArgoCD known hosts
|
||||
|
||||
ArgoCD refuses SSH connections to unknown hosts. Pre-populate the known hosts
|
||||
configmap before ArgoCD starts:
|
||||
|
||||
```bash
|
||||
# Create the namespace first
|
||||
kubectl create namespace onepassword-connect
|
||||
|
||||
# Apply the credentials file (downloaded from 1Password)
|
||||
kubectl create secret generic op-credentials \
|
||||
--from-file=1password-credentials.json=/path/to/1password-credentials.json \
|
||||
-n onepassword-connect
|
||||
|
||||
# Apply the Connect API token
|
||||
kubectl create secret generic connect-token \
|
||||
--from-literal=token=<your-connect-token> \
|
||||
-n onepassword-connect
|
||||
ssh-keyscan -t ed25519 gitea.mk-labs.cloud | \
|
||||
kubectl create configmap argocd-ssh-known-hosts-cm \
|
||||
--namespace argocd \
|
||||
--from-file=ssh_known_hosts=/dev/stdin \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
```
|
||||
|
||||
## Step 3 — Apply the App of Apps
|
||||
---
|
||||
|
||||
This is the second and final manual apply. From this point on, ArgoCD manages
|
||||
everything, including itself.
|
||||
## Step 3 — Create the Gitea repo secret
|
||||
|
||||
This is the only credential that must be manually created. Once ESO is running,
|
||||
secrets will be managed via 1Password. For now, create it directly:
|
||||
|
||||
```bash
|
||||
kubectl create secret generic gitea-repo \
|
||||
--namespace argocd \
|
||||
--from-literal=type=git \
|
||||
--from-literal=url=git@gitea.mk-labs.cloud:rblundon/homelab.git \
|
||||
--from-file=sshPrivateKey=/home/wed/.ssh/argocd_gitea
|
||||
|
||||
kubectl label secret gitea-repo -n argocd \
|
||||
argocd.argoproj.io/secret-type=repository
|
||||
```
|
||||
|
||||
Verify:
|
||||
```bash
|
||||
kubectl -n argocd get secret gitea-repo
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — Install ArgoCD via Helm
|
||||
|
||||
```bash
|
||||
helm repo add argo https://argoproj.github.io/argo-helm
|
||||
helm repo update
|
||||
|
||||
helm upgrade --install argocd argo/argo-cd \
|
||||
--namespace argocd \
|
||||
--create-namespace \
|
||||
--version 7.8.23 \
|
||||
--values cluster/argocd/values.yaml \
|
||||
--wait
|
||||
```
|
||||
|
||||
Wait for all ArgoCD pods to be ready:
|
||||
|
||||
```bash
|
||||
kubectl -n argocd get pods --watch
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5 — Get the initial admin password
|
||||
|
||||
```bash
|
||||
kubectl -n argocd get secret argocd-initial-admin-secret \
|
||||
-o jsonpath='{.data.password}' | base64 -d; echo
|
||||
```
|
||||
|
||||
Access the UI via port-forward (ingress-nginx isn't installed yet at this point):
|
||||
|
||||
```bash
|
||||
kubectl port-forward svc/argocd-server -n argocd 8080:80
|
||||
```
|
||||
|
||||
Browse to http://localhost:8080 — login with `admin` and the password above.
|
||||
|
||||
---
|
||||
|
||||
## Step 6 — Apply the app-of-apps
|
||||
|
||||
This is the second and final manual `kubectl apply`. It hands control to ArgoCD:
|
||||
|
||||
```bash
|
||||
kubectl apply -n argocd -f cluster/argocd/apps-of-apps.yaml
|
||||
```
|
||||
|
||||
## Step 4 — Watch it go
|
||||
From this point, all changes are made via Git. ArgoCD will begin syncing
|
||||
`cluster/platform/` and `cluster/applications/` automatically.
|
||||
|
||||
Watch the sync:
|
||||
```bash
|
||||
# Watch all Applications come up in wave order
|
||||
kubectl get applications -n argocd -w
|
||||
|
||||
# Or use the ArgoCD UI
|
||||
kubectl port-forward svc/argocd-server -n argocd 8080:443
|
||||
# Then open https://localhost:8080
|
||||
# Default admin password:
|
||||
kubectl -n argocd get secret argocd-initial-admin-secret \
|
||||
-o jsonpath="{.data.password}" | base64 -d
|
||||
kubectl -n argocd get applications --watch
|
||||
```
|
||||
|
||||
## Step 2.5 — Vendor Gateway API CRDs
|
||||
---
|
||||
|
||||
Before applying the App of Apps, vendor the Gateway API CRDs into git.
|
||||
Run from the repo root on city-hall:
|
||||
## Step 7 — Approve CSRs
|
||||
|
||||
After nodes join and kubelet starts requesting certs, approve pending CSRs:
|
||||
|
||||
```bash
|
||||
curl -L https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml \
|
||||
-o cluster/platform/gateway-api/manifests/standard-install.yaml
|
||||
|
||||
curl -L https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/experimental-install.yaml \
|
||||
-o cluster/platform/gateway-api/manifests/experimental-install.yaml
|
||||
|
||||
git add cluster/platform/gateway-api/manifests/
|
||||
git commit -m "feat(platform): vendor gateway-api CRDs v1.2.1"
|
||||
git push
|
||||
kubectl get csr --no-headers | awk '{print $1}' | xargs kubectl certificate approve
|
||||
```
|
||||
|
||||
## Wave Order
|
||||
Run this periodically until no pending CSRs remain.
|
||||
|
||||
| Wave | Application | Depends On |
|
||||
|------|---------------------|---------------------------------|
|
||||
| 1 | external-secrets | (none) |
|
||||
| 2 | onepassword-connect | external-secrets |
|
||||
| 3 | cert-manager | onepassword-connect (CF secret) |
|
||||
| 3 | gateway-api-crds | (none — CRDs only) |
|
||||
| 4 | cilium-config | gateway-api-crds (CRDs exist) |
|
||||
| 5 | ingress-nginx | cilium-config (LB pool) |
|
||||
| 5 | gateway (resource) | cilium-config (GatewayClass) |
|
||||
| 6 | external-dns | ingress-nginx (IP to register) |
|
||||
| 10+ | applications/* | platform complete |
|
||||
---
|
||||
|
||||
## Upgrading ArgoCD
|
||||
## Expected platform sync order
|
||||
|
||||
```bash
|
||||
kubectl apply -n argocd \
|
||||
--server-side \
|
||||
--force-conflicts \
|
||||
-f https://raw.githubusercontent.com/argoproj/argo-cd/v<NEW_VERSION>/manifests/install.yaml
|
||||
```
|
||||
Apps sync in wave order via `argocd.argoproj.io/sync-wave` annotations:
|
||||
|
||||
Update the version comment in `cluster/argocd/install.yaml` to match.
|
||||
| Wave | App |
|
||||
|------|-----|
|
||||
| 0 | apps-of-apps (root) |
|
||||
| 1 | cert-manager CRDs |
|
||||
| 2 | cert-manager, external-secrets |
|
||||
| 3 | ingress-nginx |
|
||||
| 4 | external-dns |
|
||||
| 5+ | application workloads |
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**ArgoCD can't reach Gitea:**
|
||||
```bash
|
||||
# Check sync status of all apps
|
||||
kubectl get applications -n argocd
|
||||
|
||||
# Check why an app is out of sync
|
||||
kubectl describe application <name> -n argocd
|
||||
|
||||
# Force a sync
|
||||
kubectl patch application <name> -n argocd \
|
||||
--type merge \
|
||||
-p '{"operation": {"initiatedBy": {"username": "admin"}, "sync": {}}}'
|
||||
|
||||
# Check ESO secret sync
|
||||
kubectl get externalsecret -A
|
||||
kubectl describe externalsecret <name> -n <namespace>
|
||||
kubectl -n argocd logs deployment/argocd-repo-server | grep -i "error\|ssh"
|
||||
```
|
||||
Check that the `gitea-repo` secret exists and the known hosts configmap has
|
||||
`gitea.mk-labs.cloud`.
|
||||
|
||||
**App stuck in `Unknown` state:**
|
||||
```bash
|
||||
kubectl -n argocd get application <name> -o yaml | grep -A10 "conditions"
|
||||
```
|
||||
|
||||
**self-reference loop (app syncing itself):**
|
||||
Ensure `apps-of-apps.yaml` uses `include: "*/application.yaml"` with
|
||||
`recurse: true` — this prevents ArgoCD from picking up its own manifest.
|
||||
|
||||
**ArgoCD pruning itself:**
|
||||
The `argocd` namespace Applications must have `prune: false`. Never enable
|
||||
prune on any Application that manages the `argocd` namespace.
|
||||
|
||||
**Helm valueFiles field:**
|
||||
The correct field name is `valueFiles` (plural). `valuesFile` and `valuesFiles`
|
||||
are silently ignored — ArgoCD syncs green but your values aren't applied.
|
||||
|
||||
---
|
||||
|
||||
## Post-bootstrap
|
||||
|
||||
Once ingress-nginx and cert-manager are synced:
|
||||
- ArgoCD UI available at https://argocd.local.mk-labs.cloud
|
||||
- Change the admin password: `argocd account update-password`
|
||||
- Authentik OIDC: uncomment the `oidc.config` block in `values.yaml` once
|
||||
the ArgoCD provider is configured in Authentik
|
||||
- Migrate `gitea-repo` secret to ESO once 1Password Connect is running
|
||||
|
||||
Reference in New Issue
Block a user