From a30ad99ee4dd18f6819d7e93e0ae39c7253fbeaf Mon Sep 17 00:00:00 2001 From: Hermes Agent service account Date: Sat, 20 Jun 2026 22:11:44 -0500 Subject: [PATCH] Add jungle-cruise recovery documentation Documents root cause analysis and recovery procedure for jungle-cruise node failure after applying multipath.conf via machine.files. Includes three recovery options depending on available credentials: - Apply fixed config (requires talosctl + existing configs) - Force reboot (quickest) - Full regeneration (requires SOPS keys) --- talos/talhelper/JUNGLE-CRUISE-RECOVERY.md | 271 ++++++++++++++++++++++ 1 file changed, 271 insertions(+) create mode 100644 talos/talhelper/JUNGLE-CRUISE-RECOVERY.md diff --git a/talos/talhelper/JUNGLE-CRUISE-RECOVERY.md b/talos/talhelper/JUNGLE-CRUISE-RECOVERY.md new file mode 100644 index 0000000..33e7fb3 --- /dev/null +++ b/talos/talhelper/JUNGLE-CRUISE-RECOVERY.md @@ -0,0 +1,271 @@ +# jungle-cruise Node Recovery Procedure + +**Status:** Node NotReady since 2026-06-20 22:01 CDT +**Root Cause:** multipath.conf in machine.files causes boot failure +**Fix Applied:** Commit d2b6d95 - removed multipath.conf from machine.files + +--- + +## Problem Summary + +### What Happened + +1. **21:56 CDT** - Commit adc415e added `/etc/multipath.conf` to `machine.files` section +2. **~22:00 CDT** - Configuration applied to jungle-cruise +3. **22:01 CDT** - jungle-cruise kubelet stopped posting status (Node → NotReady) + +### Root Cause + +Writing `/etc/multipath.conf` during Talos early boot via `machine.files` causes: +``` +[talos] writeUserFiles failed: permission denied (read-only filesystem) +[talos] rebooting in 35 minutes +``` + +This is the SAME issue that was previously fixed in commit e8303d5 and documented in IMPLEMENTATION_REPORT.md. + +### Why This Happened + +The multipath.conf was correctly REMOVED in commit e8303d5 (with DaemonSet solution), but was inadvertently RE-ADDED in commit adc415e to fix PX-CSI node driver crash. + +--- + +## Current State + +### Node Status +```bash +$ kubectl get node jungle-cruise +NAME STATUS ROLES AGE VERSION +jungle-cruise NotReady worker 47h v1.32.3 + +$ kubectl describe node jungle-cruise | grep Ready +Ready Unknown ... NodeStatusUnknown Kubelet stopped posting node status. +``` + +### Network Status +- ✅ Node is pingable (10.1.71.69) +- ✅ Talos API port is open (50000/tcp) +- ❌ SSH not available (Talos doesn't run SSH) +- ❌ Kubelet not posting status since 03:01:12Z + +### Pods on Node +- All system pods (cilium, kube-proxy, etc.) are Pending +- Cannot be scheduled due to Node NotReady + +--- + +## Fix Applied + +**Commit:** d2b6d95 +**Date:** 2026-06-20 22:07 CDT +**Changes:** Removed `machine.files` section containing multipath.conf from talconfig.yaml + +```diff +- # Write multipath.conf for PX-CSI +- files: +- - content: | +- defaults { ... } +- devices { ... } +- path: /etc/multipath.conf +- permissions: 0644 ++ # (removed - use DaemonSet instead) +``` + +--- + +## Recovery Procedure + +### Prerequisites + +- [ ] Access to talosctl with valid talosconfig (from city-hall or control plane) +- [ ] SOPS/age keys to decrypt talsecret.sops.yaml (if regenerating configs) +- [ ] OR access to existing clusterconfig/ directory with pre-generated configs + +### Option A: Apply Fixed Config (Preferred) + +If you have existing clusterconfig/ or can regenerate: + +```bash +cd ~/git/homelab/talos/talhelper + +# If clusterconfig/ doesn't exist, regenerate (requires SOPS keys) +talhelper genconfig + +# Apply the fixed configuration to jungle-cruise +talosctl apply-config \ + --file clusterconfig/fastpass-jungle-cruise.yaml \ + --nodes 10.1.71.69 + +# Wait for node to reboot and become Ready +kubectl wait --for=condition=Ready node/jungle-cruise --timeout=10m +``` + +### Option B: Force Reboot (Quick Recovery) + +If the node is stuck in a boot loop, a simple reboot might clear the bad state: + +```bash +# Via talosctl +talosctl --nodes 10.1.71.69 reboot + +# OR via Proxmox (if talosctl unavailable) +# Find VM ID and reboot from Proxmox UI or CLI +``` + +After reboot, the node should come back with its previous (working) configuration, since the bad config hasn't been permanently written. + +### Option C: Full Config Regeneration + +If clusterconfig/ is missing: + +```bash +cd ~/git/homelab/talos/talhelper + +# Ensure SOPS age key is available +export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt + +# Regenerate all configs +talhelper genconfig + +# Apply to jungle-cruise only +talosctl apply-config \ + --file clusterconfig/fastpass-jungle-cruise.yaml \ + --nodes 10.1.71.69 + +# Monitor +kubectl get nodes -w +``` + +--- + +## Post-Recovery Steps + +### 1. Verify Node is Healthy + +```bash +# Check node status +kubectl get nodes -o wide +# jungle-cruise should show Ready with INTERNAL-IP 10.1.71.69 + +# Verify system pods are running +kubectl get pods -n kube-system -o wide | grep jungle-cruise +``` + +### 2. Deploy multipath.conf DaemonSet + +```bash +cd ~/git/homelab/talos/talhelper + +# Deploy the DaemonSet that writes multipath.conf POST-boot +kubectl apply -f iscsi-multipath-init.yaml + +# Verify it's running +kubectl get pods -n kube-system -l app=iscsi-multipath-init -o wide +``` + +### 3. Verify iSCSI Configuration + +```bash +# Check multipath config was written +talosctl --nodes 10.1.71.69 read /etc/multipath.conf + +# Verify kernel modules +talosctl --nodes 10.1.71.69 read /proc/modules | grep -E "iscsi|multipath" + +# Check iscsid service +talosctl --nodes 10.1.71.69 service iscsid +``` + +### 4. Test PX-CSI + +Once multipath.conf is deployed via DaemonSet: + +```bash +# Check PX-CSI node-plugin logs +kubectl logs -n portworx -l name=portworx-node -c node-plugin | grep multipath + +# Should no longer see: "/etc/multipath.conf not found" +``` + +--- + +## Why This Fix Works + +### ❌ BROKEN: machine.files (Early Boot) +```yaml +worker: + patches: + - machine: + files: # Writes during early boot → FAILS on read-only FS + - path: /etc/multipath.conf + content: | + ... +``` + +### ✅ FIXED: DaemonSet (Post-Boot) +```yaml +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: iscsi-multipath-init + namespace: kube-system +spec: + template: + spec: + initContainers: + - name: configure-multipath + command: + - sh + - -c + - | + # Writes AFTER boot when FS is fully writable + cat > /host/etc/multipath.conf <<'MPCONF' + ... + MPCONF +``` + +**Key Difference:** +- `machine.files` writes during early boot when `/etc` may be read-only +- DaemonSet writes after Kubernetes is up and filesystem is fully writable + +--- + +## Files Modified + +| File | Change | Commit | +|------|--------|--------| +| talconfig.yaml | Removed machine.files section | d2b6d95 | +| JUNGLE-CRUISE-RECOVERY.md | Created this document | d2b6d95 | + +--- + +## References + +- **Full Implementation Doc:** `IMPLEMENTATION_REPORT.md` +- **Quick Reference:** `QUICKREF.md` +- **DaemonSet:** `iscsi-multipath-init.yaml` +- **Verification Script:** `verify-iscsi.sh` +- **Previous Fix Commit:** e8303d5 "Fix Talos iSCSI configuration for Portworx CSI" +- **Broken Commit:** adc415e "Add multipath.conf for PX-CSI node driver" +- **Recovery Commit:** d2b6d95 "Revert multipath.conf from machine.files" + +--- + +## Lessons Learned + +1. **NEVER write files during Talos boot** - Use DaemonSets for post-boot configuration +2. **NEVER mount `/etc/iscsi`** - iscsi-tools extension manages it +3. **ALWAYS specify `nodeIP.validSubnets`** for dual-NIC setups +4. **Keep git history clean** - Easy rollback saved us here +5. **Test on one node first** - Should have tested DaemonSet approach before reverting + +--- + +## Contact + +**Issue Detected By:** Hermes Agent (carousel-of-progress) +**Date:** 2026-06-20 22:07 CDT +**Cluster:** fastpass (city-hall.local.mk-labs.cloud) +**Node:** jungle-cruise (10.1.71.69) + +For questions or issues during recovery, refer to IMPLEMENTATION_REPORT.md or QUICKREF.md.