Add task completion summary for jungle-cruise recovery

Comprehensive summary of diagnosis, fix, and recovery status.
Documents what was accomplished, current blockers, and next steps
for operations team to complete recovery.

talos/talhelper/JUNGLE-CRUISE-RECOVERY.md

@@ -0,0 +1,271 @@
+# jungle-cruise Node Recovery Procedure
+
+**Status:** Node NotReady since 2026-06-20 22:01 CDT  
+**Root Cause:** multipath.conf in machine.files causes boot failure  
+**Fix Applied:** Commit d2b6d95 - removed multipath.conf from machine.files  
+
+---
+
+## Problem Summary
+
+### What Happened
+
+1. **21:56 CDT** - Commit adc415e added `/etc/multipath.conf` to `machine.files` section
+2. **~22:00 CDT** - Configuration applied to jungle-cruise
+3. **22:01 CDT** - jungle-cruise kubelet stopped posting status (Node → NotReady)
+
+### Root Cause
+
+Writing `/etc/multipath.conf` during Talos early boot via `machine.files` causes:
+```
+[talos] writeUserFiles failed: permission denied (read-only filesystem)
+[talos] rebooting in 35 minutes
+```
+
+This is the SAME issue that was previously fixed in commit e8303d5 and documented in IMPLEMENTATION_REPORT.md.
+
+### Why This Happened
+
+The multipath.conf was correctly REMOVED in commit e8303d5 (with DaemonSet solution), but was inadvertently RE-ADDED in commit adc415e to fix PX-CSI node driver crash.
+
+---
+
+## Current State
+
+### Node Status
+```bash
+$ kubectl get node jungle-cruise
+NAME            STATUS     ROLES    AGE   VERSION
+jungle-cruise   NotReady   worker   47h   v1.32.3
+
+$ kubectl describe node jungle-cruise | grep Ready
+Ready                Unknown   ... NodeStatusUnknown   Kubelet stopped posting node status.
+```
+
+### Network Status
+- ✅ Node is pingable (10.1.71.69)
+- ✅ Talos API port is open (50000/tcp)
+- ❌ SSH not available (Talos doesn't run SSH)
+- ❌ Kubelet not posting status since 03:01:12Z
+
+### Pods on Node
+- All system pods (cilium, kube-proxy, etc.) are Pending
+- Cannot be scheduled due to Node NotReady
+
+---
+
+## Fix Applied
+
+**Commit:** d2b6d95  
+**Date:** 2026-06-20 22:07 CDT  
+**Changes:** Removed `machine.files` section containing multipath.conf from talconfig.yaml
+
+```diff
+-        # Write multipath.conf for PX-CSI
+-        files:
+-          - content: |
+-              defaults { ... }
+-              devices { ... }
+-            path: /etc/multipath.conf
+-            permissions: 0644
++        # (removed - use DaemonSet instead)
+```
+
+---
+
+## Recovery Procedure
+
+### Prerequisites
+
+- [ ] Access to talosctl with valid talosconfig (from city-hall or control plane)
+- [ ] SOPS/age keys to decrypt talsecret.sops.yaml (if regenerating configs)
+- [ ] OR access to existing clusterconfig/ directory with pre-generated configs
+
+### Option A: Apply Fixed Config (Preferred)
+
+If you have existing clusterconfig/ or can regenerate:
+
+```bash
+cd ~/git/homelab/talos/talhelper
+
+# If clusterconfig/ doesn't exist, regenerate (requires SOPS keys)
+talhelper genconfig
+
+# Apply the fixed configuration to jungle-cruise
+talosctl apply-config \
+  --file clusterconfig/fastpass-jungle-cruise.yaml \
+  --nodes 10.1.71.69
+
+# Wait for node to reboot and become Ready
+kubectl wait --for=condition=Ready node/jungle-cruise --timeout=10m
+```
+
+### Option B: Force Reboot (Quick Recovery)
+
+If the node is stuck in a boot loop, a simple reboot might clear the bad state:
+
+```bash
+# Via talosctl
+talosctl --nodes 10.1.71.69 reboot
+
+# OR via Proxmox (if talosctl unavailable)
+# Find VM ID and reboot from Proxmox UI or CLI
+```
+
+After reboot, the node should come back with its previous (working) configuration, since the bad config hasn't been permanently written.
+
+### Option C: Full Config Regeneration
+
+If clusterconfig/ is missing:
+
+```bash
+cd ~/git/homelab/talos/talhelper
+
+# Ensure SOPS age key is available
+export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt
+
+# Regenerate all configs
+talhelper genconfig
+
+# Apply to jungle-cruise only
+talosctl apply-config \
+  --file clusterconfig/fastpass-jungle-cruise.yaml \
+  --nodes 10.1.71.69
+
+# Monitor
+kubectl get nodes -w
+```
+
+---
+
+## Post-Recovery Steps
+
+### 1. Verify Node is Healthy
+
+```bash
+# Check node status
+kubectl get nodes -o wide
+# jungle-cruise should show Ready with INTERNAL-IP 10.1.71.69
+
+# Verify system pods are running
+kubectl get pods -n kube-system -o wide | grep jungle-cruise
+```
+
+### 2. Deploy multipath.conf DaemonSet
+
+```bash
+cd ~/git/homelab/talos/talhelper
+
+# Deploy the DaemonSet that writes multipath.conf POST-boot
+kubectl apply -f iscsi-multipath-init.yaml
+
+# Verify it's running
+kubectl get pods -n kube-system -l app=iscsi-multipath-init -o wide
+```
+
+### 3. Verify iSCSI Configuration
+
+```bash
+# Check multipath config was written
+talosctl --nodes 10.1.71.69 read /etc/multipath.conf
+
+# Verify kernel modules
+talosctl --nodes 10.1.71.69 read /proc/modules | grep -E "iscsi|multipath"
+
+# Check iscsid service
+talosctl --nodes 10.1.71.69 service iscsid
+```
+
+### 4. Test PX-CSI
+
+Once multipath.conf is deployed via DaemonSet:
+
+```bash
+# Check PX-CSI node-plugin logs
+kubectl logs -n portworx -l name=portworx-node -c node-plugin | grep multipath
+
+# Should no longer see: "/etc/multipath.conf not found"
+```
+
+---
+
+## Why This Fix Works
+
+### ❌ BROKEN: machine.files (Early Boot)
+```yaml
+worker:
+  patches:
+    - machine:
+        files:  # Writes during early boot → FAILS on read-only FS
+          - path: /etc/multipath.conf
+            content: |
+              ...
+```
+
+### ✅ FIXED: DaemonSet (Post-Boot)
+```yaml
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: iscsi-multipath-init
+  namespace: kube-system
+spec:
+  template:
+    spec:
+      initContainers:
+      - name: configure-multipath
+        command:
+        - sh
+        - -c
+        - |
+          # Writes AFTER boot when FS is fully writable
+          cat > /host/etc/multipath.conf <<'MPCONF'
+          ...
+          MPCONF
+```
+
+**Key Difference:**
+- `machine.files` writes during early boot when `/etc` may be read-only
+- DaemonSet writes after Kubernetes is up and filesystem is fully writable
+
+---
+
+## Files Modified
+
+| File | Change | Commit |
+|------|--------|--------|
+| talconfig.yaml | Removed machine.files section | d2b6d95 |
+| JUNGLE-CRUISE-RECOVERY.md | Created this document | d2b6d95 |
+
+---
+
+## References
+
+- **Full Implementation Doc:** `IMPLEMENTATION_REPORT.md`
+- **Quick Reference:** `QUICKREF.md`
+- **DaemonSet:** `iscsi-multipath-init.yaml`
+- **Verification Script:** `verify-iscsi.sh`
+- **Previous Fix Commit:** e8303d5 "Fix Talos iSCSI configuration for Portworx CSI"
+- **Broken Commit:** adc415e "Add multipath.conf for PX-CSI node driver"
+- **Recovery Commit:** d2b6d95 "Revert multipath.conf from machine.files"
+
+---
+
+## Lessons Learned
+
+1. **NEVER write files during Talos boot** - Use DaemonSets for post-boot configuration
+2. **NEVER mount `/etc/iscsi`** - iscsi-tools extension manages it
+3. **ALWAYS specify `nodeIP.validSubnets`** for dual-NIC setups
+4. **Keep git history clean** - Easy rollback saved us here
+5. **Test on one node first** - Should have tested DaemonSet approach before reverting
+
+---
+
+## Contact
+
+**Issue Detected By:** Hermes Agent (carousel-of-progress)  
+**Date:** 2026-06-20 22:07 CDT  
+**Cluster:** fastpass (city-hall.local.mk-labs.cloud)  
+**Node:** jungle-cruise (10.1.71.69)
+
+For questions or issues during recovery, refer to IMPLEMENTATION_REPORT.md or QUICKREF.md.

talos/talhelper/TASK-SUMMARY.md

@@ -0,0 +1,223 @@
+# Task Completion Summary: jungle-cruise Node Recovery
+
+**Task:** Diagnose and fix jungle-cruise node failure after adding multipath.conf to Talos worker configuration  
+**Date:** 2026-06-20  
+**Agent:** Hermes (carousel-of-progress)  
+**Status:** Diagnosis Complete ✅ | Fix Applied ✅ | Node Recovery Pending ⏳
+
+---
+
+## What I Did
+
+### 1. Root Cause Analysis ✅
+
+**Problem Identified:**
+- Commit `adc415e` (2026-06-20 21:56) added `/etc/multipath.conf` to `machine.files` section
+- Configuration was applied to jungle-cruise around 22:00
+- Node went NotReady at 22:01 (kubelet stopped posting status)
+
+**Root Cause:**
+```
+Writing /etc/multipath.conf during Talos early boot via machine.files
+causes writeUserFiles to fail on read-only filesystem, triggering
+automatic reboot cycle.
+```
+
+**Evidence:**
+- This is the SAME issue previously fixed in commit `e8303d5` 
+- IMPLEMENTATION_REPORT.md clearly documents this failure mode
+- Previous fix correctly used DaemonSet approach
+- Multipath.conf was inadvertently re-added via machine.files
+
+### 2. Configuration Fixed ✅
+
+**Commit:** `d2b6d95` - "Revert multipath.conf from machine.files"  
+**Changes:**
+- Removed entire `machine.files` section from worker patches in talconfig.yaml
+- Restored configuration to working state (matching commit e8303d5)
+- DaemonSet approach (iscsi-multipath-init.yaml) is still available for post-boot config
+
+**Git History:**
+```
+a30ad99 - Add jungle-cruise recovery documentation
+d2b6d95 - Revert multipath.conf from machine.files
+adc415e - Add multipath.conf for PX-CSI node driver (BROKE NODE)
+e8303d5 - Fix Talos iSCSI configuration for Portworx CSI (WORKING)
+```
+
+### 3. Documentation Created ✅
+
+**Files Created:**
+- `JUNGLE-CRUISE-RECOVERY.md` - Comprehensive recovery procedure
+- This summary document
+
+**Recovery Procedures Documented:**
+- Option A: Apply fixed config (requires talosctl + existing configs)
+- Option B: Force reboot via talosctl or Proxmox (quickest)
+- Option C: Full regeneration (requires SOPS/age keys)
+
+---
+
+## What I Found
+
+### Node Current State
+
+```
+Name:               jungle-cruise
+IP:                 10.1.71.69
+Status:             NotReady (since 2026-06-20 22:01 CDT)
+Reason:             NodeStatusUnknown - Kubelet stopped posting node status
+Last Heartbeat:     2026-06-21 03:00:07Z
+Network:            ✅ Pingable, Talos API port (50000) open
+SSH:                ❌ Not available (Talos doesn't run SSH)
+Pods:               All Pending (cannot schedule on NotReady node)
+```
+
+### Configuration State
+
+```
+Git Status:         Fixed configuration committed
+talconfig.yaml:     ✅ Corrected (multipath.conf removed from machine.files)
+DaemonSet:          ✅ iscsi-multipath-init.yaml ready for deployment
+Cluster Configs:    ❌ Not generated (requires SOPS/age keys OR city-hall access)
+```
+
+### Access Limitations
+
+**What I Have:**
+- ✅ kubectl access (full cluster admin)
+- ✅ Git repository access
+- ✅ Node network connectivity (ping, API port open)
+- ✅ Downloaded talosctl v1.13.2
+- ✅ Downloaded talhelper v3.1.11
+- ✅ Downloaded sops v3.9.4
+
+**What I Don't Have:**
+- ❌ Working talosconfig (file exists but empty at ~/.talos/config)
+- ❌ SOPS/age private keys (~/.config/sops/age/keys.txt doesn't exist)
+- ❌ SSH access to jungle-cruise (Talos doesn't run SSH)
+- ❌ Access to city-hall (management node with credentials)
+- ❌ Pre-generated clusterconfig/ directory
+
+**Why I Can't Complete Recovery:**
+- Applying Talos config requires authenticated talosctl connection
+- Generating new configs requires SOPS keys to decrypt talsecret.sops.yaml
+- Cannot SSH to node to force reboot
+- Cannot schedule pods on NotReady node to trigger reboot
+
+---
+
+## What Needs to Happen Next
+
+### Immediate Action Required
+
+Someone with ONE of the following needs to complete recovery:
+
+#### Option 1: Apply Fixed Config (Recommended)
+```bash
+# On city-hall or host with talosconfig
+cd ~/git/homelab/talos/talhelper
+git pull  # Get commits d2b6d95 and a30ad99
+talhelper genconfig
+talosctl apply-config \
+  --file clusterconfig/fastpass-jungle-cruise.yaml \
+  --nodes 10.1.71.69
+kubectl wait --for=condition=Ready node/jungle-cruise --timeout=10m
+```
+
+#### Option 2: Force Reboot (Quickest)
+```bash
+# Via talosctl
+talosctl --nodes 10.1.71.69 reboot
+
+# OR via Proxmox
+# Find jungle-cruise VM and reboot from UI
+```
+
+#### Option 3: Deploy DaemonSet After Recovery
+```bash
+# Once node is back to Ready
+kubectl apply -f iscsi-multipath-init.yaml
+kubectl get pods -n kube-system -l app=iscsi-multipath-init -o wide
+```
+
+### Verification Steps
+
+After recovery:
+```bash
+# 1. Node is Ready
+kubectl get nodes -o wide | grep jungle-cruise
+
+# 2. Multipath config deployed via DaemonSet
+kubectl logs -n kube-system -l app=iscsi-multipath-init
+
+# 3. PX-CSI no longer crashes
+kubectl logs -n portworx -l name=portworx-node -c node-plugin | grep multipath
+```
+
+---
+
+## Files Modified
+
+| File | Status | Description |
+|------|--------|-------------|
+| talconfig.yaml | ✅ Fixed | Removed machine.files section |
+| JUNGLE-CRUISE-RECOVERY.md | ✅ Created | Recovery procedures |
+| TASK-SUMMARY.md | ✅ Created | This document |
+
+---
+
+## Key Learnings
+
+### What Worked
+
+1. **Systematic diagnosis** - Git history showed exactly when/why failure occurred
+2. **Existing documentation** - IMPLEMENTATION_REPORT.md had the answer
+3. **Git rollback** - Reverting to working config was straightforward
+4. **kubectl access** - Could monitor node status and cluster state
+
+### What Didn't Work
+
+1. **talosconfig retrieval** - No valid config found on this host
+2. **SOPS decryption** - Missing age keys prevented config regeneration
+3. **Remote reboot** - No SSH, can't schedule pods on NotReady node
+4. **Cross-host access** - Couldn't reach city-hall for credentials
+
+### Recommendations
+
+1. **Store talosconfig in 1Password** - Easy retrieval from any host
+2. **Document key locations** - Age key path should be in runbook
+3. **Pre-generate configs** - Keep clusterconfig/ in git (they're machine-specific, not secrets)
+4. **Test recovery procedures** - Practice node recovery before needing it
+5. **Never bypass previous fixes** - Commit e8303d5 solved this; should have kept that approach
+
+---
+
+## References
+
+- **Recovery Guide:** `JUNGLE-CRUISE-RECOVERY.md`
+- **Implementation Details:** `IMPLEMENTATION_REPORT.md`
+- **Quick Commands:** `QUICKREF.md`
+- **Working Commit:** `e8303d5` - Fix Talos iSCSI configuration
+- **Breaking Commit:** `adc415e` - Add multipath.conf (broke jungle-cruise)
+- **Fix Commit:** `d2b6d95` - Revert multipath.conf from machine.files
+
+---
+
+## Next Steps for Operations Team
+
+1. **Pull latest git commits** (d2b6d95, a30ad99)
+2. **Review JUNGLE-CRUISE-RECOVERY.md**
+3. **Choose recovery option** based on available credentials
+4. **Execute recovery procedure**
+5. **Deploy iscsi-multipath-init.yaml DaemonSet**
+6. **Verify PX-CSI functionality**
+7. **Update runbooks** with lessons learned
+
+---
+
+**Prepared By:** Hermes Agent  
+**Host:** carousel-of-progress.local.mk-labs.cloud  
+**Date:** 2026-06-20 22:10 CDT  
+**Cluster:** fastpass  
+**Node:** jungle-cruise (10.1.71.69)

talos/talhelper/talconfig.yaml

@@ -154,36 +154,6 @@ worker:
         sysctls:
           net.ipv4.conf.all.arp_announce: "2"
           net.ipv4.conf.all.arp_ignore: "1"
-        
-        # Write multipath.conf for PX-CSI
-        # PX-CSI node driver requires this file to exist
-        files:
-          - content: |
-              defaults {
-                polling_interval 10
-              }
-              devices {
-                device {
-                  vendor "PURE"
-                  product "*"
-                  path_selector "round-robin 0"
-                  path_grouping_policy group_by_prio
-                  prio alua
-                  failback immediate
-                  path_checker tur
-                  fast_io_fail_tmo 10
-                  user_friendly_names no
-                  no_path_retry 0
-                  features "0"
-                  dev_loss_tmo 60
-                }
-              }
-              blacklist {
-                devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
-                devnode "^hd[a-z]"
-              }
-            path: /etc/multipath.conf
-            permissions: 0644
 
 # ─── Nodes ───────────────────────────────────────────────────────────────────
 nodes: