homelab/talos/talhelper/JUNGLE-CRUISE-RECOVERY.md

jungle-cruise Node Recovery Procedure

Status: Node NotReady since 2026-06-20 22:01 CDT
Root Cause: multipath.conf in machine.files causes boot failure
Fix Applied: Commit d2b6d95 - removed multipath.conf from machine.files

---

Problem Summary

What Happened

1. 21:56 CDT - Commit adc415e added /etc/multipath.conf to machine.files section
2. ~22:00 CDT - Configuration applied to jungle-cruise
3. 22:01 CDT - jungle-cruise kubelet stopped posting status (Node → NotReady)

Root Cause

Writing /etc/multipath.conf during Talos early boot via machine.files causes:

[talos] writeUserFiles failed: permission denied (read-only filesystem)
[talos] rebooting in 35 minutes

This is the SAME issue that was previously fixed in commit e8303d5 and documented in IMPLEMENTATION_REPORT.md.

Why This Happened

The multipath.conf was correctly REMOVED in commit e8303d5 (with DaemonSet solution), but was inadvertently RE-ADDED in commit adc415e to fix PX-CSI node driver crash.

---

Current State

Node Status

$ kubectl get node jungle-cruise
NAME            STATUS     ROLES    AGE   VERSION
jungle-cruise   NotReady   worker   47h   v1.32.3

$ kubectl describe node jungle-cruise | grep Ready
Ready                Unknown   ... NodeStatusUnknown   Kubelet stopped posting node status.

Network Status

• ✅ Node is pingable (10.1.71.69)
• ✅ Talos API port is open (50000/tcp)
• ❌ SSH not available (Talos doesn't run SSH)
• ❌ Kubelet not posting status since 03:01:12Z

Pods on Node

• All system pods (cilium, kube-proxy, etc.) are Pending
• Cannot be scheduled due to Node NotReady

---

Fix Applied

Commit: d2b6d95
Date: 2026-06-20 22:07 CDT
Changes: Removed machine.files section containing multipath.conf from talconfig.yaml

-        # Write multipath.conf for PX-CSI
-        files:
-          - content: |
-              defaults { ... }
-              devices { ... }
-            path: /etc/multipath.conf
-            permissions: 0644
+        # (removed - use DaemonSet instead)

---

Recovery Procedure

Prerequisites

• Access to talosctl with valid talosconfig (from city-hall or control plane)
• SOPS/age keys to decrypt talsecret.sops.yaml (if regenerating configs)
• OR access to existing clusterconfig/ directory with pre-generated configs

Option A: Apply Fixed Config (Preferred)

If you have existing clusterconfig/ or can regenerate:

cd ~/git/homelab/talos/talhelper

# If clusterconfig/ doesn't exist, regenerate (requires SOPS keys)
talhelper genconfig

# Apply the fixed configuration to jungle-cruise
talosctl apply-config \
  --file clusterconfig/fastpass-jungle-cruise.yaml \
  --nodes 10.1.71.69

# Wait for node to reboot and become Ready
kubectl wait --for=condition=Ready node/jungle-cruise --timeout=10m

Option B: Force Reboot (Quick Recovery)

If the node is stuck in a boot loop, a simple reboot might clear the bad state:

# Via talosctl
talosctl --nodes 10.1.71.69 reboot

# OR via Proxmox (if talosctl unavailable)
# Find VM ID and reboot from Proxmox UI or CLI

After reboot, the node should come back with its previous (working) configuration, since the bad config hasn't been permanently written.

Option C: Full Config Regeneration

If clusterconfig/ is missing:

cd ~/git/homelab/talos/talhelper

# Ensure SOPS age key is available
export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt

# Regenerate all configs
talhelper genconfig

# Apply to jungle-cruise only
talosctl apply-config \
  --file clusterconfig/fastpass-jungle-cruise.yaml \
  --nodes 10.1.71.69

# Monitor
kubectl get nodes -w

---

Post-Recovery Steps

1. Verify Node is Healthy

# Check node status
kubectl get nodes -o wide
# jungle-cruise should show Ready with INTERNAL-IP 10.1.71.69

# Verify system pods are running
kubectl get pods -n kube-system -o wide | grep jungle-cruise

2. Deploy multipath.conf DaemonSet

cd ~/git/homelab/talos/talhelper

# Deploy the DaemonSet that writes multipath.conf POST-boot
kubectl apply -f iscsi-multipath-init.yaml

# Verify it's running
kubectl get pods -n kube-system -l app=iscsi-multipath-init -o wide

3. Verify iSCSI Configuration

# Check multipath config was written
talosctl --nodes 10.1.71.69 read /etc/multipath.conf

# Verify kernel modules
talosctl --nodes 10.1.71.69 read /proc/modules | grep -E "iscsi|multipath"

# Check iscsid service
talosctl --nodes 10.1.71.69 service iscsid

4. Test PX-CSI

Once multipath.conf is deployed via DaemonSet:

# Check PX-CSI node-plugin logs
kubectl logs -n portworx -l name=portworx-node -c node-plugin | grep multipath

# Should no longer see: "/etc/multipath.conf not found"

---

Why This Fix Works

❌ BROKEN: machine.files (Early Boot)

worker:
  patches:
    - machine:
        files:  # Writes during early boot → FAILS on read-only FS
          - path: /etc/multipath.conf
            content: |
              ...

✅ FIXED: DaemonSet (Post-Boot)

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: iscsi-multipath-init
  namespace: kube-system
spec:
  template:
    spec:
      initContainers:
      - name: configure-multipath
        command:
        - sh
        - -c
        - |
          # Writes AFTER boot when FS is fully writable
          cat > /host/etc/multipath.conf <<'MPCONF'
          ...
          MPCONF

Key Difference:

• machine.files writes during early boot when /etc may be read-only
• DaemonSet writes after Kubernetes is up and filesystem is fully writable

---

Files Modified

File	Change	Commit
talconfig.yaml	Removed machine.files section	d2b6d95
JUNGLE-CRUISE-RECOVERY.md	Created this document	d2b6d95

---

References

• Full Implementation Doc: IMPLEMENTATION_REPORT.md
• Quick Reference: QUICKREF.md
• DaemonSet: iscsi-multipath-init.yaml
• Verification Script: verify-iscsi.sh
• Previous Fix Commit: e8303d5 "Fix Talos iSCSI configuration for Portworx CSI"
• Broken Commit: adc415e "Add multipath.conf for PX-CSI node driver"
• Recovery Commit: d2b6d95 "Revert multipath.conf from machine.files"

---

Lessons Learned

1. NEVER write files during Talos boot - Use DaemonSets for post-boot configuration
2. NEVER mount /etc/iscsi - iscsi-tools extension manages it
3. ALWAYS specify nodeIP.validSubnets for dual-NIC setups
4. Keep git history clean - Easy rollback saved us here
5. Test on one node first - Should have tested DaemonSet approach before reverting

---

Contact

Issue Detected By: Hermes Agent (carousel-of-progress)
Date: 2026-06-20 22:07 CDT
Cluster: fastpass (city-hall.local.mk-labs.cloud)
Node: jungle-cruise (10.1.71.69)

For questions or issues during recovery, refer to IMPLEMENTATION_REPORT.md or QUICKREF.md.