# jungle-cruise Node Recovery Procedure

**Status:** Node NotReady since 2026-06-20 22:01 CDT  
**Root Cause:** multipath.conf in machine.files causes boot failure  
**Fix Applied:** Commit d2b6d95 - removed multipath.conf from machine.files  

---

## Problem Summary

### What Happened

1. **21:56 CDT** - Commit adc415e added `/etc/multipath.conf` to `machine.files` section
2. **~22:00 CDT** - Configuration applied to jungle-cruise
3. **22:01 CDT** - jungle-cruise kubelet stopped posting status (Node → NotReady)

### Root Cause

Writing `/etc/multipath.conf` during Talos early boot via `machine.files` causes:
```
[talos] writeUserFiles failed: permission denied (read-only filesystem)
[talos] rebooting in 35 minutes
```

This is the SAME issue that was previously fixed in commit e8303d5 and documented in IMPLEMENTATION_REPORT.md.

### Why This Happened

The multipath.conf was correctly REMOVED in commit e8303d5 (with DaemonSet solution), but was inadvertently RE-ADDED in commit adc415e to fix PX-CSI node driver crash.

---

## Current State

### Node Status
```bash
$ kubectl get node jungle-cruise
NAME            STATUS     ROLES    AGE   VERSION
jungle-cruise   NotReady   worker   47h   v1.32.3

$ kubectl describe node jungle-cruise | grep Ready
Ready                Unknown   ... NodeStatusUnknown   Kubelet stopped posting node status.
```

### Network Status
- ✅ Node is pingable (10.1.71.69)
- ✅ Talos API port is open (50000/tcp)
- ❌ SSH not available (Talos doesn't run SSH)
- ❌ Kubelet not posting status since 03:01:12Z

### Pods on Node
- All system pods (cilium, kube-proxy, etc.) are Pending
- Cannot be scheduled due to Node NotReady

---

## Fix Applied

**Commit:** d2b6d95  
**Date:** 2026-06-20 22:07 CDT  
**Changes:** Removed `machine.files` section containing multipath.conf from talconfig.yaml

```diff
-        # Write multipath.conf for PX-CSI
-        files:
-          - content: |
-              defaults { ... }
-              devices { ... }
-            path: /etc/multipath.conf
-            permissions: 0644
+        # (removed - use DaemonSet instead)
```

---

## Recovery Procedure

### Prerequisites

- [ ] Access to talosctl with valid talosconfig (from city-hall or control plane)
- [ ] SOPS/age keys to decrypt talsecret.sops.yaml (if regenerating configs)
- [ ] OR access to existing clusterconfig/ directory with pre-generated configs

### Option A: Apply Fixed Config (Preferred)

If you have existing clusterconfig/ or can regenerate:

```bash
cd ~/git/homelab/talos/talhelper

# If clusterconfig/ doesn't exist, regenerate (requires SOPS keys)
talhelper genconfig

# Apply the fixed configuration to jungle-cruise
talosctl apply-config \
  --file clusterconfig/fastpass-jungle-cruise.yaml \
  --nodes 10.1.71.69

# Wait for node to reboot and become Ready
kubectl wait --for=condition=Ready node/jungle-cruise --timeout=10m
```

### Option B: Force Reboot (Quick Recovery)

If the node is stuck in a boot loop, a simple reboot might clear the bad state:

```bash
# Via talosctl
talosctl --nodes 10.1.71.69 reboot

# OR via Proxmox (if talosctl unavailable)
# Find VM ID and reboot from Proxmox UI or CLI
```

After reboot, the node should come back with its previous (working) configuration, since the bad config hasn't been permanently written.

### Option C: Full Config Regeneration

If clusterconfig/ is missing:

```bash
cd ~/git/homelab/talos/talhelper

# Ensure SOPS age key is available
export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt

# Regenerate all configs
talhelper genconfig

# Apply to jungle-cruise only
talosctl apply-config \
  --file clusterconfig/fastpass-jungle-cruise.yaml \
  --nodes 10.1.71.69

# Monitor
kubectl get nodes -w
```

---

## Post-Recovery Steps

### 1. Verify Node is Healthy

```bash
# Check node status
kubectl get nodes -o wide
# jungle-cruise should show Ready with INTERNAL-IP 10.1.71.69

# Verify system pods are running
kubectl get pods -n kube-system -o wide | grep jungle-cruise
```

### 2. Deploy multipath.conf DaemonSet

```bash
cd ~/git/homelab/talos/talhelper

# Deploy the DaemonSet that writes multipath.conf POST-boot
kubectl apply -f iscsi-multipath-init.yaml

# Verify it's running
kubectl get pods -n kube-system -l app=iscsi-multipath-init -o wide
```

### 3. Verify iSCSI Configuration

```bash
# Check multipath config was written
talosctl --nodes 10.1.71.69 read /etc/multipath.conf

# Verify kernel modules
talosctl --nodes 10.1.71.69 read /proc/modules | grep -E "iscsi|multipath"

# Check iscsid service
talosctl --nodes 10.1.71.69 service iscsid
```

### 4. Test PX-CSI

Once multipath.conf is deployed via DaemonSet:

```bash
# Check PX-CSI node-plugin logs
kubectl logs -n portworx -l name=portworx-node -c node-plugin | grep multipath

# Should no longer see: "/etc/multipath.conf not found"
```

---

## Why This Fix Works

### ❌ BROKEN: machine.files (Early Boot)
```yaml
worker:
  patches:
    - machine:
        files:  # Writes during early boot → FAILS on read-only FS
          - path: /etc/multipath.conf
            content: |
              ...
```

### ✅ FIXED: DaemonSet (Post-Boot)
```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: iscsi-multipath-init
  namespace: kube-system
spec:
  template:
    spec:
      initContainers:
      - name: configure-multipath
        command:
        - sh
        - -c
        - |
          # Writes AFTER boot when FS is fully writable
          cat > /host/etc/multipath.conf <<'MPCONF'
          ...
          MPCONF
```

**Key Difference:**
- `machine.files` writes during early boot when `/etc` may be read-only
- DaemonSet writes after Kubernetes is up and filesystem is fully writable

---

## Files Modified

| File | Change | Commit |
|------|--------|--------|
| talconfig.yaml | Removed machine.files section | d2b6d95 |
| JUNGLE-CRUISE-RECOVERY.md | Created this document | d2b6d95 |

---

## References

- **Full Implementation Doc:** `IMPLEMENTATION_REPORT.md`
- **Quick Reference:** `QUICKREF.md`
- **DaemonSet:** `iscsi-multipath-init.yaml`
- **Verification Script:** `verify-iscsi.sh`
- **Previous Fix Commit:** e8303d5 "Fix Talos iSCSI configuration for Portworx CSI"
- **Broken Commit:** adc415e "Add multipath.conf for PX-CSI node driver"
- **Recovery Commit:** d2b6d95 "Revert multipath.conf from machine.files"

---

## Lessons Learned

1. **NEVER write files during Talos boot** - Use DaemonSets for post-boot configuration
2. **NEVER mount `/etc/iscsi`** - iscsi-tools extension manages it
3. **ALWAYS specify `nodeIP.validSubnets`** for dual-NIC setups
4. **Keep git history clean** - Easy rollback saved us here
5. **Test on one node first** - Should have tested DaemonSet approach before reverting

---

## Contact

**Issue Detected By:** Hermes Agent (carousel-of-progress)  
**Date:** 2026-06-20 22:07 CDT  
**Cluster:** fastpass (city-hall.local.mk-labs.cloud)  
**Node:** jungle-cruise (10.1.71.69)

For questions or issues during recovery, refer to IMPLEMENTATION_REPORT.md or QUICKREF.md.