Add jungle-cruise recovery documentation
Documents root cause analysis and recovery procedure for jungle-cruise node failure after applying multipath.conf via machine.files. Includes three recovery options depending on available credentials: - Apply fixed config (requires talosctl + existing configs) - Force reboot (quickest) - Full regeneration (requires SOPS keys)
This commit is contained in:
Hermes Agent service account
271
talos/talhelper/JUNGLE-CRUISE-RECOVERY.md
Normal file
271
talos/talhelper/JUNGLE-CRUISE-RECOVERY.md
Normal file
@@ -0,0 +1,271 @@
|
|||||||
|
# jungle-cruise Node Recovery Procedure
|
||||||
|
|
||||||
|
**Status:** Node NotReady since 2026-06-20 22:01 CDT
|
||||||
|
**Root Cause:** multipath.conf in machine.files causes boot failure
|
||||||
|
**Fix Applied:** Commit d2b6d95 - removed multipath.conf from machine.files
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Problem Summary
|
||||||
|
|
||||||
|
### What Happened
|
||||||
|
|
||||||
|
1. **21:56 CDT** - Commit adc415e added `/etc/multipath.conf` to `machine.files` section
|
||||||
|
2. **~22:00 CDT** - Configuration applied to jungle-cruise
|
||||||
|
3. **22:01 CDT** - jungle-cruise kubelet stopped posting status (Node → NotReady)
|
||||||
|
|
||||||
|
### Root Cause
|
||||||
|
|
||||||
|
Writing `/etc/multipath.conf` during Talos early boot via `machine.files` causes:
|
||||||
|
```
|
||||||
|
[talos] writeUserFiles failed: permission denied (read-only filesystem)
|
||||||
|
[talos] rebooting in 35 minutes
|
||||||
|
```
|
||||||
|
|
||||||
|
This is the SAME issue that was previously fixed in commit e8303d5 and documented in IMPLEMENTATION_REPORT.md.
|
||||||
|
|
||||||
|
### Why This Happened
|
||||||
|
|
||||||
|
The multipath.conf was correctly REMOVED in commit e8303d5 (with DaemonSet solution), but was inadvertently RE-ADDED in commit adc415e to fix PX-CSI node driver crash.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current State
|
||||||
|
|
||||||
|
### Node Status
|
||||||
|
```bash
|
||||||
|
$ kubectl get node jungle-cruise
|
||||||
|
NAME STATUS ROLES AGE VERSION
|
||||||
|
jungle-cruise NotReady worker 47h v1.32.3
|
||||||
|
|
||||||
|
$ kubectl describe node jungle-cruise | grep Ready
|
||||||
|
Ready Unknown ... NodeStatusUnknown Kubelet stopped posting node status.
|
||||||
|
```
|
||||||
|
|
||||||
|
### Network Status
|
||||||
|
- ✅ Node is pingable (10.1.71.69)
|
||||||
|
- ✅ Talos API port is open (50000/tcp)
|
||||||
|
- ❌ SSH not available (Talos doesn't run SSH)
|
||||||
|
- ❌ Kubelet not posting status since 03:01:12Z
|
||||||
|
|
||||||
|
### Pods on Node
|
||||||
|
- All system pods (cilium, kube-proxy, etc.) are Pending
|
||||||
|
- Cannot be scheduled due to Node NotReady
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Fix Applied
|
||||||
|
|
||||||
|
**Commit:** d2b6d95
|
||||||
|
**Date:** 2026-06-20 22:07 CDT
|
||||||
|
**Changes:** Removed `machine.files` section containing multipath.conf from talconfig.yaml
|
||||||
|
|
||||||
|
```diff
|
||||||
|
- # Write multipath.conf for PX-CSI
|
||||||
|
- files:
|
||||||
|
- - content: |
|
||||||
|
- defaults { ... }
|
||||||
|
- devices { ... }
|
||||||
|
- path: /etc/multipath.conf
|
||||||
|
- permissions: 0644
|
||||||
|
+ # (removed - use DaemonSet instead)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Recovery Procedure
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
|
||||||
|
- [ ] Access to talosctl with valid talosconfig (from city-hall or control plane)
|
||||||
|
- [ ] SOPS/age keys to decrypt talsecret.sops.yaml (if regenerating configs)
|
||||||
|
- [ ] OR access to existing clusterconfig/ directory with pre-generated configs
|
||||||
|
|
||||||
|
### Option A: Apply Fixed Config (Preferred)
|
||||||
|
|
||||||
|
If you have existing clusterconfig/ or can regenerate:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/git/homelab/talos/talhelper
|
||||||
|
|
||||||
|
# If clusterconfig/ doesn't exist, regenerate (requires SOPS keys)
|
||||||
|
talhelper genconfig
|
||||||
|
|
||||||
|
# Apply the fixed configuration to jungle-cruise
|
||||||
|
talosctl apply-config \
|
||||||
|
--file clusterconfig/fastpass-jungle-cruise.yaml \
|
||||||
|
--nodes 10.1.71.69
|
||||||
|
|
||||||
|
# Wait for node to reboot and become Ready
|
||||||
|
kubectl wait --for=condition=Ready node/jungle-cruise --timeout=10m
|
||||||
|
```
|
||||||
|
|
||||||
|
### Option B: Force Reboot (Quick Recovery)
|
||||||
|
|
||||||
|
If the node is stuck in a boot loop, a simple reboot might clear the bad state:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Via talosctl
|
||||||
|
talosctl --nodes 10.1.71.69 reboot
|
||||||
|
|
||||||
|
# OR via Proxmox (if talosctl unavailable)
|
||||||
|
# Find VM ID and reboot from Proxmox UI or CLI
|
||||||
|
```
|
||||||
|
|
||||||
|
After reboot, the node should come back with its previous (working) configuration, since the bad config hasn't been permanently written.
|
||||||
|
|
||||||
|
### Option C: Full Config Regeneration
|
||||||
|
|
||||||
|
If clusterconfig/ is missing:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/git/homelab/talos/talhelper
|
||||||
|
|
||||||
|
# Ensure SOPS age key is available
|
||||||
|
export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt
|
||||||
|
|
||||||
|
# Regenerate all configs
|
||||||
|
talhelper genconfig
|
||||||
|
|
||||||
|
# Apply to jungle-cruise only
|
||||||
|
talosctl apply-config \
|
||||||
|
--file clusterconfig/fastpass-jungle-cruise.yaml \
|
||||||
|
--nodes 10.1.71.69
|
||||||
|
|
||||||
|
# Monitor
|
||||||
|
kubectl get nodes -w
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Post-Recovery Steps
|
||||||
|
|
||||||
|
### 1. Verify Node is Healthy
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check node status
|
||||||
|
kubectl get nodes -o wide
|
||||||
|
# jungle-cruise should show Ready with INTERNAL-IP 10.1.71.69
|
||||||
|
|
||||||
|
# Verify system pods are running
|
||||||
|
kubectl get pods -n kube-system -o wide | grep jungle-cruise
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Deploy multipath.conf DaemonSet
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/git/homelab/talos/talhelper
|
||||||
|
|
||||||
|
# Deploy the DaemonSet that writes multipath.conf POST-boot
|
||||||
|
kubectl apply -f iscsi-multipath-init.yaml
|
||||||
|
|
||||||
|
# Verify it's running
|
||||||
|
kubectl get pods -n kube-system -l app=iscsi-multipath-init -o wide
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Verify iSCSI Configuration
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check multipath config was written
|
||||||
|
talosctl --nodes 10.1.71.69 read /etc/multipath.conf
|
||||||
|
|
||||||
|
# Verify kernel modules
|
||||||
|
talosctl --nodes 10.1.71.69 read /proc/modules | grep -E "iscsi|multipath"
|
||||||
|
|
||||||
|
# Check iscsid service
|
||||||
|
talosctl --nodes 10.1.71.69 service iscsid
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Test PX-CSI
|
||||||
|
|
||||||
|
Once multipath.conf is deployed via DaemonSet:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check PX-CSI node-plugin logs
|
||||||
|
kubectl logs -n portworx -l name=portworx-node -c node-plugin | grep multipath
|
||||||
|
|
||||||
|
# Should no longer see: "/etc/multipath.conf not found"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Why This Fix Works
|
||||||
|
|
||||||
|
### ❌ BROKEN: machine.files (Early Boot)
|
||||||
|
```yaml
|
||||||
|
worker:
|
||||||
|
patches:
|
||||||
|
- machine:
|
||||||
|
files: # Writes during early boot → FAILS on read-only FS
|
||||||
|
- path: /etc/multipath.conf
|
||||||
|
content: |
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
### ✅ FIXED: DaemonSet (Post-Boot)
|
||||||
|
```yaml
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: DaemonSet
|
||||||
|
metadata:
|
||||||
|
name: iscsi-multipath-init
|
||||||
|
namespace: kube-system
|
||||||
|
spec:
|
||||||
|
template:
|
||||||
|
spec:
|
||||||
|
initContainers:
|
||||||
|
- name: configure-multipath
|
||||||
|
command:
|
||||||
|
- sh
|
||||||
|
- -c
|
||||||
|
- |
|
||||||
|
# Writes AFTER boot when FS is fully writable
|
||||||
|
cat > /host/etc/multipath.conf <<'MPCONF'
|
||||||
|
...
|
||||||
|
MPCONF
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key Difference:**
|
||||||
|
- `machine.files` writes during early boot when `/etc` may be read-only
|
||||||
|
- DaemonSet writes after Kubernetes is up and filesystem is fully writable
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Files Modified
|
||||||
|
|
||||||
|
| File | Change | Commit |
|
||||||
|
|------|--------|--------|
|
||||||
|
| talconfig.yaml | Removed machine.files section | d2b6d95 |
|
||||||
|
| JUNGLE-CRUISE-RECOVERY.md | Created this document | d2b6d95 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- **Full Implementation Doc:** `IMPLEMENTATION_REPORT.md`
|
||||||
|
- **Quick Reference:** `QUICKREF.md`
|
||||||
|
- **DaemonSet:** `iscsi-multipath-init.yaml`
|
||||||
|
- **Verification Script:** `verify-iscsi.sh`
|
||||||
|
- **Previous Fix Commit:** e8303d5 "Fix Talos iSCSI configuration for Portworx CSI"
|
||||||
|
- **Broken Commit:** adc415e "Add multipath.conf for PX-CSI node driver"
|
||||||
|
- **Recovery Commit:** d2b6d95 "Revert multipath.conf from machine.files"
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Lessons Learned
|
||||||
|
|
||||||
|
1. **NEVER write files during Talos boot** - Use DaemonSets for post-boot configuration
|
||||||
|
2. **NEVER mount `/etc/iscsi`** - iscsi-tools extension manages it
|
||||||
|
3. **ALWAYS specify `nodeIP.validSubnets`** for dual-NIC setups
|
||||||
|
4. **Keep git history clean** - Easy rollback saved us here
|
||||||
|
5. **Test on one node first** - Should have tested DaemonSet approach before reverting
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Contact
|
||||||
|
|
||||||
|
**Issue Detected By:** Hermes Agent (carousel-of-progress)
|
||||||
|
**Date:** 2026-06-20 22:07 CDT
|
||||||
|
**Cluster:** fastpass (city-hall.local.mk-labs.cloud)
|
||||||
|
**Node:** jungle-cruise (10.1.71.69)
|
||||||
|
|
||||||
|
For questions or issues during recovery, refer to IMPLEMENTATION_REPORT.md or QUICKREF.md.
|
||||||
Reference in New Issue
Block a user