Add jungle-cruise recovery documentation
Documents root cause analysis and recovery procedure for jungle-cruise node failure after applying multipath.conf via machine.files. Includes three recovery options depending on available credentials: - Apply fixed config (requires talosctl + existing configs) - Force reboot (quickest) - Full regeneration (requires SOPS keys)
This commit is contained in:
Hermes Agent service account
271
talos/talhelper/JUNGLE-CRUISE-RECOVERY.md
Normal file
271
talos/talhelper/JUNGLE-CRUISE-RECOVERY.md
Normal file
@@ -0,0 +1,271 @@
|
||||
# jungle-cruise Node Recovery Procedure
|
||||
|
||||
**Status:** Node NotReady since 2026-06-20 22:01 CDT
|
||||
**Root Cause:** multipath.conf in machine.files causes boot failure
|
||||
**Fix Applied:** Commit d2b6d95 - removed multipath.conf from machine.files
|
||||
|
||||
---
|
||||
|
||||
## Problem Summary
|
||||
|
||||
### What Happened
|
||||
|
||||
1. **21:56 CDT** - Commit adc415e added `/etc/multipath.conf` to `machine.files` section
|
||||
2. **~22:00 CDT** - Configuration applied to jungle-cruise
|
||||
3. **22:01 CDT** - jungle-cruise kubelet stopped posting status (Node → NotReady)
|
||||
|
||||
### Root Cause
|
||||
|
||||
Writing `/etc/multipath.conf` during Talos early boot via `machine.files` causes:
|
||||
```
|
||||
[talos] writeUserFiles failed: permission denied (read-only filesystem)
|
||||
[talos] rebooting in 35 minutes
|
||||
```
|
||||
|
||||
This is the SAME issue that was previously fixed in commit e8303d5 and documented in IMPLEMENTATION_REPORT.md.
|
||||
|
||||
### Why This Happened
|
||||
|
||||
The multipath.conf was correctly REMOVED in commit e8303d5 (with DaemonSet solution), but was inadvertently RE-ADDED in commit adc415e to fix PX-CSI node driver crash.
|
||||
|
||||
---
|
||||
|
||||
## Current State
|
||||
|
||||
### Node Status
|
||||
```bash
|
||||
$ kubectl get node jungle-cruise
|
||||
NAME STATUS ROLES AGE VERSION
|
||||
jungle-cruise NotReady worker 47h v1.32.3
|
||||
|
||||
$ kubectl describe node jungle-cruise | grep Ready
|
||||
Ready Unknown ... NodeStatusUnknown Kubelet stopped posting node status.
|
||||
```
|
||||
|
||||
### Network Status
|
||||
- ✅ Node is pingable (10.1.71.69)
|
||||
- ✅ Talos API port is open (50000/tcp)
|
||||
- ❌ SSH not available (Talos doesn't run SSH)
|
||||
- ❌ Kubelet not posting status since 03:01:12Z
|
||||
|
||||
### Pods on Node
|
||||
- All system pods (cilium, kube-proxy, etc.) are Pending
|
||||
- Cannot be scheduled due to Node NotReady
|
||||
|
||||
---
|
||||
|
||||
## Fix Applied
|
||||
|
||||
**Commit:** d2b6d95
|
||||
**Date:** 2026-06-20 22:07 CDT
|
||||
**Changes:** Removed `machine.files` section containing multipath.conf from talconfig.yaml
|
||||
|
||||
```diff
|
||||
- # Write multipath.conf for PX-CSI
|
||||
- files:
|
||||
- - content: |
|
||||
- defaults { ... }
|
||||
- devices { ... }
|
||||
- path: /etc/multipath.conf
|
||||
- permissions: 0644
|
||||
+ # (removed - use DaemonSet instead)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Recovery Procedure
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- [ ] Access to talosctl with valid talosconfig (from city-hall or control plane)
|
||||
- [ ] SOPS/age keys to decrypt talsecret.sops.yaml (if regenerating configs)
|
||||
- [ ] OR access to existing clusterconfig/ directory with pre-generated configs
|
||||
|
||||
### Option A: Apply Fixed Config (Preferred)
|
||||
|
||||
If you have existing clusterconfig/ or can regenerate:
|
||||
|
||||
```bash
|
||||
cd ~/git/homelab/talos/talhelper
|
||||
|
||||
# If clusterconfig/ doesn't exist, regenerate (requires SOPS keys)
|
||||
talhelper genconfig
|
||||
|
||||
# Apply the fixed configuration to jungle-cruise
|
||||
talosctl apply-config \
|
||||
--file clusterconfig/fastpass-jungle-cruise.yaml \
|
||||
--nodes 10.1.71.69
|
||||
|
||||
# Wait for node to reboot and become Ready
|
||||
kubectl wait --for=condition=Ready node/jungle-cruise --timeout=10m
|
||||
```
|
||||
|
||||
### Option B: Force Reboot (Quick Recovery)
|
||||
|
||||
If the node is stuck in a boot loop, a simple reboot might clear the bad state:
|
||||
|
||||
```bash
|
||||
# Via talosctl
|
||||
talosctl --nodes 10.1.71.69 reboot
|
||||
|
||||
# OR via Proxmox (if talosctl unavailable)
|
||||
# Find VM ID and reboot from Proxmox UI or CLI
|
||||
```
|
||||
|
||||
After reboot, the node should come back with its previous (working) configuration, since the bad config hasn't been permanently written.
|
||||
|
||||
### Option C: Full Config Regeneration
|
||||
|
||||
If clusterconfig/ is missing:
|
||||
|
||||
```bash
|
||||
cd ~/git/homelab/talos/talhelper
|
||||
|
||||
# Ensure SOPS age key is available
|
||||
export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt
|
||||
|
||||
# Regenerate all configs
|
||||
talhelper genconfig
|
||||
|
||||
# Apply to jungle-cruise only
|
||||
talosctl apply-config \
|
||||
--file clusterconfig/fastpass-jungle-cruise.yaml \
|
||||
--nodes 10.1.71.69
|
||||
|
||||
# Monitor
|
||||
kubectl get nodes -w
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Post-Recovery Steps
|
||||
|
||||
### 1. Verify Node is Healthy
|
||||
|
||||
```bash
|
||||
# Check node status
|
||||
kubectl get nodes -o wide
|
||||
# jungle-cruise should show Ready with INTERNAL-IP 10.1.71.69
|
||||
|
||||
# Verify system pods are running
|
||||
kubectl get pods -n kube-system -o wide | grep jungle-cruise
|
||||
```
|
||||
|
||||
### 2. Deploy multipath.conf DaemonSet
|
||||
|
||||
```bash
|
||||
cd ~/git/homelab/talos/talhelper
|
||||
|
||||
# Deploy the DaemonSet that writes multipath.conf POST-boot
|
||||
kubectl apply -f iscsi-multipath-init.yaml
|
||||
|
||||
# Verify it's running
|
||||
kubectl get pods -n kube-system -l app=iscsi-multipath-init -o wide
|
||||
```
|
||||
|
||||
### 3. Verify iSCSI Configuration
|
||||
|
||||
```bash
|
||||
# Check multipath config was written
|
||||
talosctl --nodes 10.1.71.69 read /etc/multipath.conf
|
||||
|
||||
# Verify kernel modules
|
||||
talosctl --nodes 10.1.71.69 read /proc/modules | grep -E "iscsi|multipath"
|
||||
|
||||
# Check iscsid service
|
||||
talosctl --nodes 10.1.71.69 service iscsid
|
||||
```
|
||||
|
||||
### 4. Test PX-CSI
|
||||
|
||||
Once multipath.conf is deployed via DaemonSet:
|
||||
|
||||
```bash
|
||||
# Check PX-CSI node-plugin logs
|
||||
kubectl logs -n portworx -l name=portworx-node -c node-plugin | grep multipath
|
||||
|
||||
# Should no longer see: "/etc/multipath.conf not found"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Why This Fix Works
|
||||
|
||||
### ❌ BROKEN: machine.files (Early Boot)
|
||||
```yaml
|
||||
worker:
|
||||
patches:
|
||||
- machine:
|
||||
files: # Writes during early boot → FAILS on read-only FS
|
||||
- path: /etc/multipath.conf
|
||||
content: |
|
||||
...
|
||||
```
|
||||
|
||||
### ✅ FIXED: DaemonSet (Post-Boot)
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: iscsi-multipath-init
|
||||
namespace: kube-system
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
initContainers:
|
||||
- name: configure-multipath
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
# Writes AFTER boot when FS is fully writable
|
||||
cat > /host/etc/multipath.conf <<'MPCONF'
|
||||
...
|
||||
MPCONF
|
||||
```
|
||||
|
||||
**Key Difference:**
|
||||
- `machine.files` writes during early boot when `/etc` may be read-only
|
||||
- DaemonSet writes after Kubernetes is up and filesystem is fully writable
|
||||
|
||||
---
|
||||
|
||||
## Files Modified
|
||||
|
||||
| File | Change | Commit |
|
||||
|------|--------|--------|
|
||||
| talconfig.yaml | Removed machine.files section | d2b6d95 |
|
||||
| JUNGLE-CRUISE-RECOVERY.md | Created this document | d2b6d95 |
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- **Full Implementation Doc:** `IMPLEMENTATION_REPORT.md`
|
||||
- **Quick Reference:** `QUICKREF.md`
|
||||
- **DaemonSet:** `iscsi-multipath-init.yaml`
|
||||
- **Verification Script:** `verify-iscsi.sh`
|
||||
- **Previous Fix Commit:** e8303d5 "Fix Talos iSCSI configuration for Portworx CSI"
|
||||
- **Broken Commit:** adc415e "Add multipath.conf for PX-CSI node driver"
|
||||
- **Recovery Commit:** d2b6d95 "Revert multipath.conf from machine.files"
|
||||
|
||||
---
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
1. **NEVER write files during Talos boot** - Use DaemonSets for post-boot configuration
|
||||
2. **NEVER mount `/etc/iscsi`** - iscsi-tools extension manages it
|
||||
3. **ALWAYS specify `nodeIP.validSubnets`** for dual-NIC setups
|
||||
4. **Keep git history clean** - Easy rollback saved us here
|
||||
5. **Test on one node first** - Should have tested DaemonSet approach before reverting
|
||||
|
||||
---
|
||||
|
||||
## Contact
|
||||
|
||||
**Issue Detected By:** Hermes Agent (carousel-of-progress)
|
||||
**Date:** 2026-06-20 22:07 CDT
|
||||
**Cluster:** fastpass (city-hall.local.mk-labs.cloud)
|
||||
**Node:** jungle-cruise (10.1.71.69)
|
||||
|
||||
For questions or issues during recovery, refer to IMPLEMENTATION_REPORT.md or QUICKREF.md.
|
||||
Reference in New Issue
Block a user