Successfully writes /system/etc/multipath.conf but cannot write to /etc
due to Talos read-only filesystem restrictions.
Attempts made:
- nsenter with sh/cat/ln - commands don't exist in Talos minimal env
- Mount /proc/1/root/etc - still read-only
- Bind mount - invalid argument
Blocker: Talos /etc is truly read-only post-boot. PX-CSI also fails
with same nsenter/command issues when trying to validate multipath.conf.
Next: Investigate PX-CSI configuration options or Talos machine config alternatives.
Use nsenter to write multipath.conf in host's mount namespace instead
of trying to write to /host/etc which is read-only in containers.
Talos mounts /etc as read-only in container namespaces but allows writes
in the host mount namespace. This fix uses nsenter to access PID 1's
mount namespace where /etc is writable.
Also removed unnecessary volumeMounts and volumes since we're using
nsenter instead of hostPath mounts.
Fixes: Init:Error - 'can't create /host/etc/multipath.conf: Read-only file system'
Comprehensive summary of diagnosis, fix, and recovery status.
Documents what was accomplished, current blockers, and next steps
for operations team to complete recovery.
Documents root cause analysis and recovery procedure for jungle-cruise
node failure after applying multipath.conf via machine.files.
Includes three recovery options depending on available credentials:
- Apply fixed config (requires talosctl + existing configs)
- Force reboot (quickest)
- Full regeneration (requires SOPS keys)
Removes /etc/multipath.conf from machine.files section which causes
jungle-cruise boot failure. This reverts the problematic change from
commit adc415e.
Root cause: Writing /etc/multipath.conf during early boot via machine.files
causes writeUserFiles to fail on read-only filesystem.
Solution: Use DaemonSet (iscsi-multipath-init.yaml) to write multipath.conf
after boot when filesystem is fully writable.
Fixes: jungle-cruise NotReady status (kubelet stopped posting)
PX-CSI requires /etc/multipath.conf to exist on nodes.
Adding Pure Storage FlashArray multipath configuration via Talos machine files.
This fixes node-plugin crash: '/etc/multipath.conf not found'
- Add dm_round_robin kernel module for Pure Storage multipath
- Uncomment and enable /etc/multipath.conf with Pure-specific settings
- Add apply-iscsi-config.sh script for rolling worker node updates
- Add comprehensive UPGRADES-AND-EXTENSIONS.md guide covering:
- System extensions via schematics and Image Factory
- Talos version upgrade procedures (control plane + workers)
- Kubernetes version upgrades
- Rolling upgrade best practices
- Troubleshooting common upgrade issues
- Add rolling-upgrade-workers.sh script for automated worker upgrades
- Includes safe wait times and confirmation prompts
