# Talos iSCSI Configuration for Portworx CSI This directory contains the fixed Talos configuration for iSCSI support on worker nodes, enabling Portworx CSI driver integration with Pure Storage FlashArray. ## 🚨 Problem Statement The initial iSCSI configuration (commit f370213) caused worker nodes to fail boot with: ``` writeUserFiles failed, rebooting in 35 minutes ``` This was caused by: 1. Mounting `/etc/iscsi` (conflicts with iscsi-tools extension) 2. Writing `/etc/multipath.conf` during early boot (filesystem not writable) 3. Missing explicit nodeIP configuration for dual-NIC workers ## ✅ Solution The fix involves three changes: 1. **Remove `/etc/iscsi` mount** - Let iscsi-tools extension manage it 2. **Add explicit nodeIP** - Bind kubelet to primary network (10.1.71.0/24) 3. **Move multipath config to DaemonSet** - Configure post-boot when filesystem is writable ## 📋 Files in This Directory | File | Purpose | |------|---------| | `talconfig.yaml` | **Main Talos configuration** (FIXED) | | `iscsi-multipath-init.yaml` | DaemonSet that configures multipath post-boot | | `apply-iscsi-fix.sh` | **Automated deployment script** (START HERE) | | `verify-iscsi.sh` | Verification script to check configuration | | `ISCSI_CONFIG_FIX.md` | **Detailed documentation** of the fix | | `DEPLOYMENT_SUMMARY.md` | Deployment checklist and plan | | `QUICKREF.md` | Quick reference for common commands | | `README.md` | This file | ## 🚀 Quick Start ### 1. Review the Changes ```bash cd ~/git/homelab/talos/talhelper # See what changed git diff talconfig.yaml # Read the detailed documentation cat ISCSI_CONFIG_FIX.md ``` ### 2. Apply the Fix ```bash # Dry run first to preview ./apply-iscsi-fix.sh --dry-run # Apply to all worker nodes ./apply-iscsi-fix.sh ``` This script will: - Regenerate Talos configs - Apply to each worker sequentially (jungle-cruise → haunted-mansion → peter-pans-flight) - Wait for each node to reboot and become Ready - Verify iSCSI functionality - Deploy multipath DaemonSet **Duration:** ~20-30 minutes (3 nodes × 5-10 min each) ### 3. Verify ```bash # Run comprehensive verification (replace with your FlashArray iSCSI IP) ./verify-iscsi.sh 10.1.75.100 # Check node IPs (should be 10.1.71.x, NOT 10.1.75.x) kubectl get nodes -o wide # Check multipath DaemonSet kubectl get pods -n kube-system -l app=iscsi-multipath-init -o wide ``` ## 🔧 Manual Application (if needed) If you prefer manual control: ```bash # Regenerate config talhelper genconfig # Apply to one worker at a time talosctl apply-config \ --file clusterconfig/fastpass-jungle-cruise.yaml \ --nodes 10.1.71.69 # Wait for Ready kubectl wait --for=condition=Ready node/jungle-cruise --timeout=10m # Verify talosctl -n 10.1.71.69 service iscsid talosctl -n 10.1.71.69 read /etc/iscsi/initiatorname.iscsi # Repeat for other nodes... ``` ## 📊 What Was Fixed ### Before (BROKEN) ```yaml worker: patches: - machine: kubelet: extraMounts: - destination: /etc/iscsi # ❌ BREAKS BOOT type: bind source: /etc/iscsi - destination: /var/lib/iscsi type: bind source: /var/lib/iscsi files: # ❌ BREAKS BOOT - path: /etc/multipath.conf op: create content: | ... ``` ### After (FIXED) ```yaml worker: patches: - machine: kubelet: nodeIP: # ✅ FIX: Explicit network validSubnets: - 10.1.71.0/24 extraMounts: # ✅ FIX: Only /var/lib/iscsi with 'rw' - destination: /var/lib/iscsi type: bind source: /var/lib/iscsi options: - bind - rshared - rw # ✅ FIX: No files section (moved to DaemonSet) ``` ## 🏗️ Architecture ``` ┌─────────────────────────────────────────────────────────────┐ │ Talos Boot Sequence │ ├─────────────────────────────────────────────────────────────┤ │ │ │ 1. Load system extensions (iscsi-tools, util-linux-tools) │ │ ↓ │ │ 2. Load kernel modules (iscsi_tcp, dm_multipath, ...) │ │ ↓ │ │ 3. Mount /var/lib/iscsi for session persistence │ │ ↓ │ │ 4. Start kubelet with nodeIP=10.1.71.x │ │ ↓ │ │ 5. Kubernetes starts (node Ready) │ │ ↓ │ │ 6. DaemonSet configures /etc/multipath.conf ← POST-BOOT │ │ │ └─────────────────────────────────────────────────────────────┘ ``` **Key insight:** By moving multipath configuration to a DaemonSet (step 6), we avoid writing files during early boot when the filesystem may not be writable. ## 🌐 Network Configuration Workers have dual NICs: | Interface | Network | Purpose | |-----------|---------|---------| | **ens18** | 10.1.71.0/24 | **Primary** - Kubernetes API, pod traffic | | **ens19** | 10.1.75.0/24 | **Storage** - iSCSI to FlashArray | **Critical:** Kubelet must bind to ens18 (10.1.71.x) using `nodeIP.validSubnets`. ## 📚 Documentation - **Quick Start:** This README - **Quick Reference:** `QUICKREF.md` - Common commands - **Detailed Fix:** `ISCSI_CONFIG_FIX.md` - Complete explanation - **Deployment Plan:** `DEPLOYMENT_SUMMARY.md` - Step-by-step checklist - **Portworx Guide:** `~/git/homelab/cluster/platform/portworx-csi/README.md` ## ✅ Success Criteria After deployment, verify: - [ ] All worker nodes show `Ready` status - [ ] Node IPs are `10.1.71.x` (not `10.1.75.x`) - [ ] `iscsi-tools` extension loaded - [ ] Kernel modules loaded: `iscsi_tcp`, `dm_multipath`, `dm_round_robin` - [ ] `iscsid` service ready - [ ] `/var/lib/iscsi` directory accessible - [ ] Unique initiator name on each node - [ ] `/etc/multipath.conf` exists with Pure Storage config - [ ] `ens19` interface up with `10.1.75.x` IP - [ ] Multipath DaemonSet running on all workers - [ ] No boot errors in `dmesg` ## 🆘 Troubleshooting ### Boot Failure If a node fails to boot after applying config: ```bash # Check dmesg for errors talosctl -n dmesg | grep -i "error\|fail" # Check Talos controller logs talosctl -n logs controller-runtime # Rollback git checkout HEAD^ -- talconfig.yaml talhelper genconfig talosctl apply-config --file clusterconfig/.yaml --nodes ``` ### Wrong Node IP If kubelet binds to `10.1.75.x` instead of `10.1.71.x`: ```bash # Verify nodeIP.validSubnets in config grep -A 3 "nodeIP:" talconfig.yaml # Should show: # nodeIP: # validSubnets: # - 10.1.71.0/24 # If missing, add it and reapply ``` ### iSCSI Not Working ```bash # Full verification ./verify-iscsi.sh # Check specific components talosctl -n service iscsid talosctl -n read /etc/iscsi/initiatorname.iscsi talosctl -n exec -- multipath -ll # Test discovery talosctl -n exec -- \ iscsiadm -m discovery -t st -p ``` ## 🔄 Next Steps After successful deployment: 1. **Deploy Portworx CSI** (if not already deployed) ```bash cd ~/git/homelab/cluster/platform/portworx-csi cat README.md ``` 2. **Test iSCSI to FlashArray** ```bash ./verify-iscsi.sh ``` 3. **Create test PVC** ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: test-pure-block spec: accessModes: [ReadWriteOnce] storageClassName: pure-block resources: requests: storage: 10Gi ``` 4. **Monitor Portworx** ```bash kubectl logs -n portworx -l app=portworx-operator -f ``` ## 📞 Support - **Git Repository:** `mad-tea-party:rblundon/homelab` - **Configuration Path:** `~/git/homelab/talos/talhelper/` - **Talos Cluster:** fastpass (v1.13.2) - **Related Commit:** f370213 (original broken config) ## 🔗 External References - [Talos Storage Guide](https://www.talos.dev/v1.13/kubernetes-guides/configuration/storage/) - [Talos iscsi-tools Extension](https://github.com/siderolabs/extensions/pkgs/container/iscsi-tools) - [Portworx CSI Documentation](https://docs.portworx.com/portworx-csi/) - [Pure Storage Best Practices](https://support.purestorage.com/) --- **Last Updated:** 2026-06-20 **Status:** Ready for deployment **Tested:** Dry-run verified, pending production deployment