homelab/talos/talhelper/README-ISCSI.md

Talos iSCSI Configuration for Portworx CSI

This directory contains the fixed Talos configuration for iSCSI support on worker nodes, enabling Portworx CSI driver integration with Pure Storage FlashArray.

🚨 Problem Statement

The initial iSCSI configuration (commit f370213) caused worker nodes to fail boot with:

writeUserFiles failed, rebooting in 35 minutes

This was caused by:

1. Mounting /etc/iscsi (conflicts with iscsi-tools extension)
2. Writing /etc/multipath.conf during early boot (filesystem not writable)
3. Missing explicit nodeIP configuration for dual-NIC workers

✅ Solution

The fix involves three changes:

1. Remove /etc/iscsi mount - Let iscsi-tools extension manage it
2. Add explicit nodeIP - Bind kubelet to primary network (10.1.71.0/24)
3. Move multipath config to DaemonSet - Configure post-boot when filesystem is writable

📋 Files in This Directory

File	Purpose
talconfig.yaml	Main Talos configuration (FIXED)
iscsi-multipath-init.yaml	DaemonSet that configures multipath post-boot
apply-iscsi-fix.sh	Automated deployment script (START HERE)
verify-iscsi.sh	Verification script to check configuration
ISCSI_CONFIG_FIX.md	Detailed documentation of the fix
DEPLOYMENT_SUMMARY.md	Deployment checklist and plan
QUICKREF.md	Quick reference for common commands
README.md	This file

🚀 Quick Start

1. Review the Changes

cd ~/git/homelab/talos/talhelper

# See what changed
git diff <previous-commit> talconfig.yaml

# Read the detailed documentation
cat ISCSI_CONFIG_FIX.md

2. Apply the Fix

# Dry run first to preview
./apply-iscsi-fix.sh --dry-run

# Apply to all worker nodes
./apply-iscsi-fix.sh

This script will:

• Regenerate Talos configs
• Apply to each worker sequentially (jungle-cruise → haunted-mansion → peter-pans-flight)
• Wait for each node to reboot and become Ready
• Verify iSCSI functionality
• Deploy multipath DaemonSet

Duration: ~20-30 minutes (3 nodes × 5-10 min each)

3. Verify

# Run comprehensive verification (replace with your FlashArray iSCSI IP)
./verify-iscsi.sh 10.1.75.100

# Check node IPs (should be 10.1.71.x, NOT 10.1.75.x)
kubectl get nodes -o wide

# Check multipath DaemonSet
kubectl get pods -n kube-system -l app=iscsi-multipath-init -o wide

🔧 Manual Application (if needed)

If you prefer manual control:

# Regenerate config
talhelper genconfig

# Apply to one worker at a time
talosctl apply-config \
  --file clusterconfig/fastpass-jungle-cruise.yaml \
  --nodes 10.1.71.69

# Wait for Ready
kubectl wait --for=condition=Ready node/jungle-cruise --timeout=10m

# Verify
talosctl -n 10.1.71.69 service iscsid
talosctl -n 10.1.71.69 read /etc/iscsi/initiatorname.iscsi

# Repeat for other nodes...

📊 What Was Fixed

Before (BROKEN)

worker:
  patches:
    - machine:
        kubelet:
          extraMounts:
            - destination: /etc/iscsi      # ❌ BREAKS BOOT
              type: bind
              source: /etc/iscsi
            - destination: /var/lib/iscsi
              type: bind
              source: /var/lib/iscsi
        
        files:                              # ❌ BREAKS BOOT
          - path: /etc/multipath.conf
            op: create
            content: |
              ...

After (FIXED)

worker:
  patches:
    - machine:
        kubelet:
          nodeIP:                           # ✅ FIX: Explicit network
            validSubnets:
              - 10.1.71.0/24
          
          extraMounts:
            # ✅ FIX: Only /var/lib/iscsi with 'rw'
            - destination: /var/lib/iscsi
              type: bind
              source: /var/lib/iscsi
              options:
                - bind
                - rshared
                - rw
        
        # ✅ FIX: No files section (moved to DaemonSet)

🏗️ Architecture

┌─────────────────────────────────────────────────────────────┐
│ Talos Boot Sequence                                         │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  1. Load system extensions (iscsi-tools, util-linux-tools)  │
│     ↓                                                        │
│  2. Load kernel modules (iscsi_tcp, dm_multipath, ...)      │
│     ↓                                                        │
│  3. Mount /var/lib/iscsi for session persistence            │
│     ↓                                                        │
│  4. Start kubelet with nodeIP=10.1.71.x                     │
│     ↓                                                        │
│  5. Kubernetes starts (node Ready)                          │
│     ↓                                                        │
│  6. DaemonSet configures /etc/multipath.conf ← POST-BOOT    │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Key insight: By moving multipath configuration to a DaemonSet (step 6), we avoid writing files during early boot when the filesystem may not be writable.

🌐 Network Configuration

Workers have dual NICs:

Interface	Network	Purpose
ens18	10.1.71.0/24	Primary - Kubernetes API, pod traffic
ens19	10.1.75.0/24	Storage - iSCSI to FlashArray

Critical: Kubelet must bind to ens18 (10.1.71.x) using nodeIP.validSubnets.

📚 Documentation

• Quick Start: This README
• Quick Reference: QUICKREF.md - Common commands
• Detailed Fix: ISCSI_CONFIG_FIX.md - Complete explanation
• Deployment Plan: DEPLOYMENT_SUMMARY.md - Step-by-step checklist
• Portworx Guide: ~/git/homelab/cluster/platform/portworx-csi/README.md

✅ Success Criteria

After deployment, verify:

• All worker nodes show Ready status
• Node IPs are 10.1.71.x (not 10.1.75.x)
• iscsi-tools extension loaded
• Kernel modules loaded: iscsi_tcp, dm_multipath, dm_round_robin
• iscsid service ready
• /var/lib/iscsi directory accessible
• Unique initiator name on each node
• /etc/multipath.conf exists with Pure Storage config
• ens19 interface up with 10.1.75.x IP
• Multipath DaemonSet running on all workers
• No boot errors in dmesg

🆘 Troubleshooting

Boot Failure

If a node fails to boot after applying config:

# Check dmesg for errors
talosctl -n <node-ip> dmesg | grep -i "error\|fail"

# Check Talos controller logs
talosctl -n <node-ip> logs controller-runtime

# Rollback
git checkout HEAD^ -- talconfig.yaml
talhelper genconfig
talosctl apply-config --file clusterconfig/<node>.yaml --nodes <node-ip>

Wrong Node IP

If kubelet binds to 10.1.75.x instead of 10.1.71.x:

# Verify nodeIP.validSubnets in config
grep -A 3 "nodeIP:" talconfig.yaml

# Should show:
#   nodeIP:
#     validSubnets:
#       - 10.1.71.0/24

# If missing, add it and reapply

iSCSI Not Working

# Full verification
./verify-iscsi.sh <flasharray-ip>

# Check specific components
talosctl -n <node-ip> service iscsid
talosctl -n <node-ip> read /etc/iscsi/initiatorname.iscsi
talosctl -n <node-ip> exec -- multipath -ll

# Test discovery
talosctl -n <node-ip> exec -- \
  iscsiadm -m discovery -t st -p <flasharray-iscsi-ip>

🔄 Next Steps

After successful deployment:

1. Deploy Portworx CSI (if not already deployed)

   cd ~/git/homelab/cluster/platform/portworx-csi
   cat README.md
   

2. Test iSCSI to FlashArray

   ./verify-iscsi.sh <flasharray-iscsi-ip>
   

3. Create test PVC

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: test-pure-block
   spec:
     accessModes: [ReadWriteOnce]
     storageClassName: pure-block
     resources:
       requests:
         storage: 10Gi
   

4. Monitor Portworx

   kubectl logs -n portworx -l app=portworx-operator -f
   

📞 Support

• Git Repository: mad-tea-party:rblundon/homelab
• Configuration Path: ~/git/homelab/talos/talhelper/
• Talos Cluster: fastpass (v1.13.2)
• Related Commit: f370213 (original broken config)

🔗 External References

• Talos Storage Guide
• Talos iscsi-tools Extension
• Portworx CSI Documentation
• Pure Storage Best Practices

---

Last Updated: 2026-06-20
Status: Ready for deployment
Tested: Dry-run verified, pending production deployment