homelab/talos/talhelper/ISCSI_CONFIG_FIX.md

Talos iSCSI Configuration Fix for Portworx CSI

Root Cause Analysis

The writeUserFiles failed error occurred because:

1. /etc/iscsi bind mount issue: Talos creates /var/lib/iscsi for the iSCSI initiator data, but /etc/iscsi is part of the read-only system partition. The iscsi-tools extension manages initiator configuration automatically; explicitly mounting /etc/iscsi is unnecessary and causes boot failures.

2. /etc/multipath.conf timing: Writing files with op: create during early boot can fail if the target filesystem is not yet writable. Talos prefers system extensions to handle such configuration.

3. Dual NIC ambiguity: With two NICs (ens18 for management, ens19 for iSCSI), kubelet needs explicit nodeIP configuration to avoid selecting the wrong interface.

4. Multipath daemon initialization: The multipath.conf file needs to exist before multipathd starts, but Talos boot sequence is strict about when files can be written.

Fixed Configuration

Option 1: Minimal Safe Configuration (Recommended)

This configuration enables iSCSI without breaking boot:

worker:
  schematic:
    customization:
      systemExtensions:
        officialExtensions:
          - siderolabs/qemu-guest-agent
          - siderolabs/util-linux-tools
          - siderolabs/iscsi-tools

  patches:
    - |-
      machine:
        # Load required kernel modules for iSCSI and multipath
        kernel:
          modules:
            - name: iscsi_tcp
            - name: dm_multipath
            - name: dm_round_robin

        # CRITICAL: Explicitly set nodeIP to primary network to avoid dual-NIC issues
        kubelet:
          nodeIP:
            validSubnets:
              - 10.1.71.0/24
          
          # Only mount /var/lib/iscsi (NOT /etc/iscsi)
          # The iscsi-tools extension manages /etc/iscsi automatically
          extraMounts:
            - destination: /var/lib/iscsi
              type: bind
              source: /var/lib/iscsi
              options:
                - bind
                - rshared
                - rw

        # ARP tuning for dual-NIC setup
        sysctls:
          net.ipv4.conf.all.arp_announce: "2"
          net.ipv4.conf.all.arp_ignore: "1"

Why this works:

• ✅ No /etc/iscsi mount (extension handles it)
• ✅ Only /var/lib/iscsi mounted (where iSCSI session data lives)
• ✅ Explicit nodeIP to avoid kubelet binding to iSCSI network
• ✅ No files written during boot (avoids writeUserFiles error)
• ✅ Pure Storage multipath can be configured post-boot via DaemonSet

Option 2: With Multipath Configuration (Advanced)

If you need multipath.conf at boot time (only needed if you have EXISTING iSCSI volumes before Portworx deploys):

worker:
  schematic:
    customization:
      systemExtensions:
        officialExtensions:
          - siderolabs/qemu-guest-agent
          - siderolabs/util-linux-tools
          - siderolabs/iscsi-tools

  patches:
    - |-
      machine:
        kernel:
          modules:
            - name: iscsi_tcp
            - name: dm_multipath
            - name: dm_round_robin

        kubelet:
          nodeIP:
            validSubnets:
              - 10.1.71.0/24
          extraMounts:
            - destination: /var/lib/iscsi
              type: bind
              source: /var/lib/iscsi
              options:
                - bind
                - rshared
                - rw

        sysctls:
          net.ipv4.conf.all.arp_announce: "2"
          net.ipv4.conf.all.arp_ignore: "1"
        
        # Use 'op: overwrite' instead of 'create' to avoid boot-time failures
        # Place in /var/ which is writable, then link if needed
        files:
          - content: |
              defaults {
                  polling_interval 10
                  find_multipaths yes
                  user_friendly_names no
              }
              devices {
                  device {
                      vendor "PURE"
                      product "FlashArray"
                      path_selector "service-time 0"
                      path_grouping_policy group_by_prio
                      prio alua
                      path_checker tur
                      fast_io_fail_tmo 10
                      user_friendly_names no
                      no_path_retry 0
                      hardware_handler "1 alua"
                      dev_loss_tmo 600
                      failback immediate
                  }
              }
              # Blacklist Portworx virtual devices
              blacklist {
                  devnode "^pxd[0-9]*"
              }
            path: /var/etc/multipath.conf
            permissions: 0644
            op: overwrite

Note: With this approach, you'd need a startup service or init container to symlink /var/etc/multipath.conf to /etc/multipath.conf. However, Option 1 is safer and sufficient for Portworx, which can configure multipath dynamically.

Recommended Approach: Option 1 + Portworx DaemonSet Init

Use Option 1 (minimal config) and let Portworx handle multipath configuration via DaemonSet init containers.

Portworx CSI driver can deploy an init DaemonSet that:

1. Configures multipath.conf post-boot
2. Starts multipathd service
3. Handles Pure Storage-specific tuning

This is safer because:

• ✅ No boot-time file writing
• ✅ Configuration happens after filesystem is fully writable
• ✅ Can be updated without rebooting nodes
• ✅ Portworx team maintains the optimal multipath settings

Application Steps

1. Apply the Fixed Configuration

cd ~/git/homelab/talos/talhelper

Edit talconfig.yaml and replace the worker section with Option 1 above.

2. Regenerate Talos Configuration

talhelper genconfig

3. Apply to Worker Nodes (One at a Time)

# jungle-cruise
talosctl apply-config --file clusterconfig/fastpass-jungle-cruise.yaml --nodes 10.1.71.69

# Wait for node to reboot and come back online
kubectl wait --for=condition=Ready node/jungle-cruise --timeout=10m

# haunted-mansion
talosctl apply-config --file clusterconfig/fastpass-haunted-mansion.yaml --nodes 10.1.71.70
kubectl wait --for=condition=Ready node/haunted-mansion --timeout=10m

# peter-pans-flight
talosctl apply-config --file clusterconfig/fastpass-peter-pans-flight.yaml --nodes 10.1.71.71
kubectl wait --for=condition=Ready node/peter-pans-flight --timeout=10m

4. Verify iSCSI is Working

After each node reboots:

NODE_IP=10.1.71.69  # Change for each node

# Check iscsid service is running
talosctl -n $NODE_IP service iscsid

# Verify kernel modules loaded
talosctl -n $NODE_IP read /proc/modules | grep -E "iscsi_tcp|dm_multipath|dm_round_robin"

# Check initiator name is set (unique per node)
talosctl -n $NODE_IP read /etc/iscsi/initiatorname.iscsi

# Verify /var/lib/iscsi exists and is writable
talosctl -n $NODE_IP ls /var/lib/iscsi

# Check kubelet is using correct nodeIP
kubectl get node -o wide | grep jungle-cruise
# Should show 10.1.71.69 as INTERNAL-IP, NOT 10.1.75.69

5. Configure Multipath (Post-Boot)

Create a DaemonSet to configure multipath on all worker nodes:

cat > ~/git/homelab/talos/iscsi-multipath-init.yaml <<'EOF'
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: iscsi-multipath-init
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: iscsi-multipath-init
  template:
    metadata:
      labels:
        app: iscsi-multipath-init
    spec:
      hostNetwork: true
      hostPID: true
      nodeSelector:
        node-role.kubernetes.io/worker: ""
      initContainers:
      - name: configure-multipath
        image: alpine:3.18
        securityContext:
          privileged: true
        command:
        - sh
        - -c
        - |
          cat > /host/etc/multipath.conf <<'MPCONF'
          defaults {
              polling_interval 10
              find_multipaths yes
              user_friendly_names no
          }
          devices {
              device {
                  vendor "PURE"
                  product "FlashArray"
                  path_selector "service-time 0"
                  path_grouping_policy group_by_prio
                  prio alua
                  path_checker tur
                  fast_io_fail_tmo 10
                  user_friendly_names no
                  no_path_retry 0
                  hardware_handler "1 alua"
                  dev_loss_tmo 600
                  failback immediate
              }
          }
          blacklist {
              devnode "^pxd[0-9]*"
          }
          MPCONF
          
          echo "Multipath configuration applied"
          nsenter -t 1 -m -u -i -n -- multipath -ll || true
        volumeMounts:
        - name: host-etc
          mountPath: /host/etc
      containers:
      - name: pause
        image: registry.k8s.io/pause:3.9
      volumes:
      - name: host-etc
        hostPath:
          path: /etc
          type: Directory
EOF

kubectl apply -f ~/git/homelab/talos/iscsi-multipath-init.yaml

Wait for DaemonSet to run on all workers:

kubectl rollout status daemonset/iscsi-multipath-init -n kube-system

6. Verify Multipath Configuration

for node in 10.1.71.69 10.1.71.70 10.1.71.71; do
  echo "=== Checking $node ==="
  talosctl -n $node read /etc/multipath.conf
  talosctl -n $node exec -- multipath -ll
done

Portworx-Specific Considerations

1. Ensure Portworx Uses Correct Network

Portworx should use the iSCSI network (10.1.75.x) for storage traffic. Configure this in the Portworx StorageCluster CR:

apiVersion: core.libopenstorage.org/v1
kind: StorageCluster
metadata:
  name: px-cluster-fastpass
  namespace: portworx
spec:
  network:
    dataInterface: ens19     # iSCSI network
    mgmtInterface: ens18     # Management network

2. Node Labels for Storage Network

Label worker nodes to indicate iSCSI capability:

kubectl label node jungle-cruise storage-network=iscsi
kubectl label node haunted-mansion storage-network=iscsi
kubectl label node peter-pans-flight storage-network=iscsi

3. Verify Pure FlashArray Connectivity

From any worker node:

# Discover iSCSI targets (replace with your FlashArray iSCSI IP)
talosctl -n 10.1.71.69 exec -- iscsiadm -m discovery -t st -p <flasharray-iscsi-ip>

# Example with 10.1.75.100 as FlashArray iSCSI endpoint
talosctl -n 10.1.71.69 exec -- iscsiadm -m discovery -t st -p 10.1.75.100

Expected output:

10.1.75.100:3260,1 iqn.2010-06.com.purestorage:flasharray.xxxxx
10.1.75.101:3260,2 iqn.2010-06.com.purestorage:flasharray.xxxxx

Troubleshooting

If Boot Still Fails

1. Remove all files sections and use only Option 1
2. Check Talos logs during boot:

   talosctl -n <node-ip> dmesg | grep -i "write\|fail\|error"
   talosctl -n <node-ip> logs controller-runtime
   

If Kubelet Uses Wrong IP

Check node internal IP:

kubectl get nodes -o wide

If showing 10.1.75.x instead of 10.1.71.x, the nodeIP.validSubnets didn't apply. Verify talconfig.yaml and regenerate.

If iSCSI Sessions Don't Connect

# Check iscsid is running
talosctl -n <node-ip> service iscsid

# Check kernel modules
talosctl -n <node-ip> exec -- lsmod | grep iscsi_tcp

# Try manual discovery
talosctl -n <node-ip> exec -- iscsiadm -m discovery -t st -p <flasharray-ip>

# Check for firewall issues on ens19
talosctl -n <node-ip> exec -- ping <flasharray-ip>

Summary of Changes

Issue	Old Config	New Config
/etc/iscsi mount	✗ Mounted (causes boot failure)	✓ Removed (extension manages it)
/var/lib/iscsi mount	✓ Correct	✓ Kept with rw option
multipath.conf	✗ Written at boot with op: create	✓ Applied post-boot via DaemonSet
Dual NIC handling	✗ No nodeIP specified	✓ nodeIP.validSubnets set to 10.1.71.0/24
File operation	op: create	N/A (moved to DaemonSet)

Git Workflow

cd ~/git/homelab
git checkout -b fix/talos-iscsi-boot

# Edit talconfig.yaml with Option 1
# Create DaemonSet manifest

git add talos/talhelper/talconfig.yaml talos/iscsi-multipath-init.yaml
git commit -m "Fix Talos iSCSI configuration to prevent boot failures

- Remove /etc/iscsi mount (iscsi-tools extension manages it)
- Add kubelet nodeIP.validSubnets to fix dual-NIC node IP selection
- Move multipath.conf to post-boot DaemonSet initialization
- Add rw option to /var/lib/iscsi mount for session persistence

Fixes boot failure: 'writeUserFiles failed, rebooting in 35 minutes'"

git push origin fix/talos-iscsi-boot

After successful testing, merge to main.