homelab/talos/talhelper/ISCSI_CONFIG_FIX.md

# Talos iSCSI Configuration Fix for Portworx CSI

## Root Cause Analysis

The `writeUserFiles failed` error occurred because:

1. **`/etc/iscsi` bind mount issue**: Talos creates `/var/lib/iscsi` for the iSCSI initiator data, but `/etc/iscsi` is part of the read-only system partition. The iscsi-tools extension manages initiator configuration automatically; explicitly mounting `/etc/iscsi` is unnecessary and causes boot failures.

2. **`/etc/multipath.conf` timing**: Writing files with `op: create` during early boot can fail if the target filesystem is not yet writable. Talos prefers system extensions to handle such configuration.

3. **Dual NIC ambiguity**: With two NICs (ens18 for management, ens19 for iSCSI), kubelet needs explicit `nodeIP` configuration to avoid selecting the wrong interface.

4. **Multipath daemon initialization**: The multipath.conf file needs to exist before multipathd starts, but Talos boot sequence is strict about when files can be written.

## Fixed Configuration

### Option 1: Minimal Safe Configuration (Recommended)

This configuration enables iSCSI without breaking boot:

```yaml
worker:
  schematic:
    customization:
      systemExtensions:
        officialExtensions:
          - siderolabs/qemu-guest-agent
          - siderolabs/util-linux-tools
          - siderolabs/iscsi-tools

  patches:
    - |-
      machine:
        # Load required kernel modules for iSCSI and multipath
        kernel:
          modules:
            - name: iscsi_tcp
            - name: dm_multipath
            - name: dm_round_robin

        # CRITICAL: Explicitly set nodeIP to primary network to avoid dual-NIC issues
        kubelet:
          nodeIP:
            validSubnets:
              - 10.1.71.0/24
          
          # Only mount /var/lib/iscsi (NOT /etc/iscsi)
          # The iscsi-tools extension manages /etc/iscsi automatically
          extraMounts:
            - destination: /var/lib/iscsi
              type: bind
              source: /var/lib/iscsi
              options:
                - bind
                - rshared
                - rw

        # ARP tuning for dual-NIC setup
        sysctls:
          net.ipv4.conf.all.arp_announce: "2"
          net.ipv4.conf.all.arp_ignore: "1"
```

**Why this works:**
- ✅ No `/etc/iscsi` mount (extension handles it)
- ✅ Only `/var/lib/iscsi` mounted (where iSCSI session data lives)
- ✅ Explicit `nodeIP` to avoid kubelet binding to iSCSI network
- ✅ No files written during boot (avoids writeUserFiles error)
- ✅ Pure Storage multipath can be configured post-boot via DaemonSet

### Option 2: With Multipath Configuration (Advanced)

If you need multipath.conf at boot time (only needed if you have EXISTING iSCSI volumes before Portworx deploys):

```yaml
worker:
  schematic:
    customization:
      systemExtensions:
        officialExtensions:
          - siderolabs/qemu-guest-agent
          - siderolabs/util-linux-tools
          - siderolabs/iscsi-tools

  patches:
    - |-
      machine:
        kernel:
          modules:
            - name: iscsi_tcp
            - name: dm_multipath
            - name: dm_round_robin

        kubelet:
          nodeIP:
            validSubnets:
              - 10.1.71.0/24
          extraMounts:
            - destination: /var/lib/iscsi
              type: bind
              source: /var/lib/iscsi
              options:
                - bind
                - rshared
                - rw

        sysctls:
          net.ipv4.conf.all.arp_announce: "2"
          net.ipv4.conf.all.arp_ignore: "1"
        
        # Use 'op: overwrite' instead of 'create' to avoid boot-time failures
        # Place in /var/ which is writable, then link if needed
        files:
          - content: |
              defaults {
                  polling_interval 10
                  find_multipaths yes
                  user_friendly_names no
              }
              devices {
                  device {
                      vendor "PURE"
                      product "FlashArray"
                      path_selector "service-time 0"
                      path_grouping_policy group_by_prio
                      prio alua
                      path_checker tur
                      fast_io_fail_tmo 10
                      user_friendly_names no
                      no_path_retry 0
                      hardware_handler "1 alua"
                      dev_loss_tmo 600
                      failback immediate
                  }
              }
              # Blacklist Portworx virtual devices
              blacklist {
                  devnode "^pxd[0-9]*"
              }
            path: /var/etc/multipath.conf
            permissions: 0644
            op: overwrite
```

**Note:** With this approach, you'd need a startup service or init container to symlink `/var/etc/multipath.conf` to `/etc/multipath.conf`. However, **Option 1 is safer and sufficient** for Portworx, which can configure multipath dynamically.

## Recommended Approach: Option 1 + Portworx DaemonSet Init

**Use Option 1 (minimal config) and let Portworx handle multipath configuration via DaemonSet init containers.**

Portworx CSI driver can deploy an init DaemonSet that:
1. Configures multipath.conf post-boot
2. Starts multipathd service
3. Handles Pure Storage-specific tuning

This is safer because:
- ✅ No boot-time file writing
- ✅ Configuration happens after filesystem is fully writable
- ✅ Can be updated without rebooting nodes
- ✅ Portworx team maintains the optimal multipath settings

## Application Steps

### 1. Apply the Fixed Configuration

```bash
cd ~/git/homelab/talos/talhelper
```

Edit `talconfig.yaml` and replace the worker section with **Option 1** above.

### 2. Regenerate Talos Configuration

```bash
talhelper genconfig
```

### 3. Apply to Worker Nodes (One at a Time)

```bash
# jungle-cruise
talosctl apply-config --file clusterconfig/fastpass-jungle-cruise.yaml --nodes 10.1.71.69

# Wait for node to reboot and come back online
kubectl wait --for=condition=Ready node/jungle-cruise --timeout=10m

# haunted-mansion
talosctl apply-config --file clusterconfig/fastpass-haunted-mansion.yaml --nodes 10.1.71.70
kubectl wait --for=condition=Ready node/haunted-mansion --timeout=10m

# peter-pans-flight
talosctl apply-config --file clusterconfig/fastpass-peter-pans-flight.yaml --nodes 10.1.71.71
kubectl wait --for=condition=Ready node/peter-pans-flight --timeout=10m
```

### 4. Verify iSCSI is Working

After each node reboots:

```bash
NODE_IP=10.1.71.69  # Change for each node

# Check iscsid service is running
talosctl -n $NODE_IP service iscsid

# Verify kernel modules loaded
talosctl -n $NODE_IP read /proc/modules | grep -E "iscsi_tcp|dm_multipath|dm_round_robin"

# Check initiator name is set (unique per node)
talosctl -n $NODE_IP read /etc/iscsi/initiatorname.iscsi

# Verify /var/lib/iscsi exists and is writable
talosctl -n $NODE_IP ls /var/lib/iscsi

# Check kubelet is using correct nodeIP
kubectl get node -o wide | grep jungle-cruise
# Should show 10.1.71.69 as INTERNAL-IP, NOT 10.1.75.69
```

### 5. Configure Multipath (Post-Boot)

Create a DaemonSet to configure multipath on all worker nodes:

```bash
cat > ~/git/homelab/talos/iscsi-multipath-init.yaml <<'EOF'
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: iscsi-multipath-init
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: iscsi-multipath-init
  template:
    metadata:
      labels:
        app: iscsi-multipath-init
    spec:
      hostNetwork: true
      hostPID: true
      nodeSelector:
        node-role.kubernetes.io/worker: ""
      initContainers:
      - name: configure-multipath
        image: alpine:3.18
        securityContext:
          privileged: true
        command:
        - sh
        - -c
        - |
          cat > /host/etc/multipath.conf <<'MPCONF'
          defaults {
              polling_interval 10
              find_multipaths yes
              user_friendly_names no
          }
          devices {
              device {
                  vendor "PURE"
                  product "FlashArray"
                  path_selector "service-time 0"
                  path_grouping_policy group_by_prio
                  prio alua
                  path_checker tur
                  fast_io_fail_tmo 10
                  user_friendly_names no
                  no_path_retry 0
                  hardware_handler "1 alua"
                  dev_loss_tmo 600
                  failback immediate
              }
          }
          blacklist {
              devnode "^pxd[0-9]*"
          }
          MPCONF
          
          echo "Multipath configuration applied"
          nsenter -t 1 -m -u -i -n -- multipath -ll || true
        volumeMounts:
        - name: host-etc
          mountPath: /host/etc
      containers:
      - name: pause
        image: registry.k8s.io/pause:3.9
      volumes:
      - name: host-etc
        hostPath:
          path: /etc
          type: Directory
EOF

kubectl apply -f ~/git/homelab/talos/iscsi-multipath-init.yaml
```

Wait for DaemonSet to run on all workers:

```bash
kubectl rollout status daemonset/iscsi-multipath-init -n kube-system
```

### 6. Verify Multipath Configuration

```bash
for node in 10.1.71.69 10.1.71.70 10.1.71.71; do
  echo "=== Checking $node ==="
  talosctl -n $node read /etc/multipath.conf
  talosctl -n $node exec -- multipath -ll
done
```

## Portworx-Specific Considerations

### 1. Ensure Portworx Uses Correct Network

Portworx should use the **iSCSI network (10.1.75.x)** for storage traffic. Configure this in the Portworx StorageCluster CR:

```yaml
apiVersion: core.libopenstorage.org/v1
kind: StorageCluster
metadata:
  name: px-cluster-fastpass
  namespace: portworx
spec:
  network:
    dataInterface: ens19     # iSCSI network
    mgmtInterface: ens18     # Management network
```

### 2. Node Labels for Storage Network

Label worker nodes to indicate iSCSI capability:

```bash
kubectl label node jungle-cruise storage-network=iscsi
kubectl label node haunted-mansion storage-network=iscsi
kubectl label node peter-pans-flight storage-network=iscsi
```

### 3. Verify Pure FlashArray Connectivity

From any worker node:

```bash
# Discover iSCSI targets (replace with your FlashArray iSCSI IP)
talosctl -n 10.1.71.69 exec -- iscsiadm -m discovery -t st -p <flasharray-iscsi-ip>

# Example with 10.1.75.100 as FlashArray iSCSI endpoint
talosctl -n 10.1.71.69 exec -- iscsiadm -m discovery -t st -p 10.1.75.100
```

Expected output:
```
10.1.75.100:3260,1 iqn.2010-06.com.purestorage:flasharray.xxxxx
10.1.75.101:3260,2 iqn.2010-06.com.purestorage:flasharray.xxxxx
```

## Troubleshooting

### If Boot Still Fails

1. **Remove all files sections** and use only Option 1
2. **Check Talos logs during boot:**
   ```bash
   talosctl -n <node-ip> dmesg | grep -i "write\|fail\|error"
   talosctl -n <node-ip> logs controller-runtime
   ```

### If Kubelet Uses Wrong IP

Check node internal IP:
```bash
kubectl get nodes -o wide
```

If showing 10.1.75.x instead of 10.1.71.x, the `nodeIP.validSubnets` didn't apply. Verify talconfig.yaml and regenerate.

### If iSCSI Sessions Don't Connect

```bash
# Check iscsid is running
talosctl -n <node-ip> service iscsid

# Check kernel modules
talosctl -n <node-ip> exec -- lsmod | grep iscsi_tcp

# Try manual discovery
talosctl -n <node-ip> exec -- iscsiadm -m discovery -t st -p <flasharray-ip>

# Check for firewall issues on ens19
talosctl -n <node-ip> exec -- ping <flasharray-ip>
```

## Summary of Changes

| Issue | Old Config | New Config |
|-------|-----------|-----------|
| `/etc/iscsi` mount | ✗ Mounted (causes boot failure) | ✓ Removed (extension manages it) |
| `/var/lib/iscsi` mount | ✓ Correct | ✓ Kept with `rw` option |
| `multipath.conf` | ✗ Written at boot with `op: create` | ✓ Applied post-boot via DaemonSet |
| Dual NIC handling | ✗ No nodeIP specified | ✓ `nodeIP.validSubnets` set to 10.1.71.0/24 |
| File operation | `op: create` | N/A (moved to DaemonSet) |

## Git Workflow

```bash
cd ~/git/homelab
git checkout -b fix/talos-iscsi-boot

# Edit talconfig.yaml with Option 1
# Create DaemonSet manifest

git add talos/talhelper/talconfig.yaml talos/iscsi-multipath-init.yaml
git commit -m "Fix Talos iSCSI configuration to prevent boot failures

- Remove /etc/iscsi mount (iscsi-tools extension manages it)
- Add kubelet nodeIP.validSubnets to fix dual-NIC node IP selection
- Move multipath.conf to post-boot DaemonSet initialization
- Add rw option to /var/lib/iscsi mount for session persistence

Fixes boot failure: 'writeUserFiles failed, rebooting in 35 minutes'"

git push origin fix/talos-iscsi-boot
```

After successful testing, merge to main.