homelab/talos/talhelper/README-ISCSI.md

# Talos iSCSI Configuration for Portworx CSI

This directory contains the fixed Talos configuration for iSCSI support on worker nodes, enabling Portworx CSI driver integration with Pure Storage FlashArray.

## 🚨 Problem Statement

The initial iSCSI configuration (commit f370213) caused worker nodes to fail boot with:
```
writeUserFiles failed, rebooting in 35 minutes
```

This was caused by:
1. Mounting `/etc/iscsi` (conflicts with iscsi-tools extension)
2. Writing `/etc/multipath.conf` during early boot (filesystem not writable)
3. Missing explicit nodeIP configuration for dual-NIC workers

## ✅ Solution

The fix involves three changes:

1. **Remove `/etc/iscsi` mount** - Let iscsi-tools extension manage it
2. **Add explicit nodeIP** - Bind kubelet to primary network (10.1.71.0/24)
3. **Move multipath config to DaemonSet** - Configure post-boot when filesystem is writable

## 📋 Files in This Directory

| File | Purpose |
|------|---------|
| `talconfig.yaml` | **Main Talos configuration** (FIXED) |
| `iscsi-multipath-init.yaml` | DaemonSet that configures multipath post-boot |
| `apply-iscsi-fix.sh` | **Automated deployment script** (START HERE) |
| `verify-iscsi.sh` | Verification script to check configuration |
| `ISCSI_CONFIG_FIX.md` | **Detailed documentation** of the fix |
| `DEPLOYMENT_SUMMARY.md` | Deployment checklist and plan |
| `QUICKREF.md` | Quick reference for common commands |
| `README.md` | This file |

## 🚀 Quick Start

### 1. Review the Changes

```bash
cd ~/git/homelab/talos/talhelper

# See what changed
git diff <previous-commit> talconfig.yaml

# Read the detailed documentation
cat ISCSI_CONFIG_FIX.md
```

### 2. Apply the Fix

```bash
# Dry run first to preview
./apply-iscsi-fix.sh --dry-run

# Apply to all worker nodes
./apply-iscsi-fix.sh
```

This script will:
- Regenerate Talos configs
- Apply to each worker sequentially (jungle-cruise → haunted-mansion → peter-pans-flight)
- Wait for each node to reboot and become Ready
- Verify iSCSI functionality
- Deploy multipath DaemonSet

**Duration:** ~20-30 minutes (3 nodes × 5-10 min each)

### 3. Verify

```bash
# Run comprehensive verification (replace with your FlashArray iSCSI IP)
./verify-iscsi.sh 10.1.75.100

# Check node IPs (should be 10.1.71.x, NOT 10.1.75.x)
kubectl get nodes -o wide

# Check multipath DaemonSet
kubectl get pods -n kube-system -l app=iscsi-multipath-init -o wide
```

## 🔧 Manual Application (if needed)

If you prefer manual control:

```bash
# Regenerate config
talhelper genconfig

# Apply to one worker at a time
talosctl apply-config \
  --file clusterconfig/fastpass-jungle-cruise.yaml \
  --nodes 10.1.71.69

# Wait for Ready
kubectl wait --for=condition=Ready node/jungle-cruise --timeout=10m

# Verify
talosctl -n 10.1.71.69 service iscsid
talosctl -n 10.1.71.69 read /etc/iscsi/initiatorname.iscsi

# Repeat for other nodes...
```

## 📊 What Was Fixed

### Before (BROKEN)

```yaml
worker:
  patches:
    - machine:
        kubelet:
          extraMounts:
            - destination: /etc/iscsi      # ❌ BREAKS BOOT
              type: bind
              source: /etc/iscsi
            - destination: /var/lib/iscsi
              type: bind
              source: /var/lib/iscsi
        
        files:                              # ❌ BREAKS BOOT
          - path: /etc/multipath.conf
            op: create
            content: |
              ...
```

### After (FIXED)

```yaml
worker:
  patches:
    - machine:
        kubelet:
          nodeIP:                           # ✅ FIX: Explicit network
            validSubnets:
              - 10.1.71.0/24
          
          extraMounts:
            # ✅ FIX: Only /var/lib/iscsi with 'rw'
            - destination: /var/lib/iscsi
              type: bind
              source: /var/lib/iscsi
              options:
                - bind
                - rshared
                - rw
        
        # ✅ FIX: No files section (moved to DaemonSet)
```

## 🏗️ Architecture

```
┌─────────────────────────────────────────────────────────────┐
│ Talos Boot Sequence                                         │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  1. Load system extensions (iscsi-tools, util-linux-tools)  │
│     ↓                                                        │
│  2. Load kernel modules (iscsi_tcp, dm_multipath, ...)      │
│     ↓                                                        │
│  3. Mount /var/lib/iscsi for session persistence            │
│     ↓                                                        │
│  4. Start kubelet with nodeIP=10.1.71.x                     │
│     ↓                                                        │
│  5. Kubernetes starts (node Ready)                          │
│     ↓                                                        │
│  6. DaemonSet configures /etc/multipath.conf ← POST-BOOT    │
│                                                              │
└─────────────────────────────────────────────────────────────┘
```

**Key insight:** By moving multipath configuration to a DaemonSet (step 6), we avoid writing files during early boot when the filesystem may not be writable.

## 🌐 Network Configuration

Workers have dual NICs:

| Interface | Network | Purpose |
|-----------|---------|---------|
| **ens18** | 10.1.71.0/24 | **Primary** - Kubernetes API, pod traffic |
| **ens19** | 10.1.75.0/24 | **Storage** - iSCSI to FlashArray |

**Critical:** Kubelet must bind to ens18 (10.1.71.x) using `nodeIP.validSubnets`.

## 📚 Documentation

- **Quick Start:** This README
- **Quick Reference:** `QUICKREF.md` - Common commands
- **Detailed Fix:** `ISCSI_CONFIG_FIX.md` - Complete explanation
- **Deployment Plan:** `DEPLOYMENT_SUMMARY.md` - Step-by-step checklist
- **Portworx Guide:** `~/git/homelab/cluster/platform/portworx-csi/README.md`

## ✅ Success Criteria

After deployment, verify:

- [ ] All worker nodes show `Ready` status
- [ ] Node IPs are `10.1.71.x` (not `10.1.75.x`)
- [ ] `iscsi-tools` extension loaded
- [ ] Kernel modules loaded: `iscsi_tcp`, `dm_multipath`, `dm_round_robin`
- [ ] `iscsid` service ready
- [ ] `/var/lib/iscsi` directory accessible
- [ ] Unique initiator name on each node
- [ ] `/etc/multipath.conf` exists with Pure Storage config
- [ ] `ens19` interface up with `10.1.75.x` IP
- [ ] Multipath DaemonSet running on all workers
- [ ] No boot errors in `dmesg`

## 🆘 Troubleshooting

### Boot Failure

If a node fails to boot after applying config:

```bash
# Check dmesg for errors
talosctl -n <node-ip> dmesg | grep -i "error\|fail"

# Check Talos controller logs
talosctl -n <node-ip> logs controller-runtime

# Rollback
git checkout HEAD^ -- talconfig.yaml
talhelper genconfig
talosctl apply-config --file clusterconfig/<node>.yaml --nodes <node-ip>
```

### Wrong Node IP

If kubelet binds to `10.1.75.x` instead of `10.1.71.x`:

```bash
# Verify nodeIP.validSubnets in config
grep -A 3 "nodeIP:" talconfig.yaml

# Should show:
#   nodeIP:
#     validSubnets:
#       - 10.1.71.0/24

# If missing, add it and reapply
```

### iSCSI Not Working

```bash
# Full verification
./verify-iscsi.sh <flasharray-ip>

# Check specific components
talosctl -n <node-ip> service iscsid
talosctl -n <node-ip> read /etc/iscsi/initiatorname.iscsi
talosctl -n <node-ip> exec -- multipath -ll

# Test discovery
talosctl -n <node-ip> exec -- \
  iscsiadm -m discovery -t st -p <flasharray-iscsi-ip>
```

## 🔄 Next Steps

After successful deployment:

1. **Deploy Portworx CSI** (if not already deployed)
   ```bash
   cd ~/git/homelab/cluster/platform/portworx-csi
   cat README.md
   ```

2. **Test iSCSI to FlashArray**
   ```bash
   ./verify-iscsi.sh <flasharray-iscsi-ip>
   ```

3. **Create test PVC**
   ```yaml
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: test-pure-block
   spec:
     accessModes: [ReadWriteOnce]
     storageClassName: pure-block
     resources:
       requests:
         storage: 10Gi
   ```

4. **Monitor Portworx**
   ```bash
   kubectl logs -n portworx -l app=portworx-operator -f
   ```

## 📞 Support

- **Git Repository:** `mad-tea-party:rblundon/homelab`
- **Configuration Path:** `~/git/homelab/talos/talhelper/`
- **Talos Cluster:** fastpass (v1.13.2)
- **Related Commit:** f370213 (original broken config)

## 🔗 External References

- [Talos Storage Guide](https://www.talos.dev/v1.13/kubernetes-guides/configuration/storage/)
- [Talos iscsi-tools Extension](https://github.com/siderolabs/extensions/pkgs/container/iscsi-tools)
- [Portworx CSI Documentation](https://docs.portworx.com/portworx-csi/)
- [Pure Storage Best Practices](https://support.purestorage.com/)

---

**Last Updated:** 2026-06-20  
**Status:** Ready for deployment  
**Tested:** Dry-run verified, pending production deployment