# Talos iSCSI Quick Reference

## Quick Start

```bash
cd ~/git/homelab/talos/talhelper

# Apply the fix
./apply-iscsi-fix.sh

# Verify after deployment
./verify-iscsi.sh <flasharray-iscsi-ip>
```

## What Changed

| Component | Before (BROKEN) | After (FIXED) |
|-----------|----------------|---------------|
| `/etc/iscsi` mount | ✗ Mounted → boot failure | ✓ Removed |
| `/var/lib/iscsi` mount | `bind, rshared` | `bind, rshared, rw` |
| Multipath config | Written at boot | DaemonSet post-boot |
| Node IP | Auto-selected (wrong) | Explicit `10.1.71.0/24` |
| Boot result | **FAILS** | **WORKS** |

## Key Commands

### Check Node Status
```bash
kubectl get nodes -o wide
# All workers should show 10.1.71.x as INTERNAL-IP
```

### Verify iSCSI on a Node
```bash
NODE_IP=10.1.71.69

# Service status
talosctl -n $NODE_IP service iscsid

# Modules loaded
talosctl -n $NODE_IP read /proc/modules | grep -E "iscsi|multipath"

# Initiator name
talosctl -n $NODE_IP read /etc/iscsi/initiatorname.iscsi

# Multipath config
talosctl -n $NODE_IP read /etc/multipath.conf

# Multipath devices
talosctl -n $NODE_IP exec -- multipath -ll
```

### Test FlashArray Discovery
```bash
# Replace with your FlashArray iSCSI IP
FLASHARRAY_IP=10.1.75.100

talosctl -n 10.1.71.69 exec -- \
  iscsiadm -m discovery -t st -p $FLASHARRAY_IP
```

### Check Multipath DaemonSet
```bash
kubectl get daemonset -n kube-system iscsi-multipath-init
kubectl get pods -n kube-system -l app=iscsi-multipath-init -o wide
```

## Troubleshooting

### Node Not Ready After Update
```bash
# Check dmesg for errors
talosctl -n <node-ip> dmesg | grep -i "error\|fail"

# Check Talos logs
talosctl -n <node-ip> logs controller-runtime

# Rollback if needed
git checkout HEAD^ -- talconfig.yaml
talhelper genconfig
talosctl apply-config --file clusterconfig/<node>.yaml --nodes <node-ip>
```

### Wrong Node IP (10.1.75.x instead of 10.1.71.x)
```bash
# Verify talconfig has nodeIP.validSubnets
grep -A 3 "nodeIP:" talconfig.yaml

# Should show:
#   nodeIP:
#     validSubnets:
#       - 10.1.71.0/24

# If missing, edit talconfig.yaml and reapply
```

### Multipath Config Missing
```bash
# Check DaemonSet is running
kubectl get pods -n kube-system -l app=iscsi-multipath-init

# If not deployed:
kubectl apply -f iscsi-multipath-init.yaml

# Force recreation
kubectl delete pod -n kube-system -l app=iscsi-multipath-init
```

### iSCSI Discovery Fails
```bash
# Check network connectivity on ens19
talosctl -n <node-ip> exec -- ip addr show ens19
talosctl -n <node-ip> exec -- ping -c 3 <flasharray-ip>

# Check iscsid service
talosctl -n <node-ip> service iscsid

# Start iscsid if needed (auto-starts on discovery)
talosctl -n <node-ip> exec -- \
  iscsiadm -m discovery -t st -p <flasharray-ip>
```

## Worker Nodes

| Hostname | Management IP | iSCSI IP |
|----------|--------------|----------|
| jungle-cruise | 10.1.71.69 | 10.1.75.69 |
| haunted-mansion | 10.1.71.70 | 10.1.75.70 |
| peter-pans-flight | 10.1.71.71 | 10.1.75.71 |

## Network Layout

```
┌─────────────────┐
│   Worker Node   │
│                 │
│  ens18          │  10.1.71.x/24  ← Kubernetes traffic (primary)
│    ↓            │                  kubelet binds here
│  Default GW     │                  
│                 │
│  ens19          │  10.1.75.x/24  ← iSCSI storage traffic
│    ↓            │                  FlashArray connectivity
│  Pure FlashArray│
└─────────────────┘
```

## Files

```
~/git/homelab/talos/talhelper/
├── talconfig.yaml              # Main config (FIXED)
├── iscsi-multipath-init.yaml   # Multipath DaemonSet
├── apply-iscsi-fix.sh          # Deployment script
├── verify-iscsi.sh             # Verification script
├── ISCSI_CONFIG_FIX.md         # Full documentation
├── DEPLOYMENT_SUMMARY.md       # Deployment checklist
└── QUICKREF.md                 # This file
```

## Next Steps After Deployment

1. ✅ Verify all nodes are Ready with correct IPs
2. ✅ Test iSCSI discovery to FlashArray
3. ✅ Check multipath DaemonSet is running
4. ⏭️ Deploy/Update Portworx CSI driver
5. ⏭️ Create test PVC with `pure-block` StorageClass
6. ⏭️ Monitor Portworx pod logs for iSCSI sessions

## Important Notes

⚠️ **Do NOT mount `/etc/iscsi`** - iscsi-tools extension manages it  
⚠️ **Do NOT write files during Talos boot** - use DaemonSets instead  
✅ **Always specify nodeIP.validSubnets** for dual-NIC setups  
✅ **Test on one node first** if unsure  
✅ **Keep git history** for easy rollback  

## Documentation

- Full Fix Details: `ISCSI_CONFIG_FIX.md`
- Deployment Plan: `DEPLOYMENT_SUMMARY.md`
- Portworx Guide: `~/git/homelab/cluster/platform/portworx-csi/README.md`
- Talos Storage: https://www.talos.dev/v1.13/kubernetes-guides/configuration/storage/