homelab/talos/talhelper/QUICKREF.md

Talos iSCSI Quick Reference

Quick Start

cd ~/git/homelab/talos/talhelper

# Apply the fix
./apply-iscsi-fix.sh

# Verify after deployment
./verify-iscsi.sh <flasharray-iscsi-ip>

What Changed

Component	Before (BROKEN)	After (FIXED)
/etc/iscsi mount	✗ Mounted → boot failure	✓ Removed
/var/lib/iscsi mount	bind, rshared	bind, rshared, rw
Multipath config	Written at boot	DaemonSet post-boot
Node IP	Auto-selected (wrong)	Explicit 10.1.71.0/24
Boot result	FAILS	WORKS

Key Commands

Check Node Status

kubectl get nodes -o wide
# All workers should show 10.1.71.x as INTERNAL-IP

Verify iSCSI on a Node

NODE_IP=10.1.71.69

# Service status
talosctl -n $NODE_IP service iscsid

# Modules loaded
talosctl -n $NODE_IP read /proc/modules | grep -E "iscsi|multipath"

# Initiator name
talosctl -n $NODE_IP read /etc/iscsi/initiatorname.iscsi

# Multipath config
talosctl -n $NODE_IP read /etc/multipath.conf

# Multipath devices
talosctl -n $NODE_IP exec -- multipath -ll

Test FlashArray Discovery

# Replace with your FlashArray iSCSI IP
FLASHARRAY_IP=10.1.75.100

talosctl -n 10.1.71.69 exec -- \
  iscsiadm -m discovery -t st -p $FLASHARRAY_IP

Check Multipath DaemonSet

kubectl get daemonset -n kube-system iscsi-multipath-init
kubectl get pods -n kube-system -l app=iscsi-multipath-init -o wide

Troubleshooting

Node Not Ready After Update

# Check dmesg for errors
talosctl -n <node-ip> dmesg | grep -i "error\|fail"

# Check Talos logs
talosctl -n <node-ip> logs controller-runtime

# Rollback if needed
git checkout HEAD^ -- talconfig.yaml
talhelper genconfig
talosctl apply-config --file clusterconfig/<node>.yaml --nodes <node-ip>

Wrong Node IP (10.1.75.x instead of 10.1.71.x)

# Verify talconfig has nodeIP.validSubnets
grep -A 3 "nodeIP:" talconfig.yaml

# Should show:
#   nodeIP:
#     validSubnets:
#       - 10.1.71.0/24

# If missing, edit talconfig.yaml and reapply

Multipath Config Missing

# Check DaemonSet is running
kubectl get pods -n kube-system -l app=iscsi-multipath-init

# If not deployed:
kubectl apply -f iscsi-multipath-init.yaml

# Force recreation
kubectl delete pod -n kube-system -l app=iscsi-multipath-init

iSCSI Discovery Fails

# Check network connectivity on ens19
talosctl -n <node-ip> exec -- ip addr show ens19
talosctl -n <node-ip> exec -- ping -c 3 <flasharray-ip>

# Check iscsid service
talosctl -n <node-ip> service iscsid

# Start iscsid if needed (auto-starts on discovery)
talosctl -n <node-ip> exec -- \
  iscsiadm -m discovery -t st -p <flasharray-ip>

Worker Nodes

Hostname	Management IP	iSCSI IP
jungle-cruise	10.1.71.69	10.1.75.69
haunted-mansion	10.1.71.70	10.1.75.70
peter-pans-flight	10.1.71.71	10.1.75.71

Network Layout

┌─────────────────┐
│   Worker Node   │
│                 │
│  ens18          │  10.1.71.x/24  ← Kubernetes traffic (primary)
│    ↓            │                  kubelet binds here
│  Default GW     │                  
│                 │
│  ens19          │  10.1.75.x/24  ← iSCSI storage traffic
│    ↓            │                  FlashArray connectivity
│  Pure FlashArray│
└─────────────────┘

Files

~/git/homelab/talos/talhelper/
├── talconfig.yaml              # Main config (FIXED)
├── iscsi-multipath-init.yaml   # Multipath DaemonSet
├── apply-iscsi-fix.sh          # Deployment script
├── verify-iscsi.sh             # Verification script
├── ISCSI_CONFIG_FIX.md         # Full documentation
├── DEPLOYMENT_SUMMARY.md       # Deployment checklist
└── QUICKREF.md                 # This file

Next Steps After Deployment

1. ✅ Verify all nodes are Ready with correct IPs
2. ✅ Test iSCSI discovery to FlashArray
3. ✅ Check multipath DaemonSet is running
4. ⏭️ Deploy/Update Portworx CSI driver
5. ⏭️ Create test PVC with pure-block StorageClass
6. ⏭️ Monitor Portworx pod logs for iSCSI sessions

Important Notes

⚠️ Do NOT mount /etc/iscsi - iscsi-tools extension manages it
⚠️ Do NOT write files during Talos boot - use DaemonSets instead
✅ Always specify nodeIP.validSubnets for dual-NIC setups
✅ Test on one node first if unsure
✅ Keep git history for easy rollback

Documentation

• Full Fix Details: ISCSI_CONFIG_FIX.md
• Deployment Plan: DEPLOYMENT_SUMMARY.md
• Portworx Guide: ~/git/homelab/cluster/platform/portworx-csi/README.md
• Talos Storage: https://www.talos.dev/v1.13/kubernetes-guides/configuration/storage/