e8303d5129
Root cause: Previous config violated boot-time security model - Removed /etc/iscsi mount (iscsi-tools extension manages it) - Moved multipath.conf to post-boot DaemonSet - Added explicit kubelet nodeIP for dual-NIC workers Deliverables: - Fixed talconfig.yaml with working worker patch - iscsi-multipath-init.yaml DaemonSet for multipath config - Automated deployment and verification scripts - Complete documentation suite Ready for production deployment to fastpass worker nodes. Co-authored-by: Talos Specialist <subagent@hermes>
183 lines
4.8 KiB
Markdown
183 lines
4.8 KiB
Markdown
# Talos iSCSI Quick Reference
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
cd ~/git/homelab/talos/talhelper
|
|
|
|
# Apply the fix
|
|
./apply-iscsi-fix.sh
|
|
|
|
# Verify after deployment
|
|
./verify-iscsi.sh <flasharray-iscsi-ip>
|
|
```
|
|
|
|
## What Changed
|
|
|
|
| Component | Before (BROKEN) | After (FIXED) |
|
|
|-----------|----------------|---------------|
|
|
| `/etc/iscsi` mount | ✗ Mounted → boot failure | ✓ Removed |
|
|
| `/var/lib/iscsi` mount | `bind, rshared` | `bind, rshared, rw` |
|
|
| Multipath config | Written at boot | DaemonSet post-boot |
|
|
| Node IP | Auto-selected (wrong) | Explicit `10.1.71.0/24` |
|
|
| Boot result | **FAILS** | **WORKS** |
|
|
|
|
## Key Commands
|
|
|
|
### Check Node Status
|
|
```bash
|
|
kubectl get nodes -o wide
|
|
# All workers should show 10.1.71.x as INTERNAL-IP
|
|
```
|
|
|
|
### Verify iSCSI on a Node
|
|
```bash
|
|
NODE_IP=10.1.71.69
|
|
|
|
# Service status
|
|
talosctl -n $NODE_IP service iscsid
|
|
|
|
# Modules loaded
|
|
talosctl -n $NODE_IP read /proc/modules | grep -E "iscsi|multipath"
|
|
|
|
# Initiator name
|
|
talosctl -n $NODE_IP read /etc/iscsi/initiatorname.iscsi
|
|
|
|
# Multipath config
|
|
talosctl -n $NODE_IP read /etc/multipath.conf
|
|
|
|
# Multipath devices
|
|
talosctl -n $NODE_IP exec -- multipath -ll
|
|
```
|
|
|
|
### Test FlashArray Discovery
|
|
```bash
|
|
# Replace with your FlashArray iSCSI IP
|
|
FLASHARRAY_IP=10.1.75.100
|
|
|
|
talosctl -n 10.1.71.69 exec -- \
|
|
iscsiadm -m discovery -t st -p $FLASHARRAY_IP
|
|
```
|
|
|
|
### Check Multipath DaemonSet
|
|
```bash
|
|
kubectl get daemonset -n kube-system iscsi-multipath-init
|
|
kubectl get pods -n kube-system -l app=iscsi-multipath-init -o wide
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Node Not Ready After Update
|
|
```bash
|
|
# Check dmesg for errors
|
|
talosctl -n <node-ip> dmesg | grep -i "error\|fail"
|
|
|
|
# Check Talos logs
|
|
talosctl -n <node-ip> logs controller-runtime
|
|
|
|
# Rollback if needed
|
|
git checkout HEAD^ -- talconfig.yaml
|
|
talhelper genconfig
|
|
talosctl apply-config --file clusterconfig/<node>.yaml --nodes <node-ip>
|
|
```
|
|
|
|
### Wrong Node IP (10.1.75.x instead of 10.1.71.x)
|
|
```bash
|
|
# Verify talconfig has nodeIP.validSubnets
|
|
grep -A 3 "nodeIP:" talconfig.yaml
|
|
|
|
# Should show:
|
|
# nodeIP:
|
|
# validSubnets:
|
|
# - 10.1.71.0/24
|
|
|
|
# If missing, edit talconfig.yaml and reapply
|
|
```
|
|
|
|
### Multipath Config Missing
|
|
```bash
|
|
# Check DaemonSet is running
|
|
kubectl get pods -n kube-system -l app=iscsi-multipath-init
|
|
|
|
# If not deployed:
|
|
kubectl apply -f iscsi-multipath-init.yaml
|
|
|
|
# Force recreation
|
|
kubectl delete pod -n kube-system -l app=iscsi-multipath-init
|
|
```
|
|
|
|
### iSCSI Discovery Fails
|
|
```bash
|
|
# Check network connectivity on ens19
|
|
talosctl -n <node-ip> exec -- ip addr show ens19
|
|
talosctl -n <node-ip> exec -- ping -c 3 <flasharray-ip>
|
|
|
|
# Check iscsid service
|
|
talosctl -n <node-ip> service iscsid
|
|
|
|
# Start iscsid if needed (auto-starts on discovery)
|
|
talosctl -n <node-ip> exec -- \
|
|
iscsiadm -m discovery -t st -p <flasharray-ip>
|
|
```
|
|
|
|
## Worker Nodes
|
|
|
|
| Hostname | Management IP | iSCSI IP |
|
|
|----------|--------------|----------|
|
|
| jungle-cruise | 10.1.71.69 | 10.1.75.69 |
|
|
| haunted-mansion | 10.1.71.70 | 10.1.75.70 |
|
|
| peter-pans-flight | 10.1.71.71 | 10.1.75.71 |
|
|
|
|
## Network Layout
|
|
|
|
```
|
|
┌─────────────────┐
|
|
│ Worker Node │
|
|
│ │
|
|
│ ens18 │ 10.1.71.x/24 ← Kubernetes traffic (primary)
|
|
│ ↓ │ kubelet binds here
|
|
│ Default GW │
|
|
│ │
|
|
│ ens19 │ 10.1.75.x/24 ← iSCSI storage traffic
|
|
│ ↓ │ FlashArray connectivity
|
|
│ Pure FlashArray│
|
|
└─────────────────┘
|
|
```
|
|
|
|
## Files
|
|
|
|
```
|
|
~/git/homelab/talos/talhelper/
|
|
├── talconfig.yaml # Main config (FIXED)
|
|
├── iscsi-multipath-init.yaml # Multipath DaemonSet
|
|
├── apply-iscsi-fix.sh # Deployment script
|
|
├── verify-iscsi.sh # Verification script
|
|
├── ISCSI_CONFIG_FIX.md # Full documentation
|
|
├── DEPLOYMENT_SUMMARY.md # Deployment checklist
|
|
└── QUICKREF.md # This file
|
|
```
|
|
|
|
## Next Steps After Deployment
|
|
|
|
1. ✅ Verify all nodes are Ready with correct IPs
|
|
2. ✅ Test iSCSI discovery to FlashArray
|
|
3. ✅ Check multipath DaemonSet is running
|
|
4. ⏭️ Deploy/Update Portworx CSI driver
|
|
5. ⏭️ Create test PVC with `pure-block` StorageClass
|
|
6. ⏭️ Monitor Portworx pod logs for iSCSI sessions
|
|
|
|
## Important Notes
|
|
|
|
⚠️ **Do NOT mount `/etc/iscsi`** - iscsi-tools extension manages it
|
|
⚠️ **Do NOT write files during Talos boot** - use DaemonSets instead
|
|
✅ **Always specify nodeIP.validSubnets** for dual-NIC setups
|
|
✅ **Test on one node first** if unsure
|
|
✅ **Keep git history** for easy rollback
|
|
|
|
## Documentation
|
|
|
|
- Full Fix Details: `ISCSI_CONFIG_FIX.md`
|
|
- Deployment Plan: `DEPLOYMENT_SUMMARY.md`
|
|
- Portworx Guide: `~/git/homelab/cluster/platform/portworx-csi/README.md`
|
|
- Talos Storage: https://www.talos.dev/v1.13/kubernetes-guides/configuration/storage/
|