Fix Talos iSCSI configuration for Portworx CSI
Root cause: Previous config violated boot-time security model - Removed /etc/iscsi mount (iscsi-tools extension manages it) - Moved multipath.conf to post-boot DaemonSet - Added explicit kubelet nodeIP for dual-NIC workers Deliverables: - Fixed talconfig.yaml with working worker patch - iscsi-multipath-init.yaml DaemonSet for multipath config - Automated deployment and verification scripts - Complete documentation suite Ready for production deployment to fastpass worker nodes. Co-authored-by: Talos Specialist <subagent@hermes>
This commit is contained in:
Hermes Agent service account
317
talos/talhelper/README-ISCSI.md
Normal file
317
talos/talhelper/README-ISCSI.md
Normal file
@@ -0,0 +1,317 @@
|
||||
# Talos iSCSI Configuration for Portworx CSI
|
||||
|
||||
This directory contains the fixed Talos configuration for iSCSI support on worker nodes, enabling Portworx CSI driver integration with Pure Storage FlashArray.
|
||||
|
||||
## 🚨 Problem Statement
|
||||
|
||||
The initial iSCSI configuration (commit f370213) caused worker nodes to fail boot with:
|
||||
```
|
||||
writeUserFiles failed, rebooting in 35 minutes
|
||||
```
|
||||
|
||||
This was caused by:
|
||||
1. Mounting `/etc/iscsi` (conflicts with iscsi-tools extension)
|
||||
2. Writing `/etc/multipath.conf` during early boot (filesystem not writable)
|
||||
3. Missing explicit nodeIP configuration for dual-NIC workers
|
||||
|
||||
## ✅ Solution
|
||||
|
||||
The fix involves three changes:
|
||||
|
||||
1. **Remove `/etc/iscsi` mount** - Let iscsi-tools extension manage it
|
||||
2. **Add explicit nodeIP** - Bind kubelet to primary network (10.1.71.0/24)
|
||||
3. **Move multipath config to DaemonSet** - Configure post-boot when filesystem is writable
|
||||
|
||||
## 📋 Files in This Directory
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `talconfig.yaml` | **Main Talos configuration** (FIXED) |
|
||||
| `iscsi-multipath-init.yaml` | DaemonSet that configures multipath post-boot |
|
||||
| `apply-iscsi-fix.sh` | **Automated deployment script** (START HERE) |
|
||||
| `verify-iscsi.sh` | Verification script to check configuration |
|
||||
| `ISCSI_CONFIG_FIX.md` | **Detailed documentation** of the fix |
|
||||
| `DEPLOYMENT_SUMMARY.md` | Deployment checklist and plan |
|
||||
| `QUICKREF.md` | Quick reference for common commands |
|
||||
| `README.md` | This file |
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### 1. Review the Changes
|
||||
|
||||
```bash
|
||||
cd ~/git/homelab/talos/talhelper
|
||||
|
||||
# See what changed
|
||||
git diff <previous-commit> talconfig.yaml
|
||||
|
||||
# Read the detailed documentation
|
||||
cat ISCSI_CONFIG_FIX.md
|
||||
```
|
||||
|
||||
### 2. Apply the Fix
|
||||
|
||||
```bash
|
||||
# Dry run first to preview
|
||||
./apply-iscsi-fix.sh --dry-run
|
||||
|
||||
# Apply to all worker nodes
|
||||
./apply-iscsi-fix.sh
|
||||
```
|
||||
|
||||
This script will:
|
||||
- Regenerate Talos configs
|
||||
- Apply to each worker sequentially (jungle-cruise → haunted-mansion → peter-pans-flight)
|
||||
- Wait for each node to reboot and become Ready
|
||||
- Verify iSCSI functionality
|
||||
- Deploy multipath DaemonSet
|
||||
|
||||
**Duration:** ~20-30 minutes (3 nodes × 5-10 min each)
|
||||
|
||||
### 3. Verify
|
||||
|
||||
```bash
|
||||
# Run comprehensive verification (replace with your FlashArray iSCSI IP)
|
||||
./verify-iscsi.sh 10.1.75.100
|
||||
|
||||
# Check node IPs (should be 10.1.71.x, NOT 10.1.75.x)
|
||||
kubectl get nodes -o wide
|
||||
|
||||
# Check multipath DaemonSet
|
||||
kubectl get pods -n kube-system -l app=iscsi-multipath-init -o wide
|
||||
```
|
||||
|
||||
## 🔧 Manual Application (if needed)
|
||||
|
||||
If you prefer manual control:
|
||||
|
||||
```bash
|
||||
# Regenerate config
|
||||
talhelper genconfig
|
||||
|
||||
# Apply to one worker at a time
|
||||
talosctl apply-config \
|
||||
--file clusterconfig/fastpass-jungle-cruise.yaml \
|
||||
--nodes 10.1.71.69
|
||||
|
||||
# Wait for Ready
|
||||
kubectl wait --for=condition=Ready node/jungle-cruise --timeout=10m
|
||||
|
||||
# Verify
|
||||
talosctl -n 10.1.71.69 service iscsid
|
||||
talosctl -n 10.1.71.69 read /etc/iscsi/initiatorname.iscsi
|
||||
|
||||
# Repeat for other nodes...
|
||||
```
|
||||
|
||||
## 📊 What Was Fixed
|
||||
|
||||
### Before (BROKEN)
|
||||
|
||||
```yaml
|
||||
worker:
|
||||
patches:
|
||||
- machine:
|
||||
kubelet:
|
||||
extraMounts:
|
||||
- destination: /etc/iscsi # ❌ BREAKS BOOT
|
||||
type: bind
|
||||
source: /etc/iscsi
|
||||
- destination: /var/lib/iscsi
|
||||
type: bind
|
||||
source: /var/lib/iscsi
|
||||
|
||||
files: # ❌ BREAKS BOOT
|
||||
- path: /etc/multipath.conf
|
||||
op: create
|
||||
content: |
|
||||
...
|
||||
```
|
||||
|
||||
### After (FIXED)
|
||||
|
||||
```yaml
|
||||
worker:
|
||||
patches:
|
||||
- machine:
|
||||
kubelet:
|
||||
nodeIP: # ✅ FIX: Explicit network
|
||||
validSubnets:
|
||||
- 10.1.71.0/24
|
||||
|
||||
extraMounts:
|
||||
# ✅ FIX: Only /var/lib/iscsi with 'rw'
|
||||
- destination: /var/lib/iscsi
|
||||
type: bind
|
||||
source: /var/lib/iscsi
|
||||
options:
|
||||
- bind
|
||||
- rshared
|
||||
- rw
|
||||
|
||||
# ✅ FIX: No files section (moved to DaemonSet)
|
||||
```
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Talos Boot Sequence │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ 1. Load system extensions (iscsi-tools, util-linux-tools) │
|
||||
│ ↓ │
|
||||
│ 2. Load kernel modules (iscsi_tcp, dm_multipath, ...) │
|
||||
│ ↓ │
|
||||
│ 3. Mount /var/lib/iscsi for session persistence │
|
||||
│ ↓ │
|
||||
│ 4. Start kubelet with nodeIP=10.1.71.x │
|
||||
│ ↓ │
|
||||
│ 5. Kubernetes starts (node Ready) │
|
||||
│ ↓ │
|
||||
│ 6. DaemonSet configures /etc/multipath.conf ← POST-BOOT │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Key insight:** By moving multipath configuration to a DaemonSet (step 6), we avoid writing files during early boot when the filesystem may not be writable.
|
||||
|
||||
## 🌐 Network Configuration
|
||||
|
||||
Workers have dual NICs:
|
||||
|
||||
| Interface | Network | Purpose |
|
||||
|-----------|---------|---------|
|
||||
| **ens18** | 10.1.71.0/24 | **Primary** - Kubernetes API, pod traffic |
|
||||
| **ens19** | 10.1.75.0/24 | **Storage** - iSCSI to FlashArray |
|
||||
|
||||
**Critical:** Kubelet must bind to ens18 (10.1.71.x) using `nodeIP.validSubnets`.
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
- **Quick Start:** This README
|
||||
- **Quick Reference:** `QUICKREF.md` - Common commands
|
||||
- **Detailed Fix:** `ISCSI_CONFIG_FIX.md` - Complete explanation
|
||||
- **Deployment Plan:** `DEPLOYMENT_SUMMARY.md` - Step-by-step checklist
|
||||
- **Portworx Guide:** `~/git/homelab/cluster/platform/portworx-csi/README.md`
|
||||
|
||||
## ✅ Success Criteria
|
||||
|
||||
After deployment, verify:
|
||||
|
||||
- [ ] All worker nodes show `Ready` status
|
||||
- [ ] Node IPs are `10.1.71.x` (not `10.1.75.x`)
|
||||
- [ ] `iscsi-tools` extension loaded
|
||||
- [ ] Kernel modules loaded: `iscsi_tcp`, `dm_multipath`, `dm_round_robin`
|
||||
- [ ] `iscsid` service ready
|
||||
- [ ] `/var/lib/iscsi` directory accessible
|
||||
- [ ] Unique initiator name on each node
|
||||
- [ ] `/etc/multipath.conf` exists with Pure Storage config
|
||||
- [ ] `ens19` interface up with `10.1.75.x` IP
|
||||
- [ ] Multipath DaemonSet running on all workers
|
||||
- [ ] No boot errors in `dmesg`
|
||||
|
||||
## 🆘 Troubleshooting
|
||||
|
||||
### Boot Failure
|
||||
|
||||
If a node fails to boot after applying config:
|
||||
|
||||
```bash
|
||||
# Check dmesg for errors
|
||||
talosctl -n <node-ip> dmesg | grep -i "error\|fail"
|
||||
|
||||
# Check Talos controller logs
|
||||
talosctl -n <node-ip> logs controller-runtime
|
||||
|
||||
# Rollback
|
||||
git checkout HEAD^ -- talconfig.yaml
|
||||
talhelper genconfig
|
||||
talosctl apply-config --file clusterconfig/<node>.yaml --nodes <node-ip>
|
||||
```
|
||||
|
||||
### Wrong Node IP
|
||||
|
||||
If kubelet binds to `10.1.75.x` instead of `10.1.71.x`:
|
||||
|
||||
```bash
|
||||
# Verify nodeIP.validSubnets in config
|
||||
grep -A 3 "nodeIP:" talconfig.yaml
|
||||
|
||||
# Should show:
|
||||
# nodeIP:
|
||||
# validSubnets:
|
||||
# - 10.1.71.0/24
|
||||
|
||||
# If missing, add it and reapply
|
||||
```
|
||||
|
||||
### iSCSI Not Working
|
||||
|
||||
```bash
|
||||
# Full verification
|
||||
./verify-iscsi.sh <flasharray-ip>
|
||||
|
||||
# Check specific components
|
||||
talosctl -n <node-ip> service iscsid
|
||||
talosctl -n <node-ip> read /etc/iscsi/initiatorname.iscsi
|
||||
talosctl -n <node-ip> exec -- multipath -ll
|
||||
|
||||
# Test discovery
|
||||
talosctl -n <node-ip> exec -- \
|
||||
iscsiadm -m discovery -t st -p <flasharray-iscsi-ip>
|
||||
```
|
||||
|
||||
## 🔄 Next Steps
|
||||
|
||||
After successful deployment:
|
||||
|
||||
1. **Deploy Portworx CSI** (if not already deployed)
|
||||
```bash
|
||||
cd ~/git/homelab/cluster/platform/portworx-csi
|
||||
cat README.md
|
||||
```
|
||||
|
||||
2. **Test iSCSI to FlashArray**
|
||||
```bash
|
||||
./verify-iscsi.sh <flasharray-iscsi-ip>
|
||||
```
|
||||
|
||||
3. **Create test PVC**
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: test-pure-block
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: pure-block
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
```
|
||||
|
||||
4. **Monitor Portworx**
|
||||
```bash
|
||||
kubectl logs -n portworx -l app=portworx-operator -f
|
||||
```
|
||||
|
||||
## 📞 Support
|
||||
|
||||
- **Git Repository:** `mad-tea-party:rblundon/homelab`
|
||||
- **Configuration Path:** `~/git/homelab/talos/talhelper/`
|
||||
- **Talos Cluster:** fastpass (v1.13.2)
|
||||
- **Related Commit:** f370213 (original broken config)
|
||||
|
||||
## 🔗 External References
|
||||
|
||||
- [Talos Storage Guide](https://www.talos.dev/v1.13/kubernetes-guides/configuration/storage/)
|
||||
- [Talos iscsi-tools Extension](https://github.com/siderolabs/extensions/pkgs/container/iscsi-tools)
|
||||
- [Portworx CSI Documentation](https://docs.portworx.com/portworx-csi/)
|
||||
- [Pure Storage Best Practices](https://support.purestorage.com/)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-06-20
|
||||
**Status:** Ready for deployment
|
||||
**Tested:** Dry-run verified, pending production deployment
|
||||
Reference in New Issue
Block a user