Add Portworx CSI driver for Pure Storage FlashArray
- Deploy Portworx Operator + CSI driver via ArgoCD - Support both iSCSI block and NFS file storage from FlashArray - Integrate with 1Password External Secrets for FlashArray credentials - Include comprehensive deployment documentation and validation script - Storage classes: pure-block (iSCSI) and pure-file (NFS) - Talos Linux compatible with iSCSI/multipath configuration
This commit is contained in:
Hermes Agent service account
466
cluster/platform/portworx-csi/DEPLOYMENT-CHECKLIST.md
Normal file
466
cluster/platform/portworx-csi/DEPLOYMENT-CHECKLIST.md
Normal file
@@ -0,0 +1,466 @@
|
||||
# Portworx CSI Deployment Checklist
|
||||
|
||||
Pre-deployment checklist for Portworx CSI driver on Talos Linux with Pure Storage FlashArray.
|
||||
|
||||
## Phase 1: FlashArray Preparation
|
||||
|
||||
- [ ] **Create FlashArray storage admin user**
|
||||
- Username: `portworx-csi`
|
||||
- Role: Storage Admin
|
||||
- Password: Set in FlashArray UI
|
||||
|
||||
- [ ] **Generate FlashArray API token**
|
||||
- User: `portworx-csi`
|
||||
- Expiration: None (non-expiring)
|
||||
- Token format: `XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX`
|
||||
- **Save token immediately** (cannot be retrieved later)
|
||||
|
||||
- [ ] **Verify FlashArray management endpoint**
|
||||
- Test connectivity: `ping flasharray.mk-labs.cloud`
|
||||
- Test HTTPS: `curl -k https://flasharray.mk-labs.cloud`
|
||||
- Record IP/hostname: _______________________
|
||||
|
||||
- [ ] **Configure iSCSI targets on FlashArray**
|
||||
- Navigate to: **Settings → Network**
|
||||
- Verify iSCSI interfaces are enabled
|
||||
- Test discovery from Talos node:
|
||||
```bash
|
||||
talosctl -n <node> exec -- iscsiadm -m discovery -t st -p <flasharray-ip>
|
||||
```
|
||||
|
||||
- [ ] **(Optional) Enable FlashArray File Services for NFS**
|
||||
- Enable File Services: **Settings → File Services**
|
||||
- Create filesystem: `fastpass-nfs`
|
||||
- Create NFS policy with user mapping **disabled**
|
||||
- Record NFS endpoint: _______________________
|
||||
|
||||
## Phase 2: 1Password Secret Configuration
|
||||
|
||||
- [ ] **Create 1Password item**
|
||||
- Vault: `homelab`
|
||||
- Item name: `pure-flasharray-fastpass`
|
||||
- Category: Secure Note or API Credential
|
||||
|
||||
- [ ] **Add pure.json field to 1Password item**
|
||||
- Field name: `pure.json`
|
||||
- Field type: Text
|
||||
- Content format: JSON (see below)
|
||||
|
||||
- [ ] **Populate pure.json content**
|
||||
```json
|
||||
{
|
||||
"FlashArrays": [
|
||||
{
|
||||
"MgmtEndPoint": "flasharray.mk-labs.cloud",
|
||||
"APIToken": "paste-api-token-here",
|
||||
"NFSEndPoint": "flasharray-nfs.mk-labs.cloud"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
- Replace `flasharray.mk-labs.cloud` with actual endpoint
|
||||
- Replace `paste-api-token-here` with FlashArray API token
|
||||
- Remove `NFSEndPoint` line if not using File Services
|
||||
|
||||
- [ ] **Validate JSON syntax**
|
||||
```bash
|
||||
# Copy content from 1Password and validate
|
||||
echo '<paste-json>' | jq .
|
||||
```
|
||||
|
||||
- [ ] **Verify 1Password Connect can access item**
|
||||
```bash
|
||||
# From a pod with op CLI or via 1Password UI
|
||||
# Confirm item exists in homelab vault
|
||||
```
|
||||
|
||||
## Phase 3: Talos Node Configuration
|
||||
|
||||
- [ ] **Edit talconfig.yaml**
|
||||
- File: `~/git/homelab/talos/talhelper/talconfig.yaml`
|
||||
- Add patches from `cluster/platform/portworx-csi/talos-config-reference.yaml`
|
||||
- Sections to add:
|
||||
- [ ] System extensions (iscsi-tools, util-linux-tools)
|
||||
- [ ] Kernel modules (iscsi_tcp, dm_multipath, dm_round_robin)
|
||||
- [ ] /etc/multipath.conf file
|
||||
- [ ] /etc/udev/rules.d/99-pure-storage.rules file
|
||||
- [ ] kubelet extraMounts for /var/lib/iscsi
|
||||
|
||||
- [ ] **Generate updated Talos configs**
|
||||
```bash
|
||||
cd ~/git/homelab/talos/talhelper
|
||||
talhelper genconfig
|
||||
```
|
||||
|
||||
- [ ] **Apply Talos configuration to all worker nodes**
|
||||
```bash
|
||||
# For each worker node:
|
||||
talosctl apply-config \
|
||||
--file clusterconfig/<node-name>.yaml \
|
||||
--nodes <node-ip>
|
||||
```
|
||||
|
||||
Nodes:
|
||||
- [ ] Node 1: _________________ (IP: _____________)
|
||||
- [ ] Node 2: _________________ (IP: _____________)
|
||||
- [ ] Node 3: _________________ (IP: _____________)
|
||||
|
||||
- [ ] **Reboot all worker nodes**
|
||||
```bash
|
||||
# Reboot nodes one at a time to load kernel modules
|
||||
talosctl reboot --nodes <node-ip>
|
||||
|
||||
# Wait for node to come back online
|
||||
kubectl get nodes
|
||||
```
|
||||
|
||||
- [ ] **Verify iSCSI initiator on each node**
|
||||
```bash
|
||||
talosctl -n <node-ip> get service iscsid
|
||||
# Expected: iscsid service running
|
||||
```
|
||||
|
||||
- [ ] **Verify multipath on each node**
|
||||
```bash
|
||||
talosctl -n <node-ip> exec -- multipath -ll
|
||||
# Expected: multipath running (may show no paths yet)
|
||||
```
|
||||
|
||||
- [ ] **Verify unique iSCSI initiator names**
|
||||
```bash
|
||||
# For each node:
|
||||
talosctl -n <node-ip> read /etc/iscsi/initiatorname.iscsi
|
||||
|
||||
# Record initiator names (must be unique):
|
||||
# Node 1: iqn.1994-05.com.redhat:_______________
|
||||
# Node 2: iqn.1994-05.com.redhat:_______________
|
||||
# Node 3: iqn.1994-05.com.redhat:_______________
|
||||
```
|
||||
|
||||
- [ ] **Verify kernel modules loaded**
|
||||
```bash
|
||||
talosctl -n <node-ip> exec -- lsmod | grep -E "iscsi|dm_multipath"
|
||||
# Expected: iscsi_tcp, dm_multipath, dm_round_robin
|
||||
```
|
||||
|
||||
- [ ] **Test iSCSI discovery from each node**
|
||||
```bash
|
||||
talosctl -n <node-ip> exec -- \
|
||||
iscsiadm -m discovery -t st -p <flasharray-ip>
|
||||
# Expected: List of iSCSI targets from FlashArray
|
||||
```
|
||||
|
||||
## Phase 4: GitOps Deployment
|
||||
|
||||
- [ ] **Review manifests**
|
||||
- [ ] `cluster/platform/portworx-csi/application.yaml`
|
||||
- [ ] `cluster/platform/portworx-csi/operator-values.yaml`
|
||||
- [ ] `cluster/platform/portworx-csi/storagecluster.yaml`
|
||||
- [ ] `cluster/platform/portworx-csi/externalsecret.yaml`
|
||||
- [ ] `cluster/platform/portworx-csi/storageclass-block.yaml`
|
||||
- [ ] `cluster/platform/portworx-csi/storageclass-file.yaml`
|
||||
|
||||
- [ ] **Customize for your environment**
|
||||
- Update FlashArray endpoints if different from defaults
|
||||
- Adjust resource limits if needed
|
||||
- Review sync wave (currently: 2)
|
||||
|
||||
- [ ] **Commit to Git**
|
||||
```bash
|
||||
cd ~/git/homelab
|
||||
git add cluster/platform/portworx-csi/
|
||||
git commit -m "Add Portworx CSI driver for Pure FlashArray"
|
||||
git push origin main
|
||||
```
|
||||
|
||||
- [ ] **Verify ArgoCD auto-sync or trigger manual sync**
|
||||
```bash
|
||||
# Check ArgoCD application status
|
||||
kubectl get application -n argocd portworx-csi
|
||||
|
||||
# Manual sync if needed
|
||||
argocd app sync portworx-csi
|
||||
```
|
||||
|
||||
## Phase 5: Deployment Verification
|
||||
|
||||
- [ ] **Verify ArgoCD application is healthy**
|
||||
```bash
|
||||
kubectl get application -n argocd portworx-csi
|
||||
# Expected: STATUS=Synced, HEALTH=Healthy
|
||||
```
|
||||
|
||||
- [ ] **Check namespace created**
|
||||
```bash
|
||||
kubectl get namespace portworx
|
||||
```
|
||||
|
||||
- [ ] **Verify ExternalSecret synced**
|
||||
```bash
|
||||
kubectl get externalsecret -n portworx px-pure-secret
|
||||
# Expected: STATUS=SecretSynced
|
||||
|
||||
kubectl get secret -n portworx px-pure-secret
|
||||
# Expected: Secret exists with pure.json key
|
||||
```
|
||||
|
||||
- [ ] **Validate secret content**
|
||||
```bash
|
||||
kubectl get secret -n portworx px-pure-secret \
|
||||
-o jsonpath='{.data.pure\.json}' | base64 -d | jq .
|
||||
# Expected: Valid JSON with FlashArray config
|
||||
```
|
||||
|
||||
- [ ] **Check Portworx Operator deployed**
|
||||
```bash
|
||||
kubectl get deployment -n portworx portworx-operator
|
||||
kubectl get pods -n portworx -l app=portworx-operator
|
||||
# Expected: 1/1 pods running
|
||||
```
|
||||
|
||||
- [ ] **Verify StorageCluster created**
|
||||
```bash
|
||||
kubectl get storagecluster -n portworx px-cluster-fastpass
|
||||
# Expected: STATUS=Running or Initializing
|
||||
```
|
||||
|
||||
- [ ] **Check PX-CSI pods running**
|
||||
```bash
|
||||
kubectl get pods -n portworx
|
||||
# Expected pods:
|
||||
# - portworx-operator-...
|
||||
# - px-csi-controller-...
|
||||
# - px-csi-node-... (one per node)
|
||||
```
|
||||
|
||||
- [ ] **Verify CSI driver registered**
|
||||
```bash
|
||||
kubectl get csidrivers
|
||||
# Expected: pxd.portworx.com
|
||||
|
||||
kubectl get csinode
|
||||
# Expected: One CSINode per Kubernetes node
|
||||
```
|
||||
|
||||
- [ ] **Check StorageClasses created**
|
||||
```bash
|
||||
kubectl get storageclass pure-block pure-file
|
||||
# Expected: Both StorageClasses exist
|
||||
```
|
||||
|
||||
- [ ] **Review PX-CSI logs for errors**
|
||||
```bash
|
||||
# Controller logs
|
||||
kubectl logs -n portworx -l app=px-csi-controller --tail=50
|
||||
|
||||
# Node driver logs (on each node)
|
||||
kubectl logs -n portworx -l app=px-csi-node --tail=50
|
||||
```
|
||||
|
||||
## Phase 6: Functional Testing
|
||||
|
||||
- [ ] **Create test PVC with block storage**
|
||||
```bash
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: test-pure-block
|
||||
namespace: default
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: pure-block
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
EOF
|
||||
```
|
||||
|
||||
- [ ] **Verify PVC bound**
|
||||
```bash
|
||||
kubectl get pvc test-pure-block
|
||||
# Expected: STATUS=Bound
|
||||
```
|
||||
|
||||
- [ ] **Verify PV created on FlashArray**
|
||||
```bash
|
||||
kubectl get pv
|
||||
# Look for PV bound to test-pure-block
|
||||
|
||||
# Check FlashArray UI or API:
|
||||
# Volumes should show new volume created
|
||||
```
|
||||
|
||||
- [ ] **Create test pod using PVC**
|
||||
```bash
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: test-pure-block-pod
|
||||
namespace: default
|
||||
spec:
|
||||
containers:
|
||||
- name: test
|
||||
image: nginx:latest
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: test-pure-block
|
||||
EOF
|
||||
```
|
||||
|
||||
- [ ] **Verify pod is running**
|
||||
```bash
|
||||
kubectl get pod test-pure-block-pod
|
||||
# Expected: STATUS=Running
|
||||
```
|
||||
|
||||
- [ ] **Verify volume mounted in pod**
|
||||
```bash
|
||||
kubectl exec test-pure-block-pod -- df -h /data
|
||||
# Expected: Mount point shows 1Gi volume
|
||||
```
|
||||
|
||||
- [ ] **Test write to volume**
|
||||
```bash
|
||||
kubectl exec test-pure-block-pod -- \
|
||||
sh -c 'echo "test data" > /data/test.txt'
|
||||
|
||||
kubectl exec test-pure-block-pod -- cat /data/test.txt
|
||||
# Expected: "test data"
|
||||
```
|
||||
|
||||
- [ ] **Verify iSCSI session established**
|
||||
```bash
|
||||
# On node running test pod
|
||||
talosctl -n <node-ip> exec -- iscsiadm -m session
|
||||
# Expected: Active iSCSI session to FlashArray
|
||||
```
|
||||
|
||||
- [ ] **Clean up test resources**
|
||||
```bash
|
||||
kubectl delete pod test-pure-block-pod
|
||||
kubectl delete pvc test-pure-block
|
||||
```
|
||||
|
||||
- [ ] **(Optional) Test NFS file storage**
|
||||
- Repeat above steps with `storageClassName: pure-file`
|
||||
- Verify NFS export created on FlashArray
|
||||
|
||||
## Phase 7: Monitoring Setup
|
||||
|
||||
- [ ] **Verify Prometheus ServiceMonitor created**
|
||||
```bash
|
||||
kubectl get servicemonitor -n portworx
|
||||
```
|
||||
|
||||
- [ ] **Check Prometheus targets**
|
||||
- Navigate to Prometheus UI → Targets
|
||||
- Look for `portworx` or `px-csi` targets
|
||||
- Status should be UP
|
||||
|
||||
- [ ] **Query basic metrics**
|
||||
```promql
|
||||
# In Prometheus UI
|
||||
portworx_cluster_status
|
||||
portworx_volume_capacity_bytes
|
||||
```
|
||||
|
||||
- [ ] **(Optional) Create Grafana dashboard**
|
||||
- Import Portworx dashboard from Grafana.com
|
||||
- Or create custom dashboard
|
||||
|
||||
## Phase 8: Documentation & Handoff
|
||||
|
||||
- [ ] **Update homelab documentation**
|
||||
- Add FlashArray details to infrastructure docs
|
||||
- Document storage classes and use cases
|
||||
|
||||
- [ ] **Record credentials securely**
|
||||
- FlashArray management URL: ___________________
|
||||
- API token stored in: 1Password → homelab → pure-flasharray-fastpass
|
||||
- Portworx namespace: `portworx`
|
||||
|
||||
- [ ] **Create runbook for common tasks**
|
||||
- Volume expansion procedure
|
||||
- Snapshot creation/restore
|
||||
- Troubleshooting steps
|
||||
|
||||
- [ ] **Schedule follow-up review**
|
||||
- Date: _______________
|
||||
- Review metrics and performance
|
||||
- Adjust resource limits if needed
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If deployment fails, rollback procedure:
|
||||
|
||||
1. [ ] Delete ArgoCD application:
|
||||
```bash
|
||||
kubectl delete application -n argocd portworx-csi
|
||||
```
|
||||
|
||||
2. [ ] Delete namespace (if needed):
|
||||
```bash
|
||||
kubectl delete namespace portworx
|
||||
```
|
||||
|
||||
3. [ ] Delete StorageClasses:
|
||||
```bash
|
||||
kubectl delete storageclass pure-block pure-file
|
||||
```
|
||||
|
||||
4. [ ] Delete CRDs (if needed):
|
||||
```bash
|
||||
kubectl delete crd storageclusters.core.libopenstorage.org
|
||||
```
|
||||
|
||||
5. [ ] Revert Talos configuration (if needed):
|
||||
```bash
|
||||
# Remove portworx-csi patches from talconfig.yaml
|
||||
talhelper genconfig
|
||||
talosctl apply-config --file clusterconfig/<node>.yaml --nodes <node-ip>
|
||||
```
|
||||
|
||||
## Success Criteria
|
||||
|
||||
Deployment is successful when:
|
||||
|
||||
- [ ] All pods in `portworx` namespace are Running
|
||||
- [ ] StorageClasses `pure-block` and `pure-file` are available
|
||||
- [ ] Test PVC successfully binds to a PV
|
||||
- [ ] Test pod can mount and write to volume
|
||||
- [ ] iSCSI sessions are established on nodes
|
||||
- [ ] No errors in PX-CSI controller/node logs
|
||||
- [ ] Prometheus metrics are being scraped
|
||||
- [ ] ArgoCD application shows Healthy and Synced
|
||||
|
||||
## Notes
|
||||
|
||||
**Deployment Date:** _______________
|
||||
|
||||
**Deployed By:** _______________
|
||||
|
||||
**FlashArray Model:** _______________
|
||||
|
||||
**Purity Version:** _______________
|
||||
|
||||
**Issues Encountered:**
|
||||
-
|
||||
-
|
||||
-
|
||||
|
||||
**Resolutions:**
|
||||
-
|
||||
-
|
||||
-
|
||||
|
||||
**Follow-up Items:**
|
||||
-
|
||||
-
|
||||
-
|
||||
Reference in New Issue
Block a user