homelab/cluster/platform/portworx-csi/DEPLOYMENT-CHECKLIST.md

Portworx CSI Deployment Checklist

Pre-deployment checklist for Portworx CSI driver on Talos Linux with Pure Storage FlashArray.

Phase 1: FlashArray Preparation

• Create FlashArray storage admin user

  • Username: portworx-csi
  • Role: Storage Admin
  • Password: Set in FlashArray UI

• Generate FlashArray API token

  • User: portworx-csi
  • Expiration: None (non-expiring)
  • Token format: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
  • Save token immediately (cannot be retrieved later)

• Verify FlashArray management endpoint

  • Test connectivity: ping flasharray.mk-labs.cloud
  • Test HTTPS: curl -k https://flasharray.mk-labs.cloud
  • Record IP/hostname: _______________________

• Configure iSCSI targets on FlashArray

  • Navigate to: Settings → Network
  • Verify iSCSI interfaces are enabled
  • Test discovery from Talos node:

    talosctl -n <node> exec -- iscsiadm -m discovery -t st -p <flasharray-ip>
    

• (Optional) Enable FlashArray File Services for NFS

  • Enable File Services: Settings → File Services
  • Create filesystem: fastpass-nfs
  • Create NFS policy with user mapping disabled
  • Record NFS endpoint: _______________________

Phase 2: 1Password Secret Configuration

• Create 1Password item

  • Vault: homelab
  • Item name: pure-flasharray-fastpass
  • Category: Secure Note or API Credential

• Add pure.json field to 1Password item

  • Field name: pure.json
  • Field type: Text
  • Content format: JSON (see below)

• Populate pure.json content

  {
    "FlashArrays": [
      {
        "MgmtEndPoint": "flasharray.mk-labs.cloud",
        "APIToken": "paste-api-token-here",
        "NFSEndPoint": "flasharray-nfs.mk-labs.cloud"
      }
    ]
  }
  

  • Replace flasharray.mk-labs.cloud with actual endpoint
  • Replace paste-api-token-here with FlashArray API token
  • Remove NFSEndPoint line if not using File Services

• Validate JSON syntax

  # Copy content from 1Password and validate
  echo '<paste-json>' | jq .
  

• Verify 1Password Connect can access item

  # From a pod with op CLI or via 1Password UI
  # Confirm item exists in homelab vault
  

Phase 3: Talos Node Configuration

• Edit talconfig.yaml

  • File: ~/git/homelab/talos/talhelper/talconfig.yaml
  • Add patches from cluster/platform/portworx-csi/talos-config-reference.yaml
  • Sections to add:
    • System extensions (iscsi-tools, util-linux-tools)
    • Kernel modules (iscsi_tcp, dm_multipath, dm_round_robin)
    • /etc/multipath.conf file
    • /etc/udev/rules.d/99-pure-storage.rules file
    • kubelet extraMounts for /var/lib/iscsi

• Generate updated Talos configs

  cd ~/git/homelab/talos/talhelper
  talhelper genconfig
  

• Apply Talos configuration to all worker nodes

  # For each worker node:
  talosctl apply-config \
    --file clusterconfig/<node-name>.yaml \
    --nodes <node-ip>
  

  Nodes:

  • Node 1: _________________ (IP: _____________)
  • Node 2: _________________ (IP: _____________)
  • Node 3: _________________ (IP: _____________)

• Reboot all worker nodes

  # Reboot nodes one at a time to load kernel modules
  talosctl reboot --nodes <node-ip>
  
  # Wait for node to come back online
  kubectl get nodes
  

• Verify iSCSI initiator on each node

  talosctl -n <node-ip> get service iscsid
  # Expected: iscsid service running
  

• Verify multipath on each node

  talosctl -n <node-ip> exec -- multipath -ll
  # Expected: multipath running (may show no paths yet)
  

• Verify unique iSCSI initiator names

  # For each node:
  talosctl -n <node-ip> read /etc/iscsi/initiatorname.iscsi
  
  # Record initiator names (must be unique):
  # Node 1: iqn.1994-05.com.redhat:_______________
  # Node 2: iqn.1994-05.com.redhat:_______________
  # Node 3: iqn.1994-05.com.redhat:_______________
  

• Verify kernel modules loaded

  talosctl -n <node-ip> exec -- lsmod | grep -E "iscsi|dm_multipath"
  # Expected: iscsi_tcp, dm_multipath, dm_round_robin
  

• Test iSCSI discovery from each node

  talosctl -n <node-ip> exec -- \
    iscsiadm -m discovery -t st -p <flasharray-ip>
  # Expected: List of iSCSI targets from FlashArray
  

Phase 4: GitOps Deployment

• Review manifests

  • cluster/platform/portworx-csi/application.yaml
  • cluster/platform/portworx-csi/operator-values.yaml
  • cluster/platform/portworx-csi/storagecluster.yaml
  • cluster/platform/portworx-csi/externalsecret.yaml
  • cluster/platform/portworx-csi/storageclass-block.yaml
  • cluster/platform/portworx-csi/storageclass-file.yaml

• Customize for your environment

  • Update FlashArray endpoints if different from defaults
  • Adjust resource limits if needed
  • Review sync wave (currently: 2)

• Commit to Git

  cd ~/git/homelab
  git add cluster/platform/portworx-csi/
  git commit -m "Add Portworx CSI driver for Pure FlashArray"
  git push origin main
  

• Verify ArgoCD auto-sync or trigger manual sync

  # Check ArgoCD application status
  kubectl get application -n argocd portworx-csi
  
  # Manual sync if needed
  argocd app sync portworx-csi
  

Phase 5: Deployment Verification

• Verify ArgoCD application is healthy

  kubectl get application -n argocd portworx-csi
  # Expected: STATUS=Synced, HEALTH=Healthy
  

• Check namespace created

  kubectl get namespace portworx
  

• Verify ExternalSecret synced

  kubectl get externalsecret -n portworx px-pure-secret
  # Expected: STATUS=SecretSynced
  
  kubectl get secret -n portworx px-pure-secret
  # Expected: Secret exists with pure.json key
  

• Validate secret content

  kubectl get secret -n portworx px-pure-secret \
    -o jsonpath='{.data.pure\.json}' | base64 -d | jq .
  # Expected: Valid JSON with FlashArray config
  

• Check Portworx Operator deployed

  kubectl get deployment -n portworx portworx-operator
  kubectl get pods -n portworx -l app=portworx-operator
  # Expected: 1/1 pods running
  

• Verify StorageCluster created

  kubectl get storagecluster -n portworx px-cluster-fastpass
  # Expected: STATUS=Running or Initializing
  

• Check PX-CSI pods running

  kubectl get pods -n portworx
  # Expected pods:
  # - portworx-operator-...
  # - px-csi-controller-...
  # - px-csi-node-... (one per node)
  

• Verify CSI driver registered

  kubectl get csidrivers
  # Expected: pxd.portworx.com
  
  kubectl get csinode
  # Expected: One CSINode per Kubernetes node
  

• Check StorageClasses created

  kubectl get storageclass pure-block pure-file
  # Expected: Both StorageClasses exist
  

• Review PX-CSI logs for errors

  # Controller logs
  kubectl logs -n portworx -l app=px-csi-controller --tail=50
  
  # Node driver logs (on each node)
  kubectl logs -n portworx -l app=px-csi-node --tail=50
  

Phase 6: Functional Testing

• Create test PVC with block storage

  cat <<EOF | kubectl apply -f -
  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: test-pure-block
    namespace: default
  spec:
    accessModes:
      - ReadWriteOnce
    storageClassName: pure-block
    resources:
      requests:
        storage: 1Gi
  EOF
  

• Verify PVC bound

  kubectl get pvc test-pure-block
  # Expected: STATUS=Bound
  

• Verify PV created on FlashArray

  kubectl get pv
  # Look for PV bound to test-pure-block
  
  # Check FlashArray UI or API:
  # Volumes should show new volume created
  

• Create test pod using PVC

  cat <<EOF | kubectl apply -f -
  apiVersion: v1
  kind: Pod
  metadata:
    name: test-pure-block-pod
    namespace: default
  spec:
    containers:
    - name: test
      image: nginx:latest
      volumeMounts:
      - name: data
        mountPath: /data
    volumes:
    - name: data
      persistentVolumeClaim:
        claimName: test-pure-block
  EOF
  

• Verify pod is running

  kubectl get pod test-pure-block-pod
  # Expected: STATUS=Running
  

• Verify volume mounted in pod

  kubectl exec test-pure-block-pod -- df -h /data
  # Expected: Mount point shows 1Gi volume
  

• Test write to volume

  kubectl exec test-pure-block-pod -- \
    sh -c 'echo "test data" > /data/test.txt'
  
  kubectl exec test-pure-block-pod -- cat /data/test.txt
  # Expected: "test data"
  

• Verify iSCSI session established

  # On node running test pod
  talosctl -n <node-ip> exec -- iscsiadm -m session
  # Expected: Active iSCSI session to FlashArray
  

• Clean up test resources

  kubectl delete pod test-pure-block-pod
  kubectl delete pvc test-pure-block
  

• (Optional) Test NFS file storage

  • Repeat above steps with storageClassName: pure-file
  • Verify NFS export created on FlashArray

Phase 7: Monitoring Setup

• Verify Prometheus ServiceMonitor created

  kubectl get servicemonitor -n portworx
  

• Check Prometheus targets

  • Navigate to Prometheus UI → Targets
  • Look for portworx or px-csi targets
  • Status should be UP

• Query basic metrics

  # In Prometheus UI
  portworx_cluster_status
  portworx_volume_capacity_bytes
  

• (Optional) Create Grafana dashboard

  • Import Portworx dashboard from Grafana.com
  • Or create custom dashboard

Phase 8: Documentation & Handoff

• Update homelab documentation

  • Add FlashArray details to infrastructure docs
  • Document storage classes and use cases

• Record credentials securely

  • FlashArray management URL: ___________________
  • API token stored in: 1Password → homelab → pure-flasharray-fastpass
  • Portworx namespace: portworx

• Create runbook for common tasks

  • Volume expansion procedure
  • Snapshot creation/restore
  • Troubleshooting steps

• Schedule follow-up review

  • Date: _______________
  • Review metrics and performance
  • Adjust resource limits if needed

Rollback Plan

If deployment fails, rollback procedure:

1. Delete ArgoCD application:

   kubectl delete application -n argocd portworx-csi
   

2. Delete namespace (if needed):

   kubectl delete namespace portworx
   

3. Delete StorageClasses:

   kubectl delete storageclass pure-block pure-file
   

4. Delete CRDs (if needed):

   kubectl delete crd storageclusters.core.libopenstorage.org
   

5. Revert Talos configuration (if needed):

   # Remove portworx-csi patches from talconfig.yaml
   talhelper genconfig
   talosctl apply-config --file clusterconfig/<node>.yaml --nodes <node-ip>
   

Success Criteria

Deployment is successful when:

• All pods in portworx namespace are Running
• StorageClasses pure-block and pure-file are available
• Test PVC successfully binds to a PV
• Test pod can mount and write to volume
• iSCSI sessions are established on nodes
• No errors in PX-CSI controller/node logs
• Prometheus metrics are being scraped
• ArgoCD application shows Healthy and Synced

Notes

Deployment Date: _______________

Deployed By: _______________

FlashArray Model: _______________

Purity Version: _______________

Issues Encountered:

Resolutions:

Follow-up Items: