# Portworx CSI Driver for Pure Storage FlashArray GitOps deployment of Portworx CSI driver for Pure Storage FlashArray on Talos Linux in the `fastpass` Kubernetes cluster. ## Overview Portworx CSI enables Kubernetes to provision and manage storage directly from Pure Storage FlashArray, providing: - **Block Storage**: High-performance iSCSI volumes for databases and stateful apps - **File Storage**: NFS-based shared storage from FlashArray File Services - **Dynamic Provisioning**: Automatic volume creation via StorageClasses - **Volume Snapshots**: CSI snapshots for backup and restore - **Volume Expansion**: Online volume resizing without downtime ## Architecture ``` ┌─────────────────────────────────────────────────────────────┐ │ Kubernetes Cluster (fastpass) │ │ │ │ ┌──────────────────────────────────────────────────┐ │ │ │ ArgoCD Application (wave 2) │ │ │ │ - Portworx Operator (Helm chart) │ │ │ │ - StorageCluster CR │ │ │ │ - StorageClasses (pure-block, pure-file) │ │ │ │ - ExternalSecret (px-pure-secret) │ │ │ └──────────────────────────────────────────────────┘ │ │ ↓ │ │ ┌──────────────────────────────────────────────────┐ │ │ │ Portworx Operator (namespace: portworx) │ │ │ │ - Watches StorageCluster CR │ │ │ │ - Deploys CSI controller & node drivers │ │ │ └──────────────────────────────────────────────────┘ │ │ ↓ │ │ ┌──────────────────────────────────────────────────┐ │ │ │ PX-CSI Components │ │ │ │ - Controller (Deployment): volume provisioning │ │ │ │ - Node Driver (DaemonSet): volume attachment │ │ │ │ - Monitoring (Prometheus metrics) │ │ │ └──────────────────────────────────────────────────┘ │ │ ↓ │ │ ┌──────────────────────────────────────────────────┐ │ │ │ Talos Nodes (all) │ │ │ │ - iSCSI initiator configured │ │ │ │ - Multipath daemon running │ │ │ │ - udev rules for Pure devices │ │ │ └──────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────┘ ↓ iSCSI/NFS ┌─────────────────────────────────────────────────────────────┐ │ Pure Storage FlashArray │ │ - Block volumes (iSCSI) │ │ - File Services (NFS) [optional] │ │ - API management endpoint │ └─────────────────────────────────────────────────────────────┘ ``` ## Prerequisites ### 1. FlashArray Configuration #### Create FlashArray User and API Token 1. **Create a storage admin user:** - Navigate to FlashArray dashboard → **Settings** → **Access** → **Users** - Click **Create User** - Set role to **Storage Admin** - Username: `portworx-csi` 2. **Generate API token:** - Select the user → **Create API Token** - Leave **Expires in** blank for non-expiring token - Copy the token immediately #### Configure FlashArray for Block Storage (iSCSI) 1. **Verify iSCSI targets are configured:** ```bash # From any Talos node iscsiadm -m discovery -t st -p ``` 2. **Enable FlashArray network interfaces for iSCSI** #### Configure FlashArray File Services (Optional - for NFS) If using FlashArray File Services: 1. **Enable File Services** on FlashArray 2. **Create a filesystem** for Kubernetes volumes 3. **Create an NFS policy** with user mapping disabled: - Edit `nfs-simple` policy or create new policy - Disable user mapping to avoid UID/GID issues - Set `no_root_squash` if needed ### 2. 1Password Secret Setup Create a 1Password item with FlashArray credentials: **Vault:** `homelab` **Item Name:** `pure-flasharray-fastpass` **Field Name:** `pure.json` **Field Type:** Text **Field Value:** ```json { "FlashArrays": [ { "MgmtEndPoint": "flasharray.mk-labs.cloud", "APIToken": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "NFSEndPoint": "flasharray-nfs.mk-labs.cloud" } ] } ``` **Notes:** - Replace `flasharray.mk-labs.cloud` with your FlashArray management endpoint - Replace the API token with the actual token from FlashArray - Include `NFSEndPoint` only if using FlashArray File Services - For IPv6, enclose IP in brackets: `"[2001:db8::1]"` ### 3. Talos Node Configuration Apply the following patches to **all worker nodes** in `talos/talhelper/talconfig.yaml`: ```yaml nodes: # Add to each worker node definition patches: - |- machine: # Install system extensions install: extensions: - image: ghcr.io/siderolabs/iscsi-tools:latest - image: ghcr.io/siderolabs/util-linux-tools:latest # Load kernel modules kernel: modules: - name: iscsi_tcp - name: dm_multipath - name: dm_round_robin # Create multipath.conf files: - path: /etc/multipath.conf permissions: 0644 op: create content: | blacklist { devnode "^pxd[0-9]*" devnode "^pxd*" } defaults { polling_interval 10 find_multipaths yes user_friendly_names no } devices { device { vendor "PURE" product "FlashArray" path_selector "service-time 0" hardware_handler "1 alua" path_grouping_policy group_by_prio prio alua failback immediate path_checker tur fast_io_fail_tmo 10 user_friendly_names no no_path_retry 0 features 0 dev_loss_tmo 600 } } # Create udev rules - path: /etc/udev/rules.d/99-pure-storage.rules permissions: 0644 op: create content: | ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", \ ATTRS{vendor}=="PURE", ATTR{queue/scheduler}="none" ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", \ ATTRS{vendor}=="PURE", ATTR{queue/nr_requests}="256" # Enable iSCSI mounts in kubelet kubelet: extraMounts: - destination: /var/lib/iscsi type: bind source: /var/lib/iscsi options: - bind - rshared - rw ``` **Apply Talos configuration:** ```bash # From city-hall (talhelper host) cd ~/git/homelab/talos/talhelper talhelper genconfig talosctl apply-config --file clusterconfig/.yaml --nodes # Reboot nodes to load kernel modules talosctl reboot --nodes ``` ### 4. Verify Talos Prerequisites After applying Talos configuration and rebooting: ```bash # Check iSCSI initiator service talosctl -n get service iscsid # Verify multipath is running talosctl -n exec -- multipath -ll # Check initiator name (must be unique per node) talosctl -n read /etc/iscsi/initiatorname.iscsi # Verify kernel modules loaded talosctl -n exec -- lsmod | grep -E "iscsi|dm_multipath" # Test iSCSI discovery talosctl -n exec -- iscsiadm -m discovery -t st -p ``` ## Deployment ### 1. Commit and Push to Git ```bash cd ~/git/homelab git add cluster/platform/portworx-csi/ git commit -m "Add Portworx CSI driver for Pure FlashArray" git push origin main ``` ### 2. Deploy via ArgoCD The application is configured with `sync-wave: 2`, so it will deploy after External Secrets Operator and 1Password Connect. **Option A: Auto-sync (recommended)** ```bash # ArgoCD will automatically sync after detecting changes # Check status: kubectl get application -n argocd portworx-csi ``` **Option B: Manual sync** ```bash # Sync via CLI argocd app sync portworx-csi # Or sync via UI # Navigate to ArgoCD UI → Applications → portworx-csi → Sync ``` ### 3. Verify Deployment ```bash # Check ArgoCD application status kubectl get application -n argocd portworx-csi # Check Portworx Operator kubectl get deployment -n portworx portworx-operator kubectl get pods -n portworx -l app=portworx-operator # Check StorageCluster status kubectl get storagecluster -n portworx px-cluster-fastpass kubectl describe storagecluster -n portworx px-cluster-fastpass # Check PX-CSI pods kubectl get pods -n portworx # Verify secret was created from 1Password kubectl get secret -n portworx px-pure-secret kubectl get secret -n portworx px-pure-secret -o jsonpath='{.data.pure\.json}' | base64 -d | jq # Check CSI driver registration kubectl get csidrivers kubectl get csinode # Verify StorageClasses kubectl get storageclass pure-block pure-file ``` Expected output: ``` NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION pure-block pxd.portworx.com Delete Immediate true pure-file pxd.portworx.com Delete Immediate true ``` ## Usage ### Create a PVC with Block Storage ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: postgres-data namespace: default spec: accessModes: - ReadWriteOnce storageClassName: pure-block resources: requests: storage: 10Gi ``` ### Create a PVC with File Storage (NFS) ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: shared-data namespace: default spec: accessModes: - ReadWriteMany storageClassName: pure-file resources: requests: storage: 50Gi ``` ### Volume Expansion ```bash # Edit PVC to increase size kubectl patch pvc postgres-data -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}' # Verify expansion kubectl get pvc postgres-data ``` ### Volume Snapshots ```yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: postgres-backup namespace: default spec: volumeSnapshotClassName: px-csi-snapshot-class source: persistentVolumeClaimName: postgres-data ``` ## StorageClasses ### pure-block - **Backend:** Pure Storage FlashArray (iSCSI) - **Access Mode:** ReadWriteOnce (RWO) - **Use Case:** Databases, stateful applications requiring block storage - **Features:** High performance, low latency, volume expansion - **Filesystem:** ext4 ### pure-file - **Backend:** Pure Storage FlashArray File Services (NFS) - **Access Mode:** ReadWriteMany (RWX) - **Use Case:** Shared storage for multi-pod applications - **Features:** NFS v4.1, concurrent access, volume expansion - **Mount Options:** hard, timeo=600, retrans=3 ## Monitoring Portworx CSI exposes Prometheus metrics: ```bash # Check ServiceMonitor (if Prometheus Operator is installed) kubectl get servicemonitor -n portworx # View metrics endpoint kubectl get svc -n portworx -l app=portworx-api # Example metrics: # - portworx_volume_usage_bytes # - portworx_volume_capacity_bytes # - portworx_cluster_status ``` To enable Prometheus scraping, the `kube-prometheus-stack` will auto-discover the ServiceMonitor. ## Troubleshooting ### Pods stuck in ContainerCreating ```bash # Check PVC status kubectl get pvc kubectl describe pvc # Check PV binding kubectl get pv # Check PX-CSI node driver logs kubectl logs -n portworx -l app=px-csi-driver --tail=100 # Check iSCSI sessions on node talosctl -n exec -- iscsiadm -m session ``` ### Volume not attaching ```bash # Check StorageCluster status kubectl get storagecluster -n portworx -o yaml # Check PX-CSI controller logs kubectl logs -n portworx -l app=portworx-operator # Verify FlashArray connectivity kubectl exec -n portworx -it -- ping # Check secret is valid kubectl get secret -n portworx px-pure-secret -o yaml ``` ### Multipath issues ```bash # Check multipath status on node talosctl -n exec -- multipath -ll # Restart multipathd if needed talosctl -n exec -- systemctl restart multipathd # Verify udev rules applied talosctl -n exec -- udevadm control --reload-rules talosctl -n exec -- udevadm trigger ``` ### FlashArray API errors ```bash # Verify API token is valid # From a pod with curl: curl -k -H "api-token: " https:///api/2.0/arrays # Check PX-CSI can reach FlashArray kubectl exec -n portworx -it -- curl -k https:// ``` ## Rollback If deployment fails: ```bash # Delete ArgoCD application (keeps namespace and CRDs) kubectl delete application -n argocd portworx-csi # Or remove via ArgoCD UI # Full cleanup (if needed): kubectl delete namespace portworx kubectl delete crd storageclusters.core.libopenstorage.org kubectl delete storageclass pure-block pure-file ``` ## References - [Portworx CSI Documentation](https://docs.portworx.com/portworx-csi/) - [FlashArray Preparation Guide](https://docs.portworx.com/portworx-csi/install/prepare/flash-array) - [Talos Linux Storage Guide](https://www.talos.dev/latest/kubernetes-guides/configuration/storage/) - [Pure Storage Best Practices](https://support.purestorage.com/) ## Architecture Decisions - **Sync Wave 2:** Deployed after External Secrets Operator (wave 1) but before applications - **iSCSI Protocol:** Selected for broad compatibility; NVMe-oF can be added later - **No Default StorageClass:** Keeps existing `nfs-emporium` as default; apps opt-in to Pure - **Telemetry Disabled:** For homelab privacy; can enable for production support - **Single FlashArray:** Configuration supports multiple arrays; add to `pure.json` as needed ## Future Enhancements - [ ] Enable CSI topology for multi-zone deployments - [ ] Configure NVMe-oF protocol for lower latency - [ ] Set up FlashArray ActiveCluster for HA - [ ] Enable Portworx Fusion Controller - [ ] Configure VLAN binding for isolated storage traffic - [ ] Implement backup/restore workflows with snapshots - [ ] Add Grafana dashboards for Portworx metrics