homelab/cluster/platform/portworx-csi

Portworx CSI Driver for Pure Storage FlashArray

GitOps deployment of Portworx CSI driver for Pure Storage FlashArray on Talos Linux in the fastpass Kubernetes cluster.

Overview

Portworx CSI enables Kubernetes to provision and manage storage directly from Pure Storage FlashArray, providing:

• Block Storage: High-performance iSCSI volumes for databases and stateful apps
• File Storage: NFS-based shared storage from FlashArray File Services
• Dynamic Provisioning: Automatic volume creation via StorageClasses
• Volume Snapshots: CSI snapshots for backup and restore
• Volume Expansion: Online volume resizing without downtime

Architecture

┌─────────────────────────────────────────────────────────────┐
│ Kubernetes Cluster (fastpass)                               │
│                                                              │
│  ┌──────────────────────────────────────────────────┐       │
│  │ ArgoCD Application (wave 2)                      │       │
│  │  - Portworx Operator (Helm chart)                │       │
│  │  - StorageCluster CR                             │       │
│  │  - StorageClasses (pure-block, pure-file)        │       │
│  │  - ExternalSecret (px-pure-secret)               │       │
│  └──────────────────────────────────────────────────┘       │
│                    ↓                                         │
│  ┌──────────────────────────────────────────────────┐       │
│  │ Portworx Operator (namespace: portworx)          │       │
│  │  - Watches StorageCluster CR                     │       │
│  │  - Deploys CSI controller & node drivers         │       │
│  └──────────────────────────────────────────────────┘       │
│                    ↓                                         │
│  ┌──────────────────────────────────────────────────┐       │
│  │ PX-CSI Components                                │       │
│  │  - Controller (Deployment): volume provisioning  │       │
│  │  - Node Driver (DaemonSet): volume attachment    │       │
│  │  - Monitoring (Prometheus metrics)               │       │
│  └──────────────────────────────────────────────────┘       │
│                    ↓                                         │
│  ┌──────────────────────────────────────────────────┐       │
│  │ Talos Nodes (all)                                │       │
│  │  - iSCSI initiator configured                    │       │
│  │  - Multipath daemon running                      │       │
│  │  - udev rules for Pure devices                   │       │
│  └──────────────────────────────────────────────────┘       │
└─────────────────────────────────────────────────────────────┘
                    ↓ iSCSI/NFS
┌─────────────────────────────────────────────────────────────┐
│ Pure Storage FlashArray                                     │
│  - Block volumes (iSCSI)                                    │
│  - File Services (NFS) [optional]                           │
│  - API management endpoint                                  │
└─────────────────────────────────────────────────────────────┘

Prerequisites

1. FlashArray Configuration

Create FlashArray User and API Token

1. Create a storage admin user:

   • Navigate to FlashArray dashboard → Settings → Access → Users
   • Click Create User
   • Set role to Storage Admin
   • Username: portworx-csi

2. Generate API token:

   • Select the user → Create API Token
   • Leave Expires in blank for non-expiring token
   • Copy the token immediately

Configure FlashArray for Block Storage (iSCSI)

1. Verify iSCSI targets are configured:

   # From any Talos node
   iscsiadm -m discovery -t st -p <flasharray-mgmt-ip>
   

2. Enable FlashArray network interfaces for iSCSI

Configure FlashArray File Services (Optional - for NFS)

If using FlashArray File Services:

1. Enable File Services on FlashArray
2. Create a filesystem for Kubernetes volumes
3. Create an NFS policy with user mapping disabled:
   • Edit nfs-simple policy or create new policy
   • Disable user mapping to avoid UID/GID issues
   • Set no_root_squash if needed

2. 1Password Secret Setup

Create a 1Password item with FlashArray credentials:

Vault: homelab
Item Name: pure-flasharray-fastpass
Field Name: pure.json
Field Type: Text
Field Value:

{
  "FlashArrays": [
    {
      "MgmtEndPoint": "flasharray.mk-labs.cloud",
      "APIToken": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "NFSEndPoint": "flasharray-nfs.mk-labs.cloud"
    }
  ]
}

Notes:

• Replace flasharray.mk-labs.cloud with your FlashArray management endpoint
• Replace the API token with the actual token from FlashArray
• Include NFSEndPoint only if using FlashArray File Services
• For IPv6, enclose IP in brackets: "[2001:db8::1]"

3. Talos Node Configuration

Apply the following patches to all worker nodes in talos/talhelper/talconfig.yaml:

nodes:
  # Add to each worker node definition
  patches:
    - |-
      machine:
        # Install system extensions
        install:
          extensions:
            - image: ghcr.io/siderolabs/iscsi-tools:latest
            - image: ghcr.io/siderolabs/util-linux-tools:latest
        
        # Load kernel modules
        kernel:
          modules:
            - name: iscsi_tcp
            - name: dm_multipath
            - name: dm_round_robin
        
        # Create multipath.conf
        files:
          - path: /etc/multipath.conf
            permissions: 0644
            op: create
            content: |
              blacklist {
                devnode "^pxd[0-9]*"
                devnode "^pxd*"
              }
              defaults {
                polling_interval 10
                find_multipaths yes
                user_friendly_names no
              }
              devices {
                device {
                  vendor                   "PURE"
                  product                  "FlashArray"
                  path_selector            "service-time 0"
                  hardware_handler         "1 alua"
                  path_grouping_policy     group_by_prio
                  prio                     alua
                  failback                 immediate
                  path_checker             tur
                  fast_io_fail_tmo         10
                  user_friendly_names      no
                  no_path_retry            0
                  features                 0
                  dev_loss_tmo             600
                }
              }
          
          # Create udev rules
          - path: /etc/udev/rules.d/99-pure-storage.rules
            permissions: 0644
            op: create
            content: |
              ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", \
                ATTRS{vendor}=="PURE", ATTR{queue/scheduler}="none"
              ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", \
                ATTRS{vendor}=="PURE", ATTR{queue/nr_requests}="256"
        
        # Enable iSCSI mounts in kubelet
        kubelet:
          extraMounts:
            - destination: /var/lib/iscsi
              type: bind
              source: /var/lib/iscsi
              options:
                - bind
                - rshared
                - rw

Apply Talos configuration:

# From city-hall (talhelper host)
cd ~/git/homelab/talos/talhelper
talhelper genconfig
talosctl apply-config --file clusterconfig/<node>.yaml --nodes <node-ip>

# Reboot nodes to load kernel modules
talosctl reboot --nodes <node-ip>

4. Verify Talos Prerequisites

After applying Talos configuration and rebooting:

# Check iSCSI initiator service
talosctl -n <node-ip> get service iscsid

# Verify multipath is running
talosctl -n <node-ip> exec -- multipath -ll

# Check initiator name (must be unique per node)
talosctl -n <node-ip> read /etc/iscsi/initiatorname.iscsi

# Verify kernel modules loaded
talosctl -n <node-ip> exec -- lsmod | grep -E "iscsi|dm_multipath"

# Test iSCSI discovery
talosctl -n <node-ip> exec -- iscsiadm -m discovery -t st -p <flasharray-ip>

Deployment

1. Commit and Push to Git

cd ~/git/homelab
git add cluster/platform/portworx-csi/
git commit -m "Add Portworx CSI driver for Pure FlashArray"
git push origin main

2. Deploy via ArgoCD

The application is configured with sync-wave: 2, so it will deploy after External Secrets Operator and 1Password Connect.

Option A: Auto-sync (recommended)

# ArgoCD will automatically sync after detecting changes
# Check status:
kubectl get application -n argocd portworx-csi

Option B: Manual sync

# Sync via CLI
argocd app sync portworx-csi

# Or sync via UI
# Navigate to ArgoCD UI → Applications → portworx-csi → Sync

3. Verify Deployment

# Check ArgoCD application status
kubectl get application -n argocd portworx-csi

# Check Portworx Operator
kubectl get deployment -n portworx portworx-operator
kubectl get pods -n portworx -l app=portworx-operator

# Check StorageCluster status
kubectl get storagecluster -n portworx px-cluster-fastpass
kubectl describe storagecluster -n portworx px-cluster-fastpass

# Check PX-CSI pods
kubectl get pods -n portworx

# Verify secret was created from 1Password
kubectl get secret -n portworx px-pure-secret
kubectl get secret -n portworx px-pure-secret -o jsonpath='{.data.pure\.json}' | base64 -d | jq

# Check CSI driver registration
kubectl get csidrivers
kubectl get csinode

# Verify StorageClasses
kubectl get storageclass pure-block pure-file

Expected output:

NAME                          PROVISIONER         RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION
pure-block                    pxd.portworx.com    Delete          Immediate           true
pure-file                     pxd.portworx.com    Delete          Immediate           true

Usage

Create a PVC with Block Storage

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: pure-block
  resources:
    requests:
      storage: 10Gi

Create a PVC with File Storage (NFS)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-data
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: pure-file
  resources:
    requests:
      storage: 50Gi

Volume Expansion

# Edit PVC to increase size
kubectl patch pvc postgres-data -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

# Verify expansion
kubectl get pvc postgres-data

Volume Snapshots

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: postgres-backup
  namespace: default
spec:
  volumeSnapshotClassName: px-csi-snapshot-class
  source:
    persistentVolumeClaimName: postgres-data

StorageClasses

pure-block

• Backend: Pure Storage FlashArray (iSCSI)
• Access Mode: ReadWriteOnce (RWO)
• Use Case: Databases, stateful applications requiring block storage
• Features: High performance, low latency, volume expansion
• Filesystem: ext4

pure-file

• Backend: Pure Storage FlashArray File Services (NFS)
• Access Mode: ReadWriteMany (RWX)
• Use Case: Shared storage for multi-pod applications
• Features: NFS v4.1, concurrent access, volume expansion
• Mount Options: hard, timeo=600, retrans=3

Monitoring

Portworx CSI exposes Prometheus metrics:

# Check ServiceMonitor (if Prometheus Operator is installed)
kubectl get servicemonitor -n portworx

# View metrics endpoint
kubectl get svc -n portworx -l app=portworx-api

# Example metrics:
# - portworx_volume_usage_bytes
# - portworx_volume_capacity_bytes
# - portworx_cluster_status

To enable Prometheus scraping, the kube-prometheus-stack will auto-discover the ServiceMonitor.

Troubleshooting

Pods stuck in ContainerCreating

# Check PVC status
kubectl get pvc
kubectl describe pvc <pvc-name>

# Check PV binding
kubectl get pv

# Check PX-CSI node driver logs
kubectl logs -n portworx -l app=px-csi-driver --tail=100

# Check iSCSI sessions on node
talosctl -n <node-ip> exec -- iscsiadm -m session

Volume not attaching

# Check StorageCluster status
kubectl get storagecluster -n portworx -o yaml

# Check PX-CSI controller logs
kubectl logs -n portworx -l app=portworx-operator

# Verify FlashArray connectivity
kubectl exec -n portworx -it <px-pod> -- ping <flasharray-mgmt-ip>

# Check secret is valid
kubectl get secret -n portworx px-pure-secret -o yaml

Multipath issues

# Check multipath status on node
talosctl -n <node-ip> exec -- multipath -ll

# Restart multipathd if needed
talosctl -n <node-ip> exec -- systemctl restart multipathd

# Verify udev rules applied
talosctl -n <node-ip> exec -- udevadm control --reload-rules
talosctl -n <node-ip> exec -- udevadm trigger

FlashArray API errors

# Verify API token is valid
# From a pod with curl:
curl -k -H "api-token: <token>" https://<flasharray-mgmt-ip>/api/2.0/arrays

# Check PX-CSI can reach FlashArray
kubectl exec -n portworx -it <px-pod> -- curl -k https://<flasharray-mgmt-ip>

Rollback

If deployment fails:

# Delete ArgoCD application (keeps namespace and CRDs)
kubectl delete application -n argocd portworx-csi

# Or remove via ArgoCD UI

# Full cleanup (if needed):
kubectl delete namespace portworx
kubectl delete crd storageclusters.core.libopenstorage.org
kubectl delete storageclass pure-block pure-file

References

• Portworx CSI Documentation
• FlashArray Preparation Guide
• Talos Linux Storage Guide
• Pure Storage Best Practices

Architecture Decisions

• Sync Wave 2: Deployed after External Secrets Operator (wave 1) but before applications
• iSCSI Protocol: Selected for broad compatibility; NVMe-oF can be added later
• No Default StorageClass: Keeps existing nfs-emporium as default; apps opt-in to Pure
• Telemetry Disabled: For homelab privacy; can enable for production support
• Single FlashArray: Configuration supports multiple arrays; add to pure.json as needed

Future Enhancements

• Enable CSI topology for multi-zone deployments
• Configure NVMe-oF protocol for lower latency
• Set up FlashArray ActiveCluster for HA
• Enable Portworx Fusion Controller
• Configure VLAN binding for isolated storage traffic
• Implement backup/restore workflows with snapshots
• Add Grafana dashboards for Portworx metrics