rblundon/homelab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Portworx CSI driver for Pure Storage FlashArray

Browse Source 
- Deploy Portworx Operator + CSI driver via ArgoCD
- Support both iSCSI block and NFS file storage from FlashArray
- Integrate with 1Password External Secrets for FlashArray credentials
- Include comprehensive deployment documentation and validation script
- Storage classes: pure-block (iSCSI) and pure-file (NFS)
- Talos Linux compatible with iSCSI/multipath configuration
This commit is contained in:
Hermes Agent service account
2026-06-18 23:08:23 -05:00
parent a4a68eeb5a
commit 459dbc5d18
13 changed files with 2694 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

494
cluster/platform/portworx-csi/README.md Normal file
View File

@@ -0,0 +1,494 @@
# Portworx CSI Driver for Pure Storage FlashArray
GitOps deployment of Portworx CSI driver for Pure Storage FlashArray on Talos Linux in the `fastpass` Kubernetes cluster.
## Overview
Portworx CSI enables Kubernetes to provision and manage storage directly from Pure Storage FlashArray, providing:
- **Block Storage**: High-performance iSCSI volumes for databases and stateful apps
- **File Storage**: NFS-based shared storage from FlashArray File Services
- **Dynamic Provisioning**: Automatic volume creation via StorageClasses
- **Volume Snapshots**: CSI snapshots for backup and restore
- **Volume Expansion**: Online volume resizing without downtime
## Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ Kubernetes Cluster (fastpass) │
│ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ ArgoCD Application (wave 2) │ │
│ │ - Portworx Operator (Helm chart) │ │
│ │ - StorageCluster CR │ │
│ │ - StorageClasses (pure-block, pure-file) │ │
│ │ - ExternalSecret (px-pure-secret) │ │
│ └──────────────────────────────────────────────────┘ │
│ ↓ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ Portworx Operator (namespace: portworx) │ │
│ │ - Watches StorageCluster CR │ │
│ │ - Deploys CSI controller & node drivers │ │
│ └──────────────────────────────────────────────────┘ │
│ ↓ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ PX-CSI Components │ │
│ │ - Controller (Deployment): volume provisioning │ │
│ │ - Node Driver (DaemonSet): volume attachment │ │
│ │ - Monitoring (Prometheus metrics) │ │
│ └──────────────────────────────────────────────────┘ │
│ ↓ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ Talos Nodes (all) │ │
│ │ - iSCSI initiator configured │ │
│ │ - Multipath daemon running │ │
│ │ - udev rules for Pure devices │ │
│ └──────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
↓ iSCSI/NFS
┌─────────────────────────────────────────────────────────────┐
│ Pure Storage FlashArray │
│ - Block volumes (iSCSI) │
│ - File Services (NFS) [optional] │
│ - API management endpoint │
└─────────────────────────────────────────────────────────────┘
```
## Prerequisites
### 1. FlashArray Configuration
#### Create FlashArray User and API Token
1. **Create a storage admin user:**
- Navigate to FlashArray dashboard → **Settings** → **Access** → **Users**
- Click **Create User**
- Set role to **Storage Admin**
- Username: `portworx-csi`
2. **Generate API token:**
- Select the user → **Create API Token**
- Leave **Expires in** blank for non-expiring token
- Copy the token immediately
#### Configure FlashArray for Block Storage (iSCSI)
1. **Verify iSCSI targets are configured:**
```bash
# From any Talos node
iscsiadm -m discovery -t st -p <flasharray-mgmt-ip>
```
2. **Enable FlashArray network interfaces for iSCSI**
#### Configure FlashArray File Services (Optional - for NFS)
If using FlashArray File Services:
1. **Enable File Services** on FlashArray
2. **Create a filesystem** for Kubernetes volumes
3. **Create an NFS policy** with user mapping disabled:
- Edit `nfs-simple` policy or create new policy
- Disable user mapping to avoid UID/GID issues
- Set `no_root_squash` if needed
### 2. 1Password Secret Setup
Create a 1Password item with FlashArray credentials:
**Vault:** `homelab`
**Item Name:** `pure-flasharray-fastpass`
**Field Name:** `pure.json`
**Field Type:** Text
**Field Value:**
```json
{
"FlashArrays": [
{
"MgmtEndPoint": "flasharray.mk-labs.cloud",
"APIToken": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"NFSEndPoint": "flasharray-nfs.mk-labs.cloud"
}
]
}
```
**Notes:**
- Replace `flasharray.mk-labs.cloud` with your FlashArray management endpoint
- Replace the API token with the actual token from FlashArray
- Include `NFSEndPoint` only if using FlashArray File Services
- For IPv6, enclose IP in brackets: `"[2001:db8::1]"`
### 3. Talos Node Configuration
Apply the following patches to **all worker nodes** in `talos/talhelper/talconfig.yaml`:
```yaml
nodes:
# Add to each worker node definition
patches:
- |-
machine:
# Install system extensions
install:
extensions:
- image: ghcr.io/siderolabs/iscsi-tools:latest
- image: ghcr.io/siderolabs/util-linux-tools:latest
# Load kernel modules
kernel:
modules:
- name: iscsi_tcp
- name: dm_multipath
- name: dm_round_robin
# Create multipath.conf
files:
- path: /etc/multipath.conf
permissions: 0644
op: create
content: |
blacklist {
devnode "^pxd[0-9]*"
devnode "^pxd*"
}
defaults {
polling_interval 10
find_multipaths yes
user_friendly_names no
}
devices {
device {
vendor "PURE"
product "FlashArray"
path_selector "service-time 0"
hardware_handler "1 alua"
path_grouping_policy group_by_prio
prio alua
failback immediate
path_checker tur
fast_io_fail_tmo 10
user_friendly_names no
no_path_retry 0
features 0
dev_loss_tmo 600
}
}
# Create udev rules
- path: /etc/udev/rules.d/99-pure-storage.rules
permissions: 0644
op: create
content: |
ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", \
ATTRS{vendor}=="PURE", ATTR{queue/scheduler}="none"
ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", \
ATTRS{vendor}=="PURE", ATTR{queue/nr_requests}="256"
# Enable iSCSI mounts in kubelet
kubelet:
extraMounts:
- destination: /var/lib/iscsi
type: bind
source: /var/lib/iscsi
options:
- bind
- rshared
- rw
```
**Apply Talos configuration:**
```bash
# From city-hall (talhelper host)
cd ~/git/homelab/talos/talhelper
talhelper genconfig
talosctl apply-config --file clusterconfig/<node>.yaml --nodes <node-ip>
# Reboot nodes to load kernel modules
talosctl reboot --nodes <node-ip>
```
### 4. Verify Talos Prerequisites
After applying Talos configuration and rebooting:
```bash
# Check iSCSI initiator service
talosctl -n <node-ip> get service iscsid
# Verify multipath is running
talosctl -n <node-ip> exec -- multipath -ll
# Check initiator name (must be unique per node)
talosctl -n <node-ip> read /etc/iscsi/initiatorname.iscsi
# Verify kernel modules loaded
talosctl -n <node-ip> exec -- lsmod | grep -E "iscsi|dm_multipath"
# Test iSCSI discovery
talosctl -n <node-ip> exec -- iscsiadm -m discovery -t st -p <flasharray-ip>
```
## Deployment
### 1. Commit and Push to Git
```bash
cd ~/git/homelab
git add cluster/platform/portworx-csi/
git commit -m "Add Portworx CSI driver for Pure FlashArray"
git push origin main
```
### 2. Deploy via ArgoCD
The application is configured with `sync-wave: 2`, so it will deploy after External Secrets Operator and 1Password Connect.
**Option A: Auto-sync (recommended)**
```bash
# ArgoCD will automatically sync after detecting changes
# Check status:
kubectl get application -n argocd portworx-csi
```
**Option B: Manual sync**
```bash
# Sync via CLI
argocd app sync portworx-csi
# Or sync via UI
# Navigate to ArgoCD UI → Applications → portworx-csi → Sync
```
### 3. Verify Deployment
```bash
# Check ArgoCD application status
kubectl get application -n argocd portworx-csi
# Check Portworx Operator
kubectl get deployment -n portworx portworx-operator
kubectl get pods -n portworx -l app=portworx-operator
# Check StorageCluster status
kubectl get storagecluster -n portworx px-cluster-fastpass
kubectl describe storagecluster -n portworx px-cluster-fastpass
# Check PX-CSI pods
kubectl get pods -n portworx
# Verify secret was created from 1Password
kubectl get secret -n portworx px-pure-secret
kubectl get secret -n portworx px-pure-secret -o jsonpath='{.data.pure\.json}' | base64 -d | jq
# Check CSI driver registration
kubectl get csidrivers
kubectl get csinode
# Verify StorageClasses
kubectl get storageclass pure-block pure-file
```
Expected output:
```
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION
pure-block pxd.portworx.com Delete Immediate true
pure-file pxd.portworx.com Delete Immediate true
```
## Usage
### Create a PVC with Block Storage
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgres-data
namespace: default
spec:
accessModes:
- ReadWriteOnce
storageClassName: pure-block
resources:
requests:
storage: 10Gi
```
### Create a PVC with File Storage (NFS)
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: shared-data
namespace: default
spec:
accessModes:
- ReadWriteMany
storageClassName: pure-file
resources:
requests:
storage: 50Gi
```
### Volume Expansion
```bash
# Edit PVC to increase size
kubectl patch pvc postgres-data -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'
# Verify expansion
kubectl get pvc postgres-data
```
### Volume Snapshots
```yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: postgres-backup
namespace: default
spec:
volumeSnapshotClassName: px-csi-snapshot-class
source:
persistentVolumeClaimName: postgres-data
```
## StorageClasses
### pure-block
- **Backend:** Pure Storage FlashArray (iSCSI)
- **Access Mode:** ReadWriteOnce (RWO)
- **Use Case:** Databases, stateful applications requiring block storage
- **Features:** High performance, low latency, volume expansion
- **Filesystem:** ext4
### pure-file
- **Backend:** Pure Storage FlashArray File Services (NFS)
- **Access Mode:** ReadWriteMany (RWX)
- **Use Case:** Shared storage for multi-pod applications
- **Features:** NFS v4.1, concurrent access, volume expansion
- **Mount Options:** hard, timeo=600, retrans=3
## Monitoring
Portworx CSI exposes Prometheus metrics:
```bash
# Check ServiceMonitor (if Prometheus Operator is installed)
kubectl get servicemonitor -n portworx
# View metrics endpoint
kubectl get svc -n portworx -l app=portworx-api
# Example metrics:
# - portworx_volume_usage_bytes
# - portworx_volume_capacity_bytes
# - portworx_cluster_status
```
To enable Prometheus scraping, the `kube-prometheus-stack` will auto-discover the ServiceMonitor.
## Troubleshooting
### Pods stuck in ContainerCreating
```bash
# Check PVC status
kubectl get pvc
kubectl describe pvc <pvc-name>
# Check PV binding
kubectl get pv
# Check PX-CSI node driver logs
kubectl logs -n portworx -l app=px-csi-driver --tail=100
# Check iSCSI sessions on node
talosctl -n <node-ip> exec -- iscsiadm -m session
```
### Volume not attaching
```bash
# Check StorageCluster status
kubectl get storagecluster -n portworx -o yaml
# Check PX-CSI controller logs
kubectl logs -n portworx -l app=portworx-operator
# Verify FlashArray connectivity
kubectl exec -n portworx -it <px-pod> -- ping <flasharray-mgmt-ip>
# Check secret is valid
kubectl get secret -n portworx px-pure-secret -o yaml
```
### Multipath issues
```bash
# Check multipath status on node
talosctl -n <node-ip> exec -- multipath -ll
# Restart multipathd if needed
talosctl -n <node-ip> exec -- systemctl restart multipathd
# Verify udev rules applied
talosctl -n <node-ip> exec -- udevadm control --reload-rules
talosctl -n <node-ip> exec -- udevadm trigger
```
### FlashArray API errors
```bash
# Verify API token is valid
# From a pod with curl:
curl -k -H "api-token: <token>" https://<flasharray-mgmt-ip>/api/2.0/arrays
# Check PX-CSI can reach FlashArray
kubectl exec -n portworx -it <px-pod> -- curl -k https://<flasharray-mgmt-ip>
```
## Rollback
If deployment fails:
```bash
# Delete ArgoCD application (keeps namespace and CRDs)
kubectl delete application -n argocd portworx-csi
# Or remove via ArgoCD UI
# Full cleanup (if needed):
kubectl delete namespace portworx
kubectl delete crd storageclusters.core.libopenstorage.org
kubectl delete storageclass pure-block pure-file
```
## References
- [Portworx CSI Documentation](https://docs.portworx.com/portworx-csi/)
- [FlashArray Preparation Guide](https://docs.portworx.com/portworx-csi/install/prepare/flash-array)
- [Talos Linux Storage Guide](https://www.talos.dev/latest/kubernetes-guides/configuration/storage/)
- [Pure Storage Best Practices](https://support.purestorage.com/)
## Architecture Decisions
- **Sync Wave 2:** Deployed after External Secrets Operator (wave 1) but before applications
- **iSCSI Protocol:** Selected for broad compatibility; NVMe-oF can be added later
- **No Default StorageClass:** Keeps existing `nfs-emporium` as default; apps opt-in to Pure
- **Telemetry Disabled:** For homelab privacy; can enable for production support
- **Single FlashArray:** Configuration supports multiple arrays; add to `pure.json` as needed
## Future Enhancements
- [ ] Enable CSI topology for multi-zone deployments
- [ ] Configure NVMe-oF protocol for lower latency
- [ ] Set up FlashArray ActiveCluster for HA
- [ ] Enable Portworx Fusion Controller
- [ ] Configure VLAN binding for isolated storage traffic
- [ ] Implement backup/restore workflows with snapshots
- [ ] Add Grafana dashboards for Portworx metrics