459dbc5d18
- Deploy Portworx Operator + CSI driver via ArgoCD - Support both iSCSI block and NFS file storage from FlashArray - Integrate with 1Password External Secrets for FlashArray credentials - Include comprehensive deployment documentation and validation script - Storage classes: pure-block (iSCSI) and pure-file (NFS) - Talos Linux compatible with iSCSI/multipath configuration
495 lines
16 KiB
Markdown
495 lines
16 KiB
Markdown
# Portworx CSI Driver for Pure Storage FlashArray
|
|
|
|
GitOps deployment of Portworx CSI driver for Pure Storage FlashArray on Talos Linux in the `fastpass` Kubernetes cluster.
|
|
|
|
## Overview
|
|
|
|
Portworx CSI enables Kubernetes to provision and manage storage directly from Pure Storage FlashArray, providing:
|
|
|
|
- **Block Storage**: High-performance iSCSI volumes for databases and stateful apps
|
|
- **File Storage**: NFS-based shared storage from FlashArray File Services
|
|
- **Dynamic Provisioning**: Automatic volume creation via StorageClasses
|
|
- **Volume Snapshots**: CSI snapshots for backup and restore
|
|
- **Volume Expansion**: Online volume resizing without downtime
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Kubernetes Cluster (fastpass) │
|
|
│ │
|
|
│ ┌──────────────────────────────────────────────────┐ │
|
|
│ │ ArgoCD Application (wave 2) │ │
|
|
│ │ - Portworx Operator (Helm chart) │ │
|
|
│ │ - StorageCluster CR │ │
|
|
│ │ - StorageClasses (pure-block, pure-file) │ │
|
|
│ │ - ExternalSecret (px-pure-secret) │ │
|
|
│ └──────────────────────────────────────────────────┘ │
|
|
│ ↓ │
|
|
│ ┌──────────────────────────────────────────────────┐ │
|
|
│ │ Portworx Operator (namespace: portworx) │ │
|
|
│ │ - Watches StorageCluster CR │ │
|
|
│ │ - Deploys CSI controller & node drivers │ │
|
|
│ └──────────────────────────────────────────────────┘ │
|
|
│ ↓ │
|
|
│ ┌──────────────────────────────────────────────────┐ │
|
|
│ │ PX-CSI Components │ │
|
|
│ │ - Controller (Deployment): volume provisioning │ │
|
|
│ │ - Node Driver (DaemonSet): volume attachment │ │
|
|
│ │ - Monitoring (Prometheus metrics) │ │
|
|
│ └──────────────────────────────────────────────────┘ │
|
|
│ ↓ │
|
|
│ ┌──────────────────────────────────────────────────┐ │
|
|
│ │ Talos Nodes (all) │ │
|
|
│ │ - iSCSI initiator configured │ │
|
|
│ │ - Multipath daemon running │ │
|
|
│ │ - udev rules for Pure devices │ │
|
|
│ └──────────────────────────────────────────────────┘ │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
↓ iSCSI/NFS
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Pure Storage FlashArray │
|
|
│ - Block volumes (iSCSI) │
|
|
│ - File Services (NFS) [optional] │
|
|
│ - API management endpoint │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Prerequisites
|
|
|
|
### 1. FlashArray Configuration
|
|
|
|
#### Create FlashArray User and API Token
|
|
|
|
1. **Create a storage admin user:**
|
|
- Navigate to FlashArray dashboard → **Settings** → **Access** → **Users**
|
|
- Click **Create User**
|
|
- Set role to **Storage Admin**
|
|
- Username: `portworx-csi`
|
|
|
|
2. **Generate API token:**
|
|
- Select the user → **Create API Token**
|
|
- Leave **Expires in** blank for non-expiring token
|
|
- Copy the token immediately
|
|
|
|
#### Configure FlashArray for Block Storage (iSCSI)
|
|
|
|
1. **Verify iSCSI targets are configured:**
|
|
```bash
|
|
# From any Talos node
|
|
iscsiadm -m discovery -t st -p <flasharray-mgmt-ip>
|
|
```
|
|
|
|
2. **Enable FlashArray network interfaces for iSCSI**
|
|
|
|
#### Configure FlashArray File Services (Optional - for NFS)
|
|
|
|
If using FlashArray File Services:
|
|
|
|
1. **Enable File Services** on FlashArray
|
|
2. **Create a filesystem** for Kubernetes volumes
|
|
3. **Create an NFS policy** with user mapping disabled:
|
|
- Edit `nfs-simple` policy or create new policy
|
|
- Disable user mapping to avoid UID/GID issues
|
|
- Set `no_root_squash` if needed
|
|
|
|
### 2. 1Password Secret Setup
|
|
|
|
Create a 1Password item with FlashArray credentials:
|
|
|
|
**Vault:** `homelab`
|
|
**Item Name:** `pure-flasharray-fastpass`
|
|
**Field Name:** `pure.json`
|
|
**Field Type:** Text
|
|
**Field Value:**
|
|
```json
|
|
{
|
|
"FlashArrays": [
|
|
{
|
|
"MgmtEndPoint": "flasharray.mk-labs.cloud",
|
|
"APIToken": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
|
|
"NFSEndPoint": "flasharray-nfs.mk-labs.cloud"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
**Notes:**
|
|
- Replace `flasharray.mk-labs.cloud` with your FlashArray management endpoint
|
|
- Replace the API token with the actual token from FlashArray
|
|
- Include `NFSEndPoint` only if using FlashArray File Services
|
|
- For IPv6, enclose IP in brackets: `"[2001:db8::1]"`
|
|
|
|
### 3. Talos Node Configuration
|
|
|
|
Apply the following patches to **all worker nodes** in `talos/talhelper/talconfig.yaml`:
|
|
|
|
```yaml
|
|
nodes:
|
|
# Add to each worker node definition
|
|
patches:
|
|
- |-
|
|
machine:
|
|
# Install system extensions
|
|
install:
|
|
extensions:
|
|
- image: ghcr.io/siderolabs/iscsi-tools:latest
|
|
- image: ghcr.io/siderolabs/util-linux-tools:latest
|
|
|
|
# Load kernel modules
|
|
kernel:
|
|
modules:
|
|
- name: iscsi_tcp
|
|
- name: dm_multipath
|
|
- name: dm_round_robin
|
|
|
|
# Create multipath.conf
|
|
files:
|
|
- path: /etc/multipath.conf
|
|
permissions: 0644
|
|
op: create
|
|
content: |
|
|
blacklist {
|
|
devnode "^pxd[0-9]*"
|
|
devnode "^pxd*"
|
|
}
|
|
defaults {
|
|
polling_interval 10
|
|
find_multipaths yes
|
|
user_friendly_names no
|
|
}
|
|
devices {
|
|
device {
|
|
vendor "PURE"
|
|
product "FlashArray"
|
|
path_selector "service-time 0"
|
|
hardware_handler "1 alua"
|
|
path_grouping_policy group_by_prio
|
|
prio alua
|
|
failback immediate
|
|
path_checker tur
|
|
fast_io_fail_tmo 10
|
|
user_friendly_names no
|
|
no_path_retry 0
|
|
features 0
|
|
dev_loss_tmo 600
|
|
}
|
|
}
|
|
|
|
# Create udev rules
|
|
- path: /etc/udev/rules.d/99-pure-storage.rules
|
|
permissions: 0644
|
|
op: create
|
|
content: |
|
|
ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", \
|
|
ATTRS{vendor}=="PURE", ATTR{queue/scheduler}="none"
|
|
ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", \
|
|
ATTRS{vendor}=="PURE", ATTR{queue/nr_requests}="256"
|
|
|
|
# Enable iSCSI mounts in kubelet
|
|
kubelet:
|
|
extraMounts:
|
|
- destination: /var/lib/iscsi
|
|
type: bind
|
|
source: /var/lib/iscsi
|
|
options:
|
|
- bind
|
|
- rshared
|
|
- rw
|
|
```
|
|
|
|
**Apply Talos configuration:**
|
|
```bash
|
|
# From city-hall (talhelper host)
|
|
cd ~/git/homelab/talos/talhelper
|
|
talhelper genconfig
|
|
talosctl apply-config --file clusterconfig/<node>.yaml --nodes <node-ip>
|
|
|
|
# Reboot nodes to load kernel modules
|
|
talosctl reboot --nodes <node-ip>
|
|
```
|
|
|
|
### 4. Verify Talos Prerequisites
|
|
|
|
After applying Talos configuration and rebooting:
|
|
|
|
```bash
|
|
# Check iSCSI initiator service
|
|
talosctl -n <node-ip> get service iscsid
|
|
|
|
# Verify multipath is running
|
|
talosctl -n <node-ip> exec -- multipath -ll
|
|
|
|
# Check initiator name (must be unique per node)
|
|
talosctl -n <node-ip> read /etc/iscsi/initiatorname.iscsi
|
|
|
|
# Verify kernel modules loaded
|
|
talosctl -n <node-ip> exec -- lsmod | grep -E "iscsi|dm_multipath"
|
|
|
|
# Test iSCSI discovery
|
|
talosctl -n <node-ip> exec -- iscsiadm -m discovery -t st -p <flasharray-ip>
|
|
```
|
|
|
|
## Deployment
|
|
|
|
### 1. Commit and Push to Git
|
|
|
|
```bash
|
|
cd ~/git/homelab
|
|
git add cluster/platform/portworx-csi/
|
|
git commit -m "Add Portworx CSI driver for Pure FlashArray"
|
|
git push origin main
|
|
```
|
|
|
|
### 2. Deploy via ArgoCD
|
|
|
|
The application is configured with `sync-wave: 2`, so it will deploy after External Secrets Operator and 1Password Connect.
|
|
|
|
**Option A: Auto-sync (recommended)**
|
|
```bash
|
|
# ArgoCD will automatically sync after detecting changes
|
|
# Check status:
|
|
kubectl get application -n argocd portworx-csi
|
|
```
|
|
|
|
**Option B: Manual sync**
|
|
```bash
|
|
# Sync via CLI
|
|
argocd app sync portworx-csi
|
|
|
|
# Or sync via UI
|
|
# Navigate to ArgoCD UI → Applications → portworx-csi → Sync
|
|
```
|
|
|
|
### 3. Verify Deployment
|
|
|
|
```bash
|
|
# Check ArgoCD application status
|
|
kubectl get application -n argocd portworx-csi
|
|
|
|
# Check Portworx Operator
|
|
kubectl get deployment -n portworx portworx-operator
|
|
kubectl get pods -n portworx -l app=portworx-operator
|
|
|
|
# Check StorageCluster status
|
|
kubectl get storagecluster -n portworx px-cluster-fastpass
|
|
kubectl describe storagecluster -n portworx px-cluster-fastpass
|
|
|
|
# Check PX-CSI pods
|
|
kubectl get pods -n portworx
|
|
|
|
# Verify secret was created from 1Password
|
|
kubectl get secret -n portworx px-pure-secret
|
|
kubectl get secret -n portworx px-pure-secret -o jsonpath='{.data.pure\.json}' | base64 -d | jq
|
|
|
|
# Check CSI driver registration
|
|
kubectl get csidrivers
|
|
kubectl get csinode
|
|
|
|
# Verify StorageClasses
|
|
kubectl get storageclass pure-block pure-file
|
|
```
|
|
|
|
Expected output:
|
|
```
|
|
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION
|
|
pure-block pxd.portworx.com Delete Immediate true
|
|
pure-file pxd.portworx.com Delete Immediate true
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Create a PVC with Block Storage
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: postgres-data
|
|
namespace: default
|
|
spec:
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
storageClassName: pure-block
|
|
resources:
|
|
requests:
|
|
storage: 10Gi
|
|
```
|
|
|
|
### Create a PVC with File Storage (NFS)
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: shared-data
|
|
namespace: default
|
|
spec:
|
|
accessModes:
|
|
- ReadWriteMany
|
|
storageClassName: pure-file
|
|
resources:
|
|
requests:
|
|
storage: 50Gi
|
|
```
|
|
|
|
### Volume Expansion
|
|
|
|
```bash
|
|
# Edit PVC to increase size
|
|
kubectl patch pvc postgres-data -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'
|
|
|
|
# Verify expansion
|
|
kubectl get pvc postgres-data
|
|
```
|
|
|
|
### Volume Snapshots
|
|
|
|
```yaml
|
|
apiVersion: snapshot.storage.k8s.io/v1
|
|
kind: VolumeSnapshot
|
|
metadata:
|
|
name: postgres-backup
|
|
namespace: default
|
|
spec:
|
|
volumeSnapshotClassName: px-csi-snapshot-class
|
|
source:
|
|
persistentVolumeClaimName: postgres-data
|
|
```
|
|
|
|
## StorageClasses
|
|
|
|
### pure-block
|
|
- **Backend:** Pure Storage FlashArray (iSCSI)
|
|
- **Access Mode:** ReadWriteOnce (RWO)
|
|
- **Use Case:** Databases, stateful applications requiring block storage
|
|
- **Features:** High performance, low latency, volume expansion
|
|
- **Filesystem:** ext4
|
|
|
|
### pure-file
|
|
- **Backend:** Pure Storage FlashArray File Services (NFS)
|
|
- **Access Mode:** ReadWriteMany (RWX)
|
|
- **Use Case:** Shared storage for multi-pod applications
|
|
- **Features:** NFS v4.1, concurrent access, volume expansion
|
|
- **Mount Options:** hard, timeo=600, retrans=3
|
|
|
|
## Monitoring
|
|
|
|
Portworx CSI exposes Prometheus metrics:
|
|
|
|
```bash
|
|
# Check ServiceMonitor (if Prometheus Operator is installed)
|
|
kubectl get servicemonitor -n portworx
|
|
|
|
# View metrics endpoint
|
|
kubectl get svc -n portworx -l app=portworx-api
|
|
|
|
# Example metrics:
|
|
# - portworx_volume_usage_bytes
|
|
# - portworx_volume_capacity_bytes
|
|
# - portworx_cluster_status
|
|
```
|
|
|
|
To enable Prometheus scraping, the `kube-prometheus-stack` will auto-discover the ServiceMonitor.
|
|
|
|
## Troubleshooting
|
|
|
|
### Pods stuck in ContainerCreating
|
|
|
|
```bash
|
|
# Check PVC status
|
|
kubectl get pvc
|
|
kubectl describe pvc <pvc-name>
|
|
|
|
# Check PV binding
|
|
kubectl get pv
|
|
|
|
# Check PX-CSI node driver logs
|
|
kubectl logs -n portworx -l app=px-csi-driver --tail=100
|
|
|
|
# Check iSCSI sessions on node
|
|
talosctl -n <node-ip> exec -- iscsiadm -m session
|
|
```
|
|
|
|
### Volume not attaching
|
|
|
|
```bash
|
|
# Check StorageCluster status
|
|
kubectl get storagecluster -n portworx -o yaml
|
|
|
|
# Check PX-CSI controller logs
|
|
kubectl logs -n portworx -l app=portworx-operator
|
|
|
|
# Verify FlashArray connectivity
|
|
kubectl exec -n portworx -it <px-pod> -- ping <flasharray-mgmt-ip>
|
|
|
|
# Check secret is valid
|
|
kubectl get secret -n portworx px-pure-secret -o yaml
|
|
```
|
|
|
|
### Multipath issues
|
|
|
|
```bash
|
|
# Check multipath status on node
|
|
talosctl -n <node-ip> exec -- multipath -ll
|
|
|
|
# Restart multipathd if needed
|
|
talosctl -n <node-ip> exec -- systemctl restart multipathd
|
|
|
|
# Verify udev rules applied
|
|
talosctl -n <node-ip> exec -- udevadm control --reload-rules
|
|
talosctl -n <node-ip> exec -- udevadm trigger
|
|
```
|
|
|
|
### FlashArray API errors
|
|
|
|
```bash
|
|
# Verify API token is valid
|
|
# From a pod with curl:
|
|
curl -k -H "api-token: <token>" https://<flasharray-mgmt-ip>/api/2.0/arrays
|
|
|
|
# Check PX-CSI can reach FlashArray
|
|
kubectl exec -n portworx -it <px-pod> -- curl -k https://<flasharray-mgmt-ip>
|
|
```
|
|
|
|
## Rollback
|
|
|
|
If deployment fails:
|
|
|
|
```bash
|
|
# Delete ArgoCD application (keeps namespace and CRDs)
|
|
kubectl delete application -n argocd portworx-csi
|
|
|
|
# Or remove via ArgoCD UI
|
|
|
|
# Full cleanup (if needed):
|
|
kubectl delete namespace portworx
|
|
kubectl delete crd storageclusters.core.libopenstorage.org
|
|
kubectl delete storageclass pure-block pure-file
|
|
```
|
|
|
|
## References
|
|
|
|
- [Portworx CSI Documentation](https://docs.portworx.com/portworx-csi/)
|
|
- [FlashArray Preparation Guide](https://docs.portworx.com/portworx-csi/install/prepare/flash-array)
|
|
- [Talos Linux Storage Guide](https://www.talos.dev/latest/kubernetes-guides/configuration/storage/)
|
|
- [Pure Storage Best Practices](https://support.purestorage.com/)
|
|
|
|
## Architecture Decisions
|
|
|
|
- **Sync Wave 2:** Deployed after External Secrets Operator (wave 1) but before applications
|
|
- **iSCSI Protocol:** Selected for broad compatibility; NVMe-oF can be added later
|
|
- **No Default StorageClass:** Keeps existing `nfs-emporium` as default; apps opt-in to Pure
|
|
- **Telemetry Disabled:** For homelab privacy; can enable for production support
|
|
- **Single FlashArray:** Configuration supports multiple arrays; add to `pure.json` as needed
|
|
|
|
## Future Enhancements
|
|
|
|
- [ ] Enable CSI topology for multi-zone deployments
|
|
- [ ] Configure NVMe-oF protocol for lower latency
|
|
- [ ] Set up FlashArray ActiveCluster for HA
|
|
- [ ] Enable Portworx Fusion Controller
|
|
- [ ] Configure VLAN binding for isolated storage traffic
|
|
- [ ] Implement backup/restore workflows with snapshots
|
|
- [ ] Add Grafana dashboards for Portworx metrics
|