homelab/cluster/platform/portworx-csi/QUICKREF.md

Portworx CSI Quick Reference

Common commands and operations for Portworx CSI with Pure Storage FlashArray.

Check Status

# ArgoCD application
kubectl get application -n argocd portworx-csi

# All pods in portworx namespace
kubectl get pods -n portworx

# StorageCluster status
kubectl get storagecluster -n portworx px-cluster-fastpass
kubectl describe storagecluster -n portworx px-cluster-fastpass

# CSI driver registration
kubectl get csidrivers pxd.portworx.com
kubectl get csinode

# StorageClasses
kubectl get storageclass pure-block pure-file

Volume Operations

Create PVC (Block Storage)

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-data
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: pure-block
  resources:
    requests:
      storage: 10Gi
EOF

Create PVC (File Storage - NFS)

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-data
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: pure-file
  resources:
    requests:
      storage: 50Gi
EOF

Expand Volume

# Edit PVC to increase size
kubectl patch pvc my-app-data -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

# Verify expansion
kubectl get pvc my-app-data

Create Snapshot

kubectl apply -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: my-app-snapshot
  namespace: default
spec:
  volumeSnapshotClassName: px-csi-snapshot-class
  source:
    persistentVolumeClaimName: my-app-data
EOF

Restore from Snapshot

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-data-restored
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: pure-block
  dataSource:
    name: my-app-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  resources:
    requests:
      storage: 20Gi
EOF

Logs

# Portworx Operator
kubectl logs -n portworx -l app=portworx-operator --tail=100

# PX-CSI Controller
kubectl logs -n portworx -l app=px-csi-controller --tail=100

# PX-CSI Node driver (specific node)
kubectl logs -n portworx -l app=px-csi-node --tail=100

# Follow logs in real-time
kubectl logs -n portworx -l app=px-csi-controller -f

Troubleshooting

Check iSCSI on Talos Nodes

# List iSCSI sessions
talosctl -n <node-ip> exec -- iscsiadm -m session

# Discover FlashArray targets
talosctl -n <node-ip> exec -- iscsiadm -m discovery -t st -p <flasharray-ip>

# Check initiator name (must be unique)
talosctl -n <node-ip> read /etc/iscsi/initiatorname.iscsi

Check Multipath

# Multipath status
talosctl -n <node-ip> exec -- multipath -ll

# Reload multipath config
talosctl -n <node-ip> exec -- systemctl reload multipathd

# Multipath devices
talosctl -n <node-ip> exec -- ls -l /dev/mapper/

Verify FlashArray Connectivity

# From PX-CSI pod
kubectl exec -n portworx -it <px-pod> -- ping <flasharray-ip>

# Test API endpoint
kubectl exec -n portworx -it <px-pod> -- \
  curl -k https://<flasharray-mgmt-ip>/api/2.0/arrays

Check Secret

# Verify secret exists
kubectl get secret -n portworx px-pure-secret

# View secret content (pure.json)
kubectl get secret -n portworx px-pure-secret \
  -o jsonpath='{.data.pure\.json}' | base64 -d | jq .

# Check ExternalSecret status
kubectl get externalsecret -n portworx px-pure-secret
kubectl describe externalsecret -n portworx px-pure-secret

Volume Not Attaching

# Check PVC events
kubectl describe pvc <pvc-name>

# Check pod events
kubectl describe pod <pod-name>

# Check PV
kubectl get pv
kubectl describe pv <pv-name>

# Check volume attachment
kubectl get volumeattachment

Pod Stuck in ContainerCreating

# Check events
kubectl describe pod <pod-name>

# Check PVC status
kubectl get pvc -n <namespace>

# Check node where pod is scheduled
POD_NODE=$(kubectl get pod <pod-name> -o jsonpath='{.spec.nodeName}')
echo "Pod scheduled on: $POD_NODE"

# Check iSCSI on that node
talosctl -n $POD_NODE exec -- iscsiadm -m session

Monitoring

Prometheus Queries

# Cluster status
portworx_cluster_status

# Volume capacity
portworx_volume_capacity_bytes

# Volume usage
portworx_volume_usage_bytes

# I/O metrics
rate(portworx_volume_read_bytes[5m])
rate(portworx_volume_write_bytes[5m])

List All PVCs Using Pure Storage

# Block storage
kubectl get pvc --all-namespaces -o json | \
  jq -r '.items[] | select(.spec.storageClassName == "pure-block") | 
  "\(.metadata.namespace)/\(.metadata.name) - \(.status.phase)"'

# File storage
kubectl get pvc --all-namespaces -o json | \
  jq -r '.items[] | select(.spec.storageClassName == "pure-file") | 
  "\(.metadata.namespace)/\(.metadata.name) - \(.status.phase)"'

Storage Usage by Namespace

kubectl get pvc --all-namespaces -o json | \
  jq -r '.items[] | select(.spec.storageClassName | startswith("pure-")) | 
  "\(.metadata.namespace)\t\(.spec.resources.requests.storage)"' | \
  awk '{arr[$1]+=$2} END {for (i in arr) print i, arr[i]}'

Maintenance

Update FlashArray Credentials

# Update in 1Password:
# Vault: homelab
# Item: pure-flasharray-fastpass
# Field: pure.json

# Force ExternalSecret refresh
kubectl delete secret -n portworx px-pure-secret

# Wait for ExternalSecret to recreate it (check status)
kubectl get externalsecret -n portworx px-pure-secret -w

Restart PX-CSI Components

# Restart controller
kubectl rollout restart deployment -n portworx px-csi-controller

# Restart node drivers (DaemonSet)
kubectl rollout restart daemonset -n portworx px-csi-node

# Check rollout status
kubectl rollout status deployment -n portworx px-csi-controller
kubectl rollout status daemonset -n portworx px-csi-node

Drain Node for Maintenance

# Cordon node
kubectl cordon <node-name>

# Drain node (evict pods)
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# Perform maintenance...

# Uncordon node
kubectl uncordon <node-name>

ArgoCD Operations

Sync Application

# Sync via CLI
argocd app sync portworx-csi

# Force sync (ignore differences)
argocd app sync portworx-csi --force

# Sync specific resource
argocd app sync portworx-csi --resource storagecluster:px-cluster-fastpass

View Application Details

# App status
argocd app get portworx-csi

# App history
argocd app history portworx-csi

# App diff (compare git vs cluster)
argocd app diff portworx-csi

Rollback via ArgoCD

# List history
argocd app history portworx-csi

# Rollback to previous version
argocd app rollback portworx-csi <revision-number>

Configuration Updates

Edit StorageCluster

# Edit directly (will be overwritten by ArgoCD sync)
kubectl edit storagecluster -n portworx px-cluster-fastpass

# Permanent change: update git
cd ~/git/homelab
vi cluster/platform/portworx-csi/storagecluster.yaml
git commit -am "Update Portworx StorageCluster config"
git push

Add Environment Variable

# In storagecluster.yaml, add to spec.env:
env:
  - name: NEW_ENV_VAR
    value: "value"

Change Log Level

# In storagecluster.yaml:
env:
  - name: LOG_LEVEL
    value: "debug"  # or: info, warn, error

Talos Node Configuration

View Current Config

# Read multipath.conf
talosctl -n <node-ip> read /etc/multipath.conf

# Read udev rules
talosctl -n <node-ip> read /etc/udev/rules.d/99-pure-storage.rules

# Check loaded modules
talosctl -n <node-ip> exec -- lsmod | grep -E "iscsi|multipath"

Apply Updated Talos Config

cd ~/git/homelab/talos/talhelper

# Regenerate configs after updating talconfig.yaml
talhelper genconfig

# Apply to specific node
talosctl apply-config \
  --file clusterconfig/<node-name>.yaml \
  --nodes <node-ip>

# Reboot if kernel modules changed
talosctl reboot --nodes <node-ip>

Emergency Procedures

Complete Reinstall

# 1. Delete ArgoCD application
kubectl delete application -n argocd portworx-csi

# 2. Delete namespace
kubectl delete namespace portworx

# 3. Delete CRDs
kubectl delete crd storageclusters.core.libopenstorage.org

# 4. Delete StorageClasses
kubectl delete storageclass pure-block pure-file

# 5. Re-apply via ArgoCD
argocd app sync portworx-csi

Recover from Failed State

# 1. Check StorageCluster status
kubectl get storagecluster -n portworx -o yaml

# 2. Delete and recreate StorageCluster
kubectl delete storagecluster -n portworx px-cluster-fastpass

# 3. Let ArgoCD recreate it
argocd app sync portworx-csi --force

# 4. Monitor deployment
kubectl get pods -n portworx -w

Useful Aliases

Add to ~/.bashrc or ~/.zshrc:

# Portworx aliases
alias pxpods='kubectl get pods -n portworx'
alias pxlogs='kubectl logs -n portworx'
alias pxstc='kubectl get storagecluster -n portworx'
alias pxpvc='kubectl get pvc --all-namespaces -o wide | grep pure'
alias pxsc='kubectl get storageclass | grep pure'

References

• Full Documentation: cluster/platform/portworx-csi/README.md
• Deployment Checklist: cluster/platform/portworx-csi/DEPLOYMENT-CHECKLIST.md
• Portworx Docs: https://docs.portworx.com/portworx-csi/
• FlashArray Docs: https://support.purestorage.com/