# Portworx CSI Deployment Checklist

Pre-deployment checklist for Portworx CSI driver on Talos Linux with Pure Storage FlashArray.

## Phase 1: FlashArray Preparation

- [ ] **Create FlashArray storage admin user**
  - Username: `portworx-csi`
  - Role: Storage Admin
  - Password: Set in FlashArray UI

- [ ] **Generate FlashArray API token**
  - User: `portworx-csi`
  - Expiration: None (non-expiring)
  - Token format: `XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX`
  - **Save token immediately** (cannot be retrieved later)

- [ ] **Verify FlashArray management endpoint**
  - Test connectivity: `ping flasharray.mk-labs.cloud`
  - Test HTTPS: `curl -k https://flasharray.mk-labs.cloud`
  - Record IP/hostname: _______________________

- [ ] **Configure iSCSI targets on FlashArray**
  - Navigate to: **Settings → Network**
  - Verify iSCSI interfaces are enabled
  - Test discovery from Talos node:
    ```bash
    talosctl -n <node> exec -- iscsiadm -m discovery -t st -p <flasharray-ip>
    ```

- [ ] **(Optional) Enable FlashArray File Services for NFS**
  - Enable File Services: **Settings → File Services**
  - Create filesystem: `fastpass-nfs`
  - Create NFS policy with user mapping **disabled**
  - Record NFS endpoint: _______________________

## Phase 2: 1Password Secret Configuration

- [ ] **Create 1Password item**
  - Vault: `homelab`
  - Item name: `pure-flasharray-fastpass`
  - Category: Secure Note or API Credential

- [ ] **Add pure.json field to 1Password item**
  - Field name: `pure.json`
  - Field type: Text
  - Content format: JSON (see below)

- [ ] **Populate pure.json content**
  ```json
  {
    "FlashArrays": [
      {
        "MgmtEndPoint": "flasharray.mk-labs.cloud",
        "APIToken": "paste-api-token-here",
        "NFSEndPoint": "flasharray-nfs.mk-labs.cloud"
      }
    ]
  }
  ```
  - Replace `flasharray.mk-labs.cloud` with actual endpoint
  - Replace `paste-api-token-here` with FlashArray API token
  - Remove `NFSEndPoint` line if not using File Services

- [ ] **Validate JSON syntax**
  ```bash
  # Copy content from 1Password and validate
  echo '<paste-json>' | jq .
  ```

- [ ] **Verify 1Password Connect can access item**
  ```bash
  # From a pod with op CLI or via 1Password UI
  # Confirm item exists in homelab vault
  ```

## Phase 3: Talos Node Configuration

- [ ] **Edit talconfig.yaml**
  - File: `~/git/homelab/talos/talhelper/talconfig.yaml`
  - Add patches from `cluster/platform/portworx-csi/talos-config-reference.yaml`
  - Sections to add:
    - [ ] System extensions (iscsi-tools, util-linux-tools)
    - [ ] Kernel modules (iscsi_tcp, dm_multipath, dm_round_robin)
    - [ ] /etc/multipath.conf file
    - [ ] /etc/udev/rules.d/99-pure-storage.rules file
    - [ ] kubelet extraMounts for /var/lib/iscsi

- [ ] **Generate updated Talos configs**
  ```bash
  cd ~/git/homelab/talos/talhelper
  talhelper genconfig
  ```

- [ ] **Apply Talos configuration to all worker nodes**
  ```bash
  # For each worker node:
  talosctl apply-config \
    --file clusterconfig/<node-name>.yaml \
    --nodes <node-ip>
  ```
  
  Nodes:
  - [ ] Node 1: _________________ (IP: _____________)
  - [ ] Node 2: _________________ (IP: _____________)
  - [ ] Node 3: _________________ (IP: _____________)

- [ ] **Reboot all worker nodes**
  ```bash
  # Reboot nodes one at a time to load kernel modules
  talosctl reboot --nodes <node-ip>
  
  # Wait for node to come back online
  kubectl get nodes
  ```

- [ ] **Verify iSCSI initiator on each node**
  ```bash
  talosctl -n <node-ip> get service iscsid
  # Expected: iscsid service running
  ```

- [ ] **Verify multipath on each node**
  ```bash
  talosctl -n <node-ip> exec -- multipath -ll
  # Expected: multipath running (may show no paths yet)
  ```

- [ ] **Verify unique iSCSI initiator names**
  ```bash
  # For each node:
  talosctl -n <node-ip> read /etc/iscsi/initiatorname.iscsi
  
  # Record initiator names (must be unique):
  # Node 1: iqn.1994-05.com.redhat:_______________
  # Node 2: iqn.1994-05.com.redhat:_______________
  # Node 3: iqn.1994-05.com.redhat:_______________
  ```

- [ ] **Verify kernel modules loaded**
  ```bash
  talosctl -n <node-ip> exec -- lsmod | grep -E "iscsi|dm_multipath"
  # Expected: iscsi_tcp, dm_multipath, dm_round_robin
  ```

- [ ] **Test iSCSI discovery from each node**
  ```bash
  talosctl -n <node-ip> exec -- \
    iscsiadm -m discovery -t st -p <flasharray-ip>
  # Expected: List of iSCSI targets from FlashArray
  ```

## Phase 4: GitOps Deployment

- [ ] **Review manifests**
  - [ ] `cluster/platform/portworx-csi/application.yaml`
  - [ ] `cluster/platform/portworx-csi/operator-values.yaml`
  - [ ] `cluster/platform/portworx-csi/storagecluster.yaml`
  - [ ] `cluster/platform/portworx-csi/externalsecret.yaml`
  - [ ] `cluster/platform/portworx-csi/storageclass-block.yaml`
  - [ ] `cluster/platform/portworx-csi/storageclass-file.yaml`

- [ ] **Customize for your environment**
  - Update FlashArray endpoints if different from defaults
  - Adjust resource limits if needed
  - Review sync wave (currently: 2)

- [ ] **Commit to Git**
  ```bash
  cd ~/git/homelab
  git add cluster/platform/portworx-csi/
  git commit -m "Add Portworx CSI driver for Pure FlashArray"
  git push origin main
  ```

- [ ] **Verify ArgoCD auto-sync or trigger manual sync**
  ```bash
  # Check ArgoCD application status
  kubectl get application -n argocd portworx-csi
  
  # Manual sync if needed
  argocd app sync portworx-csi
  ```

## Phase 5: Deployment Verification

- [ ] **Verify ArgoCD application is healthy**
  ```bash
  kubectl get application -n argocd portworx-csi
  # Expected: STATUS=Synced, HEALTH=Healthy
  ```

- [ ] **Check namespace created**
  ```bash
  kubectl get namespace portworx
  ```

- [ ] **Verify ExternalSecret synced**
  ```bash
  kubectl get externalsecret -n portworx px-pure-secret
  # Expected: STATUS=SecretSynced
  
  kubectl get secret -n portworx px-pure-secret
  # Expected: Secret exists with pure.json key
  ```

- [ ] **Validate secret content**
  ```bash
  kubectl get secret -n portworx px-pure-secret \
    -o jsonpath='{.data.pure\.json}' | base64 -d | jq .
  # Expected: Valid JSON with FlashArray config
  ```

- [ ] **Check Portworx Operator deployed**
  ```bash
  kubectl get deployment -n portworx portworx-operator
  kubectl get pods -n portworx -l app=portworx-operator
  # Expected: 1/1 pods running
  ```

- [ ] **Verify StorageCluster created**
  ```bash
  kubectl get storagecluster -n portworx px-cluster-fastpass
  # Expected: STATUS=Running or Initializing
  ```

- [ ] **Check PX-CSI pods running**
  ```bash
  kubectl get pods -n portworx
  # Expected pods:
  # - portworx-operator-...
  # - px-csi-controller-...
  # - px-csi-node-... (one per node)
  ```

- [ ] **Verify CSI driver registered**
  ```bash
  kubectl get csidrivers
  # Expected: pxd.portworx.com
  
  kubectl get csinode
  # Expected: One CSINode per Kubernetes node
  ```

- [ ] **Check StorageClasses created**
  ```bash
  kubectl get storageclass pure-block pure-file
  # Expected: Both StorageClasses exist
  ```

- [ ] **Review PX-CSI logs for errors**
  ```bash
  # Controller logs
  kubectl logs -n portworx -l app=px-csi-controller --tail=50
  
  # Node driver logs (on each node)
  kubectl logs -n portworx -l app=px-csi-node --tail=50
  ```

## Phase 6: Functional Testing

- [ ] **Create test PVC with block storage**
  ```bash
  cat <<EOF | kubectl apply -f -
  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: test-pure-block
    namespace: default
  spec:
    accessModes:
      - ReadWriteOnce
    storageClassName: pure-block
    resources:
      requests:
        storage: 1Gi
  EOF
  ```

- [ ] **Verify PVC bound**
  ```bash
  kubectl get pvc test-pure-block
  # Expected: STATUS=Bound
  ```

- [ ] **Verify PV created on FlashArray**
  ```bash
  kubectl get pv
  # Look for PV bound to test-pure-block
  
  # Check FlashArray UI or API:
  # Volumes should show new volume created
  ```

- [ ] **Create test pod using PVC**
  ```bash
  cat <<EOF | kubectl apply -f -
  apiVersion: v1
  kind: Pod
  metadata:
    name: test-pure-block-pod
    namespace: default
  spec:
    containers:
    - name: test
      image: nginx:latest
      volumeMounts:
      - name: data
        mountPath: /data
    volumes:
    - name: data
      persistentVolumeClaim:
        claimName: test-pure-block
  EOF
  ```

- [ ] **Verify pod is running**
  ```bash
  kubectl get pod test-pure-block-pod
  # Expected: STATUS=Running
  ```

- [ ] **Verify volume mounted in pod**
  ```bash
  kubectl exec test-pure-block-pod -- df -h /data
  # Expected: Mount point shows 1Gi volume
  ```

- [ ] **Test write to volume**
  ```bash
  kubectl exec test-pure-block-pod -- \
    sh -c 'echo "test data" > /data/test.txt'
  
  kubectl exec test-pure-block-pod -- cat /data/test.txt
  # Expected: "test data"
  ```

- [ ] **Verify iSCSI session established**
  ```bash
  # On node running test pod
  talosctl -n <node-ip> exec -- iscsiadm -m session
  # Expected: Active iSCSI session to FlashArray
  ```

- [ ] **Clean up test resources**
  ```bash
  kubectl delete pod test-pure-block-pod
  kubectl delete pvc test-pure-block
  ```

- [ ] **(Optional) Test NFS file storage**
  - Repeat above steps with `storageClassName: pure-file`
  - Verify NFS export created on FlashArray

## Phase 7: Monitoring Setup

- [ ] **Verify Prometheus ServiceMonitor created**
  ```bash
  kubectl get servicemonitor -n portworx
  ```

- [ ] **Check Prometheus targets**
  - Navigate to Prometheus UI → Targets
  - Look for `portworx` or `px-csi` targets
  - Status should be UP

- [ ] **Query basic metrics**
  ```promql
  # In Prometheus UI
  portworx_cluster_status
  portworx_volume_capacity_bytes
  ```

- [ ] **(Optional) Create Grafana dashboard**
  - Import Portworx dashboard from Grafana.com
  - Or create custom dashboard

## Phase 8: Documentation & Handoff

- [ ] **Update homelab documentation**
  - Add FlashArray details to infrastructure docs
  - Document storage classes and use cases

- [ ] **Record credentials securely**
  - FlashArray management URL: ___________________
  - API token stored in: 1Password → homelab → pure-flasharray-fastpass
  - Portworx namespace: `portworx`

- [ ] **Create runbook for common tasks**
  - Volume expansion procedure
  - Snapshot creation/restore
  - Troubleshooting steps

- [ ] **Schedule follow-up review**
  - Date: _______________
  - Review metrics and performance
  - Adjust resource limits if needed

## Rollback Plan

If deployment fails, rollback procedure:

1. [ ] Delete ArgoCD application:
   ```bash
   kubectl delete application -n argocd portworx-csi
   ```

2. [ ] Delete namespace (if needed):
   ```bash
   kubectl delete namespace portworx
   ```

3. [ ] Delete StorageClasses:
   ```bash
   kubectl delete storageclass pure-block pure-file
   ```

4. [ ] Delete CRDs (if needed):
   ```bash
   kubectl delete crd storageclusters.core.libopenstorage.org
   ```

5. [ ] Revert Talos configuration (if needed):
   ```bash
   # Remove portworx-csi patches from talconfig.yaml
   talhelper genconfig
   talosctl apply-config --file clusterconfig/<node>.yaml --nodes <node-ip>
   ```

## Success Criteria

Deployment is successful when:

- [ ] All pods in `portworx` namespace are Running
- [ ] StorageClasses `pure-block` and `pure-file` are available
- [ ] Test PVC successfully binds to a PV
- [ ] Test pod can mount and write to volume
- [ ] iSCSI sessions are established on nodes
- [ ] No errors in PX-CSI controller/node logs
- [ ] Prometheus metrics are being scraped
- [ ] ArgoCD application shows Healthy and Synced

## Notes

**Deployment Date:** _______________

**Deployed By:** _______________

**FlashArray Model:** _______________

**Purity Version:** _______________

**Issues Encountered:**
- 
- 
- 

**Resolutions:**
- 
- 
- 

**Follow-up Items:**
- 
- 
-