Trident fails to provision volumes in a timely fashion, leaving pods waiting
Applies to
Astra Trident for Kubernetes and Openshift
Issue
When adding an application with pods to the Kubernetes or Openshift cluster, using Trident PVs, the PVs, PVCs and PODs are delayed for a long time before coming up.
Messages seen are:
Unable to attach or mount volumes: unmounted volumes=[PV_NAME], unattached volumes=[PV_NAME_2 PV_NAME PV_NAME_3 ...]:
timed out waiting for the condition
AttachVolume.Attach failed for volume "pvc-UUID" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Additional errors from Trident give more context:
- Trident logs show errors on volume creation and deletion:
backend cannot satisfy create request for volume trident_<PREFIX>_pvc_<UUID>: (ONTAP-NAS pool <AGGR>/<AGGR>; error creating
volume trident_<PREFIX>_pvc_<UUID>: API status: failed,Reason: Failed to create the volume on node \"<STORAGE_NODE>\".
Reason: Failed to store NVE key with key ID\"00000000000000000200000000000HEX-UUID0000000000000000\" on external key server
\"<HOST_FQDN>:50100\