Pod with iSCSI PersistentVolume attached, can't properly run on specific worker node after its reboot
Applies to
- Kubernetes cluster with Ubuntu/Debian worker nodes
- iSCSI PersistentVolumes (PVs)
- NetApp Astra Trident plugin
Issue
After a scheduled maintenance activity, which required rebooting the kubernetes cluster nodes, pods configured with iSCSI PVs attached can't run properly in specific worker nodes
Checking the Trident log for the specific worker node having the issue, it is possible to see traces related to iSCSI login failed to Netapp targets:
time="YYYY-MM-DDThh:mm:ssZ" level=error msg="Error logging in to iSCSI target." error="exit status 8" requestID=xxxxxxxxx-yyyy-zzzz-wwww-ttttttttttt requestSource=CSI
time="YYYY-MM-DDThh:mm:ssZ" level=error msg="Login to iSCSI target failed." err="exit status 8" portalIP=XX.YY.ZZ.WW requestID=xxxxxxxxx-yyyy-zzzz-wwww-ttttttttttt requestSource=CSI
time="YYYY-MM-DDThh:mm:ssZ" level=info msg="Target exists already" Target="iqn.1992-08.com.netapp:sn.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:vs.3" requestID=xxxxxxxxx-yyyy-zzzz-wwww-ttttttttttt requestSource=CSI
time="YYYY-MM-DDThh:mm:ssZ" level=error msg="Error logging in to iSCSI target." error="exit status 8" requestID=xxxxxxxxx-yyyy-zzzz-wwww-ttttttttttt requestSource=CSI