New Azure CVO HA 9.9.1P3 is randomly panicking and rebooting
Applies to
- Cloud Volumes ONTAP (CVO) 9.9.1P3
- Microsoft Azure
- Cloud Manager
Issue
- New CVO HA instances in Azure are panicking and rebooting at random times when certain activities are performed, like Snapmirrors.
- From the Cloud Manager SERVER.LOGs
Cloud Manager is spamming many rate limit exceeded errors and DNS related errors:
2021-12-06 23:17:08,093 UTC WARN [Poll Messages ] [ ] [ ] [ ] (config-akka.actor.default-dispatcher-135) [ClientThrottleInvocationHandler:85] Amazon service is unavailable - DNS resolution error. retrying (16 left)
2021-12-06 23:17:08,673 UTC ERROR [Poll Messages ] [ ] [ ] [ ] (config-akka.actor.default-dispatcher-125) [ConcreteConfig:306] failed to load local.agent.config with error: com.netapp.oncloud.config.server.ConfigException: Failed to retrieve metadata file local.agent.config. Timed out when sending request - Rate limit exceeded.
- When the Create Private Link action is started during CVO deployment, the following error is shown:
failed
{ "registrationEnabled": false, "privateDnsZoneName": "privatelink.blob.core.usgovcloudapi.net", "virtualNetworkId": "/subscriptions/b510a938-29e2-4493-aa2a-XXXXXXXXXXX/resourceGroups/rg-MYRG-01/providers/Microsoft.Network/virtualNetworks/vnet-theVNET-hosting-usgovvirginia-01", "region": "global", "_failure": "The following permission is missing: 'Microsoft.Network/virtualNetworks/join/action' on the '0' linked scope(s) '' or the linked scope(s) are invalid and is blocked by deny assignments on the '1'. For more information please refer to the OnCommand Cloud Manager policies documentation at [https://mysupport.netapp.com/site/in...nager-policies"|https://mysupport.netapp.com/site/in...r-policies%22] , "_resourceGroup": "rg-MYRG-hosting-usgovvirginia-01", "privateDnsLinkName": "vnet-theVNET-hosting-usgovvirginia-01_usgovvirginia_DnsLink" }