Scheduled Exchange and SQL back ups not starting after upgrading to SnapCenter 6.0
Applies to
- SnapCenter Server (SC) 6.0
- SnapCenter Plug-in for Microsoft Exchange Server (SCE) 6.0
- SnapCenter Plug-in for Microsoft SQL Server (SCSQL) 6.0
- Scheduled backups
Issue
- After upgrading SnapCenter to 6.0, Exchange and SQL back ups scheduled by SnapCenter no longer start / run.
- Any back ups started manually will run and complete successfully.
- SQL back ups configured to use the SQL scheduler also run and complete successfully.
The following errors can be seen in the below logs on the SnapCenter server:
C:\Program Files\NetApp\SnapCenter WebApp\App_Data\log\SnapManagerWeb.log:
ERROR SnapManagerWeb PID=[xxxx] TID=[xxxx] RabbitMQ broker is unreachable. Retrying in 30 seconds...C:\Users\<Username>\AppData\Roaming\RabbitMQ\log\rabbit@<SC_Server_Name>.log:
[error] <0.798.0> Error on AMQP connection <0.798.0> ([::1]:50342 -> [::1]:5672, state: starting):[error] <0.798.0> PLAIN login refused: user 'jobmanager' - invalid credentialsC:\Program Files\NetApp\Scheduler\logs\Scheduler<Date>.log:
[ ERR] System.UnauthorizedAccessException: Token not avaliable at NetApp.DataProtection.Common.Utils.Authorization.AuthorizationMiddleware.Invoke(HttpContext httpContext) in R:\rtprlse19\snapcenter\orion\nightly\R6.0N_240716_0044\blockbuster\Scheduler\src\Common.Utils\Authorization\AuthorizationMiddleware.cs:line 62 at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context) at NetApp.DataProtection.Common.Utils.ExceptionHandler.ExceptionHandlerMiddleware.Invoke(HttpContext context) in R:\rtprlse19\snapcenter\orion\nightly\R6.0N_240716_0044\blockbuster\Scheduler\src\Common.Utils\ExceptionHandler\ExceptionHandlerMiddleware.cs:line 32[ ERR] at NetApp.DataProtection.Common.Utils.Authorization.AuthorizationMiddleware.Invoke(HttpContext httpContext) in R:\rtprlse19\snapcenter\orion\nightly\R6.0N_240716_0044\blockbuster\Scheduler\src\Common.Utils\Authorization\AuthorizationMiddleware.cs:line 62 at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context) at NetApp.DataProtection.Common.Utils.ExceptionHandler.ExceptionHandlerMiddleware.Invoke(HttpContext context) in R:\rtprlse19\snapcenter\orion\nightly\R6.0N_240716_0044\blockbuster\Scheduler\src\Common.Utils\ExceptionHandler\ExceptionHandlerMiddleware.cs:line 32[ ERR] Job <JobID>= threw an unhandled Exception: RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable ---> RabbitMQ.Client.Exceptions.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile. at RabbitMQ.Client.Framing.Impl.Connection.StartAndTune() at RabbitMQ.Client.Framing.Impl.Connection.Open(Boolean insist) at RabbitMQ.Client.Framing.Impl.Connection..ctor(IConnectionFactory factory, Boolean insist, IFrameHandler frameHandler, String clientProvidedName) at RabbitMQ.Client.Framing.Impl.Connection..ctor(IConnectionFactory factory, Boolean insist, IFrameHandler frameHandler, ArrayPool`1 memoryPool, String clientProvidedName) at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.Init(IFrameHandler fh) at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.Init(IEndpointResolver endpoints) at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)