Active IQ Unified Manager web interface goes unresponsive due to 'Too many open files' error caused by file descriptor leak

Applies to

• Active IQ Unified Manager (AIQUM) 9.8, 9.9

Issue

• AIQUM WebUI is unresponsive
• API calls triggered fail with Error 500 - Internal Server Error
• Grafana fails to create report of AIQUM
• server.log shows Too many open files error

Exception handling request to /apis/XMLrequest: java.lang.RuntimeException: java.io.IOException: Cannot run program "/opt/netapp/essentials/jboss/bin/native/lib64/authenticate": error=24, Too many open files

• ocumserver.log

  2022-06-25 22:58:59,650 INFO [oncommand] [Access Log Task] [com.netapp.ipc.util.AccessLogTask] 2022-01-21 is older than 30 days
2022-06-25 22:58:59,697 ERROR [oncommand] [Access Log Task] [com.netapp.ipc.util.AccessLogTask] Got IO exception while processing access_log
java.nio.file.AccessDeniedException:D:\ProgramData\NetApp\OnCommandAppData\ocie\log\2022-01-21
at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)

• lsof -p `cat /var/run/ocie.pid` | awk '{print $9}' | sort | grep "/var/log/ocie/20" | uniq -c shows large numbers of file descriptors for /var/log/ocie/<YEAR>-<MONTH>-<DATE> directories that are older than 30 days

174 /var/log/ocie/2019-08-26
172 /var/log/ocie/2019-08-27
 :
 34 /var/log/ocie/2021-07-13
 98 /var/log/ocie/2021-07-14
 98 /var/log/ocie/2021-07-15
 40 /var/log/ocie/2021-07-16