Skip to main content
NetApp Stage KB

Volume with very high latency on a node causing high CPU

  1. Last updated
  2. Save as PDF

Views:
2
Visibility:
Public
Votes:
0
Category:
ontap-9
Specialty:
perf
Last Updated:

Applies to

ONTAP 9

Issue

  • Volume with very high latency on one node
  • Increased workload driving high CPU on node resulted in high latency
  • Latency  is observed and node utilization in monitoring tools such as Active IQ Unified Manager or Cloud Insights
  • High CPU is seen on one node but node 2 is passive and idle
    • Example: Node 1 has high CPU due to user workload, but node 2 is idle as seen in node shell sysstat -x 1 command
    • Note: Columns removed to improve readability
Cluster::> node run node1 sysstat -x 1
CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s   
                                       in    out    read  write   
79%  22453      0      0   22463  1491948   8098  664188 2631848 
76%  22448      0      0   22478  1492337   8121  607184  658216 
75%  22478      0      0   22509  1492134   8106   78844  101992 
75%  22453      0      0   23134  1492587   8108  810668 2736420 

Cluster::> qos statistics volume latency show -node node1
Workload            ID    Latency    Network    Cluster       Data       Disk        QoS      NVRAM
--------------- ------ ---------- ---------- ---------- ----------  ---------  ---------  ---------
-total-              -   136.49ms    99.00us    70.00us   136.17ms   153.00us        0ms        0ms
vserver1_vol1..   4201   206.05ms   130.00us        0ms   205.88ms    44.00us        0ms        0ms
vserver5_vol8..   7704  1309.00us   351.00us     1.00us   834.00us   114.00us        0ms     9.00us
-total-              -   140.29ms   103.00us    75.00us   139.94ms   174.00us        0ms        0ms
vserver1_vol1..   4201   379.03ms   127.00us        0ms   378.73ms   175.00us        0ms        0ms
vserver5_vol8..   7704     2.02ms   309.00us     1.30us  1820.00us   105.00us        0ms     9.00us

Cluster::> node run node2 sysstat -x 1
 CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s   
                                       in    out    read  write   
  8%      0      0      0      11     660    111    2640      24  
  1%      0      0      0       0     150    150      24       0  
  1%      0      0      0      42       7      7       0      24  
  
Cluster::> qos statistics volume latency show -node node2
Workload            ID    Latency    Network    Cluster       Data       Disk        QoS      NVRAM
--------------- ------ ---------- ---------- ---------- ---------- ---------- ---------- ----------
-total-              -     2.21ms   101.00us    75.00us   400.00us     1.62ms        0ms     7.00us 
vserver2_vol1..   3195  1295.00us   478.00us        0ms   777.00us    26.00us        0ms    14.00us

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.