High CRC errors on all cluster ports to switch and input errors on switch ports
Applies to
- ONTAP 9
- Cisco switch
Issue
- Receiving daily
CLUSTER NETWORK DEGRADEDalerts on all nodes. - s
ystem health subsystem showoutput indicates that the switch health is degraded:
nvlnnas01::> system health subsystem show
Subsystem Health
----------------- ------------------
SAS-connect ok
Environment ok
Memory ok
Service-Processor ok
Switch-Health degraded- Errors will be seen on both cluster ports, one port will have much more than the other on each node:
---- ANYVSERVER IPSpace ----
---- Cluster IPSpace ----
-- interface e1a (2 hours, 58 minutes, 25 seconds) --
RECEIVE
Total frames: 1105k | Frames/second: 103 | Total bytes: 378m
Bytes/second: 35388 | Total errors: 1230k | Errors/minute: 6898
Total discards: 1 | Discards/minute: 0 | Multi/broadcast: 11321
Non-primary u/c: 0 | Errored frames: 0 | Unsupported Op: 0
CRC errors: 1230k | Runt frames: 0 | Fragment: 0
Long frames: 0 | Jabber: 0 | Length errors: 0
Alignment errors: 0 | No buffer: 0 | Pause: 0
Jumbo: 22 | Error symbol: 0 | Bus overruns: 1
Queue drops: 0 | LRO segments: 837k | LRO bytes: 355m
LRO6 segments: 0 | LRO6 bytes: 0 | Bad UDP cksum: 0
Bad UDP6 cksum: 0 | Bad TCP cksum: 0 | Bad TCP6 cksum: 0
Mcast v6 solicit: 0 | Lagg errors: 0 | Lacp errors: 0
Lacp PDU errors: 0
TRANSMIT
Total frames: 1119k | Frames/second: 105 | Total bytes: 498m
Bytes/second: 46559 | Total errors: 0 | Errors/minute: 0
Total discards: 0 | Queue overflow: 0 | Multi/broadcast: 12091
Collisions: 0 | Pause: 7 | Jumbo: 495k
Cfg Up to Downs: 2 | TSO segments: 43235 | TSO bytes: 346m
TSO6 segments: 0 | TSO6 bytes: 0 | HW UDP cksums: 12285
HW UDP6 cksums: 0 | HW TCP cksums: 843k | HW TCP6 cksums: 0
Mcast v6 solicit: 5 | Lagg drops: 0 | Lagg no buffer: 0
Lagg no entries: 0
DEVICE
Mcast addresses: 7 | Rx MBuf Sz: 9216
LINK INFO
Speed: 100G | Duplex: full | Flowcontrol: none
Media state: active | Up to downs: 3 | HW assist: 514k
-- interface e7a (2 hours, 58 minutes, 25 seconds) --
RECEIVE
Total frames: 789k | Frames/second: 74 | Total bytes: 225m
Bytes/second: 21045 | Total errors: 65317 | Errors/minute: 366
Total discards: 0 | Discards/minute: 0 | Multi/broadcast: 62073
Non-primary u/c: 0 | Errored frames: 0 | Unsupported Op: 0
CRC errors: 65317 | Runt frames: 0 | Fragment: 0
Long frames: 0 | Jabber: 0 | Length errors: 0
Alignment errors: 0 | No buffer: 0 | Pause: 0
Jumbo: 6639 | Error symbol: 0 | Bus overruns: 0
Queue drops: 0 | LRO segments: 646k | LRO bytes: 156m
LRO6 segments: 0 | LRO6 bytes: 0 | Bad UDP cksum: 0
Bad UDP6 cksum: 0 | Bad TCP cksum: 0 | Bad TCP6 cksum: 0
Mcast v6 solicit: 0 | Lagg errors: 0 | Lacp errors: 0
Lacp PDU errors: 0
TRANSMIT
Total frames: 696k | Frames/second: 65 | Total bytes: 175m
Bytes/second: 16363 | Total errors: 0 | Errors/minute: 0
Total discards: 0 | Queue overflow: 0 | Multi/broadcast: 12054
Collisions: 0 | Pause: 7 | Jumbo: 3915k
Cfg Up to Downs: 2 | TSO segments: 4174 | TSO bytes: 47871k
TSO6 segments: 0 | TSO6 bytes: 0 | HW UDP cksums: 10904
HW UDP6 cksums: 0 | HW TCP cksums: 662k | HW TCP6 cksums: 0
Mcast v6 solicit: 5 | Lagg drops: 0 | Lagg no buffer: 0
Lagg no entries: 0
DEVICE
Mcast addresses: 8 | Rx MBuf Sz: 9216
LINK INFO
Speed: 100G | Duplex: full | Flowcontrol: none
Media state: active | Up to downs: 4 | HW assist: 514k- Input errors are seen on on the switch with the higher CRC error count in the
show tech-supportlog:
Ethernet1/3 is up
admin state is up, Dedicated Interface
Hardware: 10000/25000/40000/50000/100000 Ethernet, address: a033.4f0b.8ab4 (bia a033.4f0b.8ab4)
MTU 1500 bytes, BW 40000000 Kbit , DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is access
full-duplex, 40 Gb/s, media type is 40G
Beacon is turned off
Auto-Negotiation is turned off FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
admin fec state is auto, oper fec state is off
Last link flapped 18:29:06
Last clearing of "show interface" counters never
7 interface resets
Load-Interval #1: 30 seconds
30 seconds input rate 450376 bits/sec, 190 packets/sec
30 seconds output rate 197080 bits/sec, 177 packets/sec
input rate 450.38 Kbps, 190 pps; output rate 197.08 Kbps, 177 pps
Load-Interval #2: 5 minute (300 seconds)
300 seconds input rate 447160 bits/sec, 152 packets/sec
300 seconds output rate 270280 bits/sec, 146 packets/sec
input rate 447.16 Kbps, 152 pps; output rate 270.28 Kbps, 146 pps
RX
88366883 unicast packets 410358 multicast packets 14411 broadcast packets
88911473 input packets 29103090030 bytes
511886 jumbo packets 0 storm suppression bytes
0 runts 119821 giants 0 CRC 0 no buffer
119761 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
0 Stomped CRC
TX
86749252 unicast packets 274449 multicast packets 153002 broadcast packets
87297215 output packets 25334850548 bytes
121367 jumbo packets
120512 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause