Symptoms
Our network monitoring software found memory usage on some new production switches keep increasing. Those switches are Cisco 2960X and coming with 15.0(2) EX3 IOS.
As we know there are two types of memory in Cisco IOS: process memory and IO memory.
- When a feature is enable on an IOS device (e.g. PIM, HSRP, and etc), IOS allocates process memory for the process.
- IO memory is used when a software switched traffic hits the CPU. The CPU allocates IO memory to store the frame temporarily.
Our case is process memory increasing. What we need to do is to find out which process.
Switch1# show version Cisco IOS Software, C2960X Software (C2960X-UNIVERSALK9-M), Version 15.0(2)EX3, RELEASE SOFTWARE (fc1) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2013 by Cisco Systems, Inc. Compiled Wed 11-Sep-13 02:04 by prod_rel_team ROM: Bootstrap program is C2960X boot loader BOOTLDR: C2960X Boot Loader (C2960X-HBOOT-M) Version 15.0(2r)EX, RELEASE SOFTWARE (fc1) CSWP3024 uptime is 5 days, 23 hours, 48 minutes System returned to ROM by power-on System restarted at 05:47:41 EST Sat Mar 11 2017 System image file is "flash:/c2960x-universalk9-mz.150-2.EX3/c2960x-universalk9-mz.150-2.EX3.bin" This product contains cryptographic features and is subject to United States and local country laws governing import, export, transfer and use. Delivery of Cisco cryptographic products does not imply third-party authority to import, export, distribute or use encryption. Importers, exporters, distributors and users are responsible for compliance with U.S. and local country laws. By using this product you agree to comply with applicable laws and regulations. If you are unable to comply with U.S. and local laws, return this product immediately. A summary of U.S. laws governing Cisco cryptographic products may be found at: http://www.cisco.com/wwl/export/crypto/tool/stqrg.html If you require further assistance please contact us by sending email to [email protected] cisco WS-C2960X-48TS-LL (APM86XXX) processor (revision B0) with 262144K bytes of memory. Processor board ID FOC1744W3V5 Last reset from power-on 2 Virtual Ethernet interfaces 1 FastEthernet interface 50 Gigabit Ethernet interfaces The password-recovery mechanism is enabled. 512K bytes of flash-simulated non-volatile configuration memory. Base ethernet MAC Address : 5C:A4:8A:5C:6F:80 Motherboard assembly number : 73-14196-03 Power supply part number : 341-0537-02 Motherboard serial number : FOC17443CWD Power supply serial number : LIT17320EU6 Model revision number : B0 Motherboard revision number : B0 Model number : WS-C2960X-48TS-LL Daughterboard assembly number : 73-14200-03 Daughterboard serial number : FOC17440VRF System serial number : FOC1744W3V5 Top Assembly Part Number : 800-38011-02 Top Assembly Revision Number : B0 Version ID : V02 CLEI Code Number : CMMNG00ARA Daughterboard revision number : A0 Hardware Board Revision Number : 0x05 Switch Ports Model SW Version SW Image ------ ----- ----- ---------- ---------- * 1 50 WS-C2960X-48TS-LL 15.0(2)EX3 C2960X-UNIVERSALK9-M Configuration register is 0xF Switch1#show memory statistics history ------------------ History of Processor Mempool ------------------ 777777777777777777777777777777777777777777777777777777777777 222222222222222222222222222222222222222222222222222222222222 100 90 80 70 ********************************************************** 60 ********************************************************** 50 ********************************************************** 40 ********************************************************** 30 ********************************************************** 20 ********************************************************** 10 ********************************************************** 0....5....1....1....2....2....3....3....4....4....5....5....6 0 5 0 5 0 5 0 5 0 5 0 Free memory per second (last 60 seconds) 777777777777777777777777777777777777777777777777777777777777 222222222222222222222222222222222222222222222222222222222222 100 90 80 70 ########################################################## 60 ########################################################## 50 ########################################################## 40 ########################################################## 30 ########################################################## 20 ########################################################## 10 ########################################################## 0....5....1....1....2....2....3....3....4....4....5....5....6 0 5 0 5 0 5 0 5 0 5 0 Free memory per minute (last 60 minutes) * = maximum # = average 777777777777777777777777777777777777777777777777777777777777777777777777 222222222233333333333333444444444444444555555555555555666666666666667777 100 90 80 **############################# 70 ###################################################################### 60 ###################################################################### 50 ###################################################################### 40 ###################################################################### 30 ###################################################################### 20 ###################################################################### 10 ###################################################################### 0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.. 0 5 0 5 0 5 0 5 0 5 0 5 0 Free memory per hour (last 72 hours) * = maximum # = average ------------------ History of I/O Mempool ------------------ 555555555555555555555555555555555555555555555555555555555555 888888888888888888888888888888888888888888888888888888888888 100 90 80 70 60 ********************************************************** 50 ********************************************************** 40 ********************************************************** 30 ********************************************************** 20 ********************************************************** 10 ********************************************************** 0....5....1....1....2....2....3....3....4....4....5....5....6 0 5 0 5 0 5 0 5 0 5 0 Free memory per second (last 60 seconds) 555555555555555555555555555555555555555555555555555555555555 888888888888888888888888888888888888888888888888888888888888 100 90 80 70 60 ########################################################## 50 ########################################################## 40 ########################################################## 30 ########################################################## 20 ########################################################## 10 ########################################################## 0....5....1....1....2....2....3....3....4....4....5....5....6 0 5 0 5 0 5 0 5 0 5 0 Free memory per minute (last 60 minutes) * = maximum # = average 555555555555555555555555555555555555555555555555555555555555555555555555 888888888888888888888888888888888888888888888888888888888888888888888888 100 90 80 70 60 ###################################################################### 50 ###################################################################### 40 ###################################################################### 30 ###################################################################### 20 ###################################################################### 10 ###################################################################### 0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.. 0 5 0 5 0 5 0 5 0 5 0 5 0 Free memory per hour (last 72 hours) * = maximum # = average
From our management software , the free memory diagram shows it is continuous reducing until whole system lost responding .
 |
| Free Switch Memory Going Down in Monitoring Tools |
Troubleshooting
There are two main reasons why a switch runs out of memory.
1. A process does not free up the memory after the memory is no longer needed. This is generally known as memory leak.
2. A process does not limit the amount of memory it allocates. Eventually, the process uses up all process memory on the switch.
The goal is to identify the misbehaving process. There are two approaches.
- Identify any process leaking memory
Show process memory debug leak
Notes: This command only to be run during maintenance window or when the device is not in production .This command is handy to identify any process leaking memory. However, it does not help if the memory problem is caused by a process grabs all memory
- Identify a process grabs all memory
Potential fix:
After googled and I did find many related Cisco bug and suggestions.
1. Cisco Bug: CSCut06609 – Memory leak seen in Auth Manager process- leaks from ASP component (Not our case)
https://quickview.cloudapps.cisco.com/quickview/bug/CSCut06609
Last Modified
Jan 31, 2017
Product
Cisco IOS
Known Affected Releases
15.0(2)SE5
Description (partial)
Symptom: ++ Very slow memory leak in "Auth Manager" process seen on 2960s/2960x switches running 15.0(2)SE5. ++ Everyday we see about 64KB leak in Auth Manager process. ++ Leak is seen on a setup having no "Auth session" present (Only auth hidden sessions) and no ports running MAB or dot1X. It is not same as CSCuq53377. ++ The leak is seen in "Auth Manager" process but it is actually being caused by ASP component. Conditions: "macro auto monitor" need to be enabled globally, it could be seen even when no switch port is running MAB/dot1x.
The Fix below fixed all out problems so far:
Apparently, in the IOS there’s an “Auth manager” that can monitor all
sessions in a switch. Starting from the 15.0 IOS stream this feature is
enabled by default. As the bug describes:
“Auth Manager continues to hold more memory in Processor Pool locking
out access to the 3750 stack unless switch is rebooted”
To prevent this ‘leakage’ of memory to the processor, you have to
disable this session monitoring by issuing the following command:
no macro auto monitor
Tried this, but did not help in this case.
2. Cisco Bug: CSCut92215 – Memory Leak in hrpc_incoming_request_handler_process – 15.0(2)EX5 (Not our case)
https://quickview.cloudapps.cisco.com/quickview/bug/CSCut92215
Last Modified
Jan 23, 2017
Product
Cisco Catalyst 2960-X Series Switches
Known Affected Releases
15.0(2)EX
Description (partial)
Symptom: Show process memory sorted will display MallocLite holding memory but no freeing it SW#show process memory sorted Processor Pool Total: XXXXXXXX Used: XXXXXXXX Free: XXXXXXX I/O Pool Total: XXXXXXXX Used: XXXXXXXX Free: XXXXXXX Driver te Pool Total: XXXXXXX Used: XX Free: XXXXXXX PID TTY Allocated Freed Holding Getbufs Retbufs Process 0 0 XXXXXXXX XXXXXXXX XXXXXXXX 0 0 *Init* 0 0 X 0 XXXXXXXX 0 0 *MallocLite* ////////////////// show mem lite-chunk statistics will show a large disparity between Inuse and Free SW#show mem lite-chunk statistics Pool Inuse Free 8-Bytes XXX XXXX 20-Bytes XXXXXX XXXX <<<<<< 44-Bytes XXXX XXX 68-Bytes XXX XXX 96-Bytes XXX XXX 128-Bytes XX XXX Conditions: SW running 15.0(2)EX5
3. SNMP disable (Not our case)
Suggestion to run following three commands to reduce memory usage
– no macro auto monitor
– no cdp running
– snmp completely disabled
– no macro auto monitor
– no cdp running
– snmp completely disabled
Tried those three commands / methods, but still did not help.
4. Memory leak with hulc_flash init process (Our case)
Switch1#show processes memory sorted | i Total Processor Pool Total: 174442388 Used: 50534880 Free: 123907508 I/O Pool Total: 16777216 Used: 6924400 Free: 9852816 Driver te Pool Total: 1048576 Used: 40 Free: 1048536 57447240 Total Switch1#show processes memory sorted | i Total Processor Pool Total: 174442388 Used: 50661608 Free: 123780780 I/O Pool Total: 16777216 Used: 6924400 Free: 9852816 Driver te Pool Total: 1048576 Used: 40 Free: 1048536 57573808 Total You can see Total processor Pool Free memory is decreasing, Free I/O and Driver te Pool memory keep same.
Check more details from sorted processes memory command, *Dead* process memory is increasing by comparing the holding memory data with the data collected 5 minutes ago.
Switch1#show processes memory sorted Processor Pool Total: 174442388 Used: 50519544 Free: 123922844 I/O Pool Total: 16777216 Used: 6924400 Free: 9852816 Driver te Pool Total: 1048576 Used: 40 Free: 1048536 PID TTY Allocated Freed Holding Getbufs Retbufs Process 0 0 59344172 24818044 31046036 0 0 *Init* 0 0 0 0 19543880 0 0 *MallocLite* 0 0 40461580 39067064 1530300 6881007 949166 *Dead* 87 0 715684 146228 451160 0 0 Stack Mgr Notifi 289 0 267228 0 277388 100548 0 EEM ED Syslog 1 0 234876 36256 205780 0 0 Chunk Manager 68 0 206732 232 188500 0 0 USB Startup 205 0 1520792 1070688 154044 0 0 802.1x switch 65 0 108532 400 118292 0 0 EEM ED Identity 180 0 153020 8916 109604 0 0 DHCPD Receive 102 0 93356 212 100304 53760 0 HRPC emac reques 153 1 3418792 3336728 94288 0 0 SSH Process 300 0 70460 0 77620 0 0 EEM Server 178 0 13629416 300 75732 802008 0 CDP Protocol 138 0 65812 232 72740 0 0 REP Helper Proc 53 0 108644 84688 71160 0 0 IF-MGR control p 175 0 530220 373064 62440 0 0 Auth Manager 128 0 232 2204 61160 0 0 HACL Acl Manager 159 0 499433728 499416280 61160 10152 0 HRPC qos request 173 0 49364 0 56524 0 0 ACCT Periodic Pr 280 0 237076 202296 48632 0 0 Crypto CA 21 0 48776 64728 45920 0 9620 Entity MIB API 17 0 211669728 211633028 43860 0 0 ARP Input 189 0 33020 0 40180 0 0 IP ARP Retry Age 297 0 12408 0 37440 0 0 LICENSE AGENT 230 0 50872 9248 34396 0 0 CEF: IPv4 proces 8 0 1547520 3356300 32952 1401600 2400103 Pool Manager Switch1#show processes memory sorted Processor Pool Total: 174442388 Used: 50659912 Free: 123782476 I/O Pool Total: 16777216 Used: 6924400 Free: 9852816 Driver te Pool Total: 1048576 Used: 40 Free: 1048536 PID TTY Allocated Freed Holding Getbufs Retbufs Process 0 0 59344172 24818044 31046036 0 0 *Init* 0 0 0 0 19675040 0 0 *MallocLite* 0 0 44037920 42642260 1531252 6881007 949166 *Dead* 87 0 715684 146228 451160 0 0 Stack Mgr Notifi 289 0 267228 0 277388 100548 0 EEM ED Syslog 1 0 234876 36256 205780 0 0 Chunk Manager 68 0 206732 232 188500 0 0 USB Startup 205 0 1520792 1070688 154044 0 0 802.1x switch 65 0 108532 400 118292 0 0 EEM ED Identity 180 0 153020 8916 109604 0 0 DHCPD Receive 102 0 93356 212 100304 53760 0 HRPC emac reques 153 1 193080 111648 93592 0 0 SSH Process 300 0 70460 0 77620 0 0 EEM Server 178 0 13658484 300 75732 802008 0 CDP Protocol 138 0 65812 232 72740 0 0 REP Helper Proc 53 0 108644 84688 71160 0 0 IF-MGR control p 175 0 530220 373064 62440 0 0 Auth Manager 159 0 510696628 510679180 61160 10152 0 HRPC qos request 128 0 232 2204 61160 0 0 HACL Acl Manager 173 0 49364 0 56524 0 0 ACCT Periodic Pr 280 0 237076 202296 48632 0 0 Crypto CA 21 0 48776 64728 45920 0 9620 Entity MIB API 17 0 212097396 212060696 43860 0 0 ARP Input 189 0 33020 0 40180 0 0 IP ARP Retry Age 297 0 12408 0 37440 0 0 LICENSE AGENT 230 0 50872 9248 34396 0 0 CEF: IPv4 proces 112 0 4349612 232 33052 0 0 HLFM address lea 8 0 1547520 3356300 32952 1401600 2400103 Pool Manager 321 0 288460 157320 31420 0 0 SNMP Traps 199 0 5724 0 30884 0 0 SMI Backup Proce 197 0 3160 592 28360 0 0 SMI Director DB 60 0 74608 52800 27056 0 0 Per-Second Jobs 306 0 232 0 25392 0 0 tHUB From the command ‘show memory dead totals’ output showed there were many instances of hulc_flash init.
Switch1#show memory dead totals Head Total(b) Used(b) Free(b) Lowest(b) Largest(b) Processor 49B2FA0 174442388 50508688 123933700 123791240 103785944 I/O E000000 16777216 6924444 9852772 9736080 9805164 Driver te 2B00000 1048576 44 1048532 1048532 1048532 Dead Proc Summary for: Processor PC Total Count Name 0x020CC35C 942068 47 mifs NOR Sector 0x0036E178 201252 20 SSH Process 0x020D35AC 100220 5 mifs sector buffers 0x01B31C10 65580 1 AAA Large Chunk 0x0017E2D4 64696 517 hulc_flash init 0x020D6A08 25220 5 mifs cache_entry buffers 0x020CC91C 20044 1 mifs NOR Sector Info Pointers 0x0204DE2C 10576 1 master cfg mgr init process 0x02050388 10548 1 HTTP SSL PKI CERTIFICATE CREATE PROC 0x0099C134 9756 1 EEM Shell Director 0x020CC974 8656 10 hulc_flash init 0x0021CBE8 7316 54 Smartport macro autoexec subblock 0x02304ED0 6888 51 LLDP-MED local annex 0x020B0788 6188 1 hulc_flash init 0x01B5F2DC 4652 1 Watcher Info 0x022E3640 4100 52 LIM Index 0x014829EC 3716 3 OBFL UPTIME obfl0 0x020D3DE0 2780 5 hulc_flash init 0x020D3684 2720 5 mifs file_map buffers 0x02883318 2352 7 platform OBFL 0x00477224 2196 1 HTTP SSL PKI CERTIFICATE CREATE PROC 0x004ADE94 2124 1 HTTP SSL PKI CERTIFICATE CREATE PROC 0x020D3618 2040 5 mifs file_info buffers 0x014618C4 1796 5 OBFL UPTIME obfl0 0x020C7A68 1708 24 hulc_flash init 0x020C79F8 1668 24 hulc_flash init 0x020C79BC 1632 24 hulc_flash init 0x01B5E85C 1544 1 Watched Semapho 0x005C40A8 1460 4 HTTP SSL PKI CERTIFICATE CREATE PROC 0x020D6928 1020 15 hulc_flash init 0x020D3DA4 980 5 hulc_flash init 0x020CCA94 460 5 hulc_flash init 0x004ADD80 444 1 HTTP SSL PKI CERTIFICATE CREATE PROC 0x020CCB3C 400 5 hulc_flash init 0x020D08E0 340 5 hulc_flash init 0x020D6C40 340 5 hulc_flash init 0x000730D4 308 1 platform OBFL 0x000739B0 308 1 platform OBFL 0x00073814 308 1 platform OBFL 0x00073B14 308 1 platform OBFL 0x000732A4 308 1 platform OBFL 0x00072E5C 308 1 platform OBFL 0x00343B40 276 3 TTY timer block 0x0045A4EC 232 1 HTTP SSL PKI CERTIFICATE CREATE PROC 0x01489A94 216 1 OBFL Infra Device info 0x01B31A48 208 1 AAA Large Chunk 0x00C1869C 144 1 HTTP 0x003877F0 136 1 Parser Mode 0x020B0734 112 1 hulc_flash init 0x00BF1E80 104 1 HTTP 0x00BFE2FC 100 1 HTTP 0x0059FE88 92 1 crypto_handle_table 0x00BF1B6C 84 1 HTTP 0x020B0808 80 1 hulc_flash init 0x00363AB4 76 1 Parser Linkage 0x002FD83C 68 1 AAA MI SG NAME 0x0038785C 68 1 Parser Mode Q1 0x00387880 68 1 Parser Mode Q2 Dead Proc Summary for: I/O PC Total Count Name Dead Proc Summary for: Driver text PC Total Count Name Switch1#show memory debug leak Adding blocks for GD... Driver text memory Address Size Alloc_pc PID Alloc-Proc Name I/O memory Address Size Alloc_pc PID Alloc-Proc Name Processor memory Address Size Alloc_pc PID Alloc-Proc Name Switch1# Unfortunately, ‘show memory debug leaks’ did not find any process having leak issue.
I were able to find a post from High Memory Utilization due to hulc_flash to mention CSCth60511 and CSCua52463 are relating to this leak. It seems upgrade will fix this issue, but he did not post if issue resolved after upgraded.
Here is what I searched from Cisco Bug Search Website and there are three related bugs:
 |
| Cisco Bug Search Tool |
 |
| Results from Cisco Bug Search Tool |
4.1 CSCth60511 – Memory leak seen at hulc_flash init process
Symptom :
Upon bootup a switch might leak a small amount of memory for hulc_flash init
Process
Further Info :
Since the leak is small it does not have any functionality impact.
Customer Visible
Notifications
Save Bug
Open Support Case
Was the description about this Bug Helpful?
(0)
Last Modified:
Feb 6,2017
Status:
Fixed
Severity:
3 Moderate
| Product: | (1) |
Cisco Catalyst 3750-E Series Switches
Support Cases:
14
| Known Affected Releases: | (4) |
12.2
15.0DPA
15.0DPB
15.0SID
| Known Fixed Releases: | (23) |
15.2(6.0.21i)E
15.2(5.1.57i)E
15.2(5.0)ST
15.2(4.0.64a)E
15.2(4.0)ST
15.2(1.30)PSR
15.2(1.1)EY
15.1(2)SG1.90
15.1(1)SG5.170
15.1(1)SG5.169
15.1(1)SG5.163
15.1(1)SG5.124
15.1(1)SG5.103
15.1(1)SG5.83.1
15.1(1)SG5.78.11
15.1(1)SG5.75
15.1(1)SG5.74
15.1(1)SG15.3
15.0(5.2)DPB1
15.0(4.6)SID
15.0(2)SE
15.0(2)EX
15.0(1)SE
Unfortunately our devices is 2960x and running on 15.0(2)EX3 , which is different from what this bug says.
4.2 CSCua52463 – C2960 – Memory leak in dead memory – not able to telnet/ssh to switch
C2960 – Memory leak in dead memory – not able to telnet/ssh to switch
CSCua52463
C2960 is having memory leak in *dead* memory.
Problem observed on multiple switches.
Problem observed on multiple switches.
arnsw0034#sh mem dead
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 25D8FA0 23727744 19629156 4098588 3362220 4034820
I/O 2C00000 4194304 2473496 1720808 1646820 1609296
Driver te 1800000 1048576 44 1048532 1048532 1048532
———–
arnsw0034#sh proc mem sor
Processor Pool Total: 23727744 Used: 19628532 Free: 4099212
I/O Pool Total: 4194304 Used: 2473356 Free: 1720948
Driver te Pool Total: 1048576 Used: 40 Free: 1048536
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 22527504 4649688 14708156 0 0 *Init*
0 0 3014844 1663092 1290864 3804375 1341026 *Dead*
150 0 14687348 8409544 659316 2268 0 Auth Manager
71 0 736300 143256 472556 0 0 Stack Mgr Notifi
0 0 0 0 394484 0 0 *MallocLite*
157 0 25864596 43517164 335060 0 0 EAP Framework
179 0 2289784 23580 261832 0 0 CDP Protocol
57 0 198956 1686364 208516 0 0 EEM ED ND
———–
From “show mem dead” we can see one process continuously increasing:
…
026A5598 0000000056 026A5538 026A55F8 001 ——– ——– 000196E0 hulc_flash init
026A55F8 0000000056 026A5598 026A5658 001 ——– ——– 000196E0 hulc_flash init
026A5658 0000000040 026A55F8 026A56A8 001 ——– ——– 005360BC hulc_flash init
026A56A8 0000000028 026A5658 026A56EC 001 ——– ——– 00539ED0 hulc_flash init
026A56EC 0000000044 026A56A8 026A5740 001 ——– ——– 00E4C48C hulc_flash init
…
Customer Visible
Notifications
Save Bug
Open Support Case
Was the description about this Bug Helpful?
(0)
Last Modified:
Mar 17,2017
Status:
Other
Severity:
2 Severe
| Product: | (104) |
Cisco IOS
Cisco Catalyst 3560E-12D-E Switch
Cisco Catalyst 3560E-24TD-S Switch
Cisco Catalyst 2960S-24PD-L Switch
Cisco Catalyst 2960-48TT-S Switch
Cisco Catalyst 3750V2-24PS Switch
Cisco Catalyst 2960-24-S Switch
Cisco Catalyst 3750V2-24FS Switch
Cisco Catalyst 3560E-48PD-SF Switch
Cisco Catalyst 2960-24TC-L Switch
Cisco Catalyst 3560E-48PD-E Switch
Cisco Catalyst 3560X-48P-S Switch
Cisco Catalyst 3560-12PC-S Compact Switch
Cisco Catalyst 3560G-48PS Switch
Cisco Catalyst 2960G-24TC-L Switch
Cisco Catalyst 3560X-48PF-S Switch
Cisco Catalyst 2960S-48TD-L Switch
Cisco Catalyst 3750X-24T-S Switch
Cisco Catalyst 3750X-48PF-L Switch
Cisco Catalyst 3750-24FS Switch
Cisco Catalyst 3560G-24TS Switch
Cisco Catalyst 2960S-48LPS-L Switch
Cisco Catalyst 2960-24TC-S Switch
Cisco Catalyst 3750G-12S-SD Switch
Cisco Catalyst 3560X-48PF-L Switch
Cisco Catalyst 2960-24PC-L Switch
Cisco Catalyst 2960-48PST-S Switch
Cisco Catalyst 2960PD-8TT-L Compact Switch
Cisco Catalyst 2960G-48TC-L Switch
Cisco Catalyst 3750E-24TD-E Switch
Cisco Catalyst 3560X-48T-L Switch
Cisco Catalyst 3750E-48PD-E Switch
Cisco Catalyst 3750X-48PF-S Switch
Cisco Catalyst 3750X-24S-S Switch
Cisco Catalyst 2960S-48FPS-L Switch
Cisco Catalyst 2960-48PST-L Switch
Cisco Catalyst 3750G-24TS-1U Switch
Cisco Catalyst 3560E-24PD-S Switch
Cisco Catalyst 2960-48TC-L Switch
Cisco Catalyst 3750X-48T-S Switch
Cisco Catalyst 2960S-24TD-L Switch
Cisco Catalyst 3750E-48PD-EF Switch
Cisco Catalyst 3750V2-48TS Switch
Cisco Catalyst 3560E-12D-S Switch
Cisco Catalyst 3560E-48PD-EF Switch
Cisco Catalyst 3560G-24PS Switch
Cisco Catalyst 2960S-48LPD-L Switch
Cisco Catalyst 2960S-24TS-L Switch
Cisco Catalyst 3750G-48TS Switch
Cisco Catalyst 2960S-48TS-S Switch
Cisco Catalyst 3750X-24T-L Switch
Cisco Catalyst 3560G-48TS Switch
Cisco Catalyst 3560X-24P-L Switch
Cisco Catalyst 3750V2-48PS Switch
Cisco Catalyst 3560V2-24PS Switch
Cisco Catalyst 3750X-48T-L Switch
Cisco Catalyst 3560E-24PD-E Switch
Cisco Catalyst 2960G-8TC-L Compact Switch
Cisco Enhanced Layer 2/3 EtherSwitch Service Module
Cisco Catalyst 3750E-24TD-S Switch
Cisco Catalyst 2960S-48TS-L Switch
Cisco Catalyst 3560X-48P-L Switch
Cisco Catalyst 3750V2-24TS Switch
Cisco Catalyst 3560E-24TD-E Switch
Cisco Catalyst 2960S-24PS-L Switch
Cisco Catalyst 3560V2-24TS Switch
Cisco Catalyst 2960-8TC-S Compact Switch
Cisco Catalyst 3750X-24S-E Switch
Cisco Catalyst 3750E-48TD-S Switch
Cisco Catalyst 3560V2-24DC Switch
Cisco Catalyst 3750E-24PD-E Switch
Cisco Catalyst 2960-8TC-L Compact Switch
Cisco Catalyst 3750X-48P-L Switch
Cisco Catalyst 3560X-48T-S Switch
Cisco Catalyst 3750E-48PD-SF Switch
Cisco Catalyst 3750E-24PD-S Switch
Cisco Catalyst 3560E-12SD-S Switch
Cisco Catalyst 3560V2-48TS Switch
Cisco Catalyst 3750X-24P-L Switch
Cisco Catalyst 3750X-12S-E Switch
Cisco Catalyst 2960-48TT-L Switch
Cisco Catalyst 2960-48TC-S Switch
Cisco Catalyst 3560X-24T-S Switch
Cisco Catalyst 3750X-48P-S Switch
Cisco Catalyst 3560-8PC Compact Switch
Cisco Catalyst 2960S-48FPD-L Switch
Cisco Catalyst 3560X-24T-L Switch
Cisco Catalyst 2960-24LC-S Switch
Cisco Catalyst 3560V2-48PS Switch
Cisco Catalyst 3750G-48PS Switch
Cisco Catalyst 3750X-12S-S Switch
Cisco Catalyst 2960-24PC-S Switch
Cisco Catalyst 2960-24LT-L Switch
Cisco Catalyst 3560E-48PD-S Switch
Cisco Catalyst 3560E-12SD-E Switch
Cisco Catalyst 3750G-24PS Switch
Cisco Catalyst 3560E-48TD-S Switch
Cisco Catalyst 2960-24TT-L Switch
Cisco Catalyst 3560X-24P-S Switch
Cisco Catalyst 3750X-24P-S Switch
Cisco Catalyst 3750E-48PD-S Switch
Cisco Catalyst 3560E-48TD-E Swtch
Cisco Catalyst 3750E-48TD-E Switch
Cisco Catalyst 2960S-24TS-S Switch
Support Cases:
10
4.3 CSCua52463 – C2960 – Memory leak in dead memory – not able to telnet/ssh to switch
hulc_flash init and init memory leaks seen in 3750 during dot1x testing
CSCuw93176
Symptom:
hulc_flash init and init memory leaks seen during dot1x testing on 3750G
hulc_flash init and init memory leaks seen during dot1x testing on 3750G
Conditions:
This issue can be seen with or without any dot1x configured on the 3750G switch. Observed this even with the default config.
Workaround:
No workaround exists for this.
Further Problem Description:
Customer Visible
Notifications
Save Bug
Open Support Case
Was the description about this Bug Helpful?
(0)
Last Modified:
Feb 1,2017
Status:
Other
Severity:
3 Moderate
| Product: | (143) |
Cisco Catalyst 3750 Series Switches
Cisco Catalyst 2960-48TT-S Switch
Cisco Catalyst 2960-24-S Switch
Cisco Catalyst 3560E-12D-E Switch
Cisco Catalyst 2960S-24PD-L Switch
Cisco Catalyst 3560E-48PD-E Switch
Cisco Catalyst 3560X-48T-E Switch
Cisco Catalyst 3560E-48PD-SF Switch
Cisco Catalyst 3560X-48P-S Switch
Cisco ME 3400-24TS-A Switch
Cisco ME 3400EG-2CS-A Switch
Cisco Catalyst 2960G-24TC-L Switch
Cisco Catalyst 3560-12PC-S Compact Switch
Cisco Catalyst 3750G-12S Switch
Cisco Catalyst 3750V2-24FS Switch
Cisco Catalyst 2960-24TC-L Switch
Cisco Catalyst 3560E-24TD-S Switch
Cisco Catalyst 3750V2-24PS Switch
Cisco Catalyst 3560G-48PS Switch
Cisco Catalyst 3560G-24TS Switch
Cisco Catalyst 3560X-48PF-S Switch
Cisco Catalyst 2960S-48LPS-L Switch
Cisco ME 3400E-24TS-M Switch
Cisco IE 3000-8TC Industrial Ethernet Switch
Cisco Catalyst 3750X-48P-E Switch
Cisco Catalyst 3750G-16TD Switch
Cisco Catalyst 3750 Metro 24-AC Switch
Cisco Catalyst 2960S-48TD-L Switch
Cisco Catalyst 2960-24TC-S Switch
Cisco Catalyst 3750-48PS Switch
Cisco Catalyst 3750-24FS Switch
Cisco Catalyst 3750X-48PF-L Switch
Cisco Catalyst 3750X-24T-S Switch
Cisco Catalyst 3560-24TS Switch
Cisco Catalyst 3750X-24S-S Switch
Cisco Catalyst 3560X-48PF-L Switch
Cisco Catalyst 2960G-48TC-L Switch
Cisco Catalyst 2960-48PST-S Switch
Cisco Catalyst 3750X-48PF-S Switch
Cisco Catalyst 3750X-48PF-E Switch
Cisco Catalyst 3750G-24TS-1U Switch
Cisco Catalyst 2960S-48FPS-L Switch
Cisco Catalyst 3750E-24TD-E Switch
Cisco Catalyst 2960-48PST-L Switch
Cisco Catalyst 3750-48TS Switch
Cisco Catalyst 2960PD-8TT-L Compact Switch
Cisco Catalyst 3750E-48PD-E Switch
Cisco Catalyst 2960-24PC-L Switch
Cisco Catalyst 3750 Metro 24-DC Switch
Cisco ME 3400G-12CS-A Switch
Cisco Catalyst 3750G-12S-SD Switch
Cisco Catalyst 3560X-48T-L Switch
Cisco Catalyst 3560E-24PD-S Switch
Cisco ME 3400G-2CS-A Switch
Cisco Catalyst 2960S-24TD-L Switch
Cisco Catalyst 3750G-48TS Switch
Cisco Catalyst 2960S-48LPD-L Switch
Cisco Catalyst 3750V2-48PS Switch
Cisco Catalyst 2350-48TD-SD Switch
Cisco Catalyst 3560-48TS Switch
Cisco Catalyst 3750X-48T-S Switch
Cisco Catalyst 2350-48TD-S Switch
Cisco Catalyst 2960-48TC-L Switch
Cisco Catalyst 3560E-48PD-EF Switch
Cisco ME 3400EG-12CS-M Switch
Cisco Catalyst 3750X-24T-L Switch
Cisco Catalyst 2960S-24TS-L Switch
Cisco Catalyst 3560X-24T-E Switch
Cisco Catalyst 3750V2-48TS Switch
Cisco Catalyst 3750E-48PD-EF Switch
Cisco Catalyst 3560E-12D-S Switch
Cisco Catalyst 3560G-24PS Switch
Cisco Catalyst 3560X-24P-L Switch
Cisco Catalyst 3560G-48TS Switch
Cisco Catalyst 3750G-24TS Switch
Cisco Catalyst 2960S-48TS-S Switch
Cisco Catalyst 2960S-24PS-L Switch
Cisco Catalyst 3750V2-24TS Switch
Cisco Catalyst 3560X-24P-E Switch
Cisco Catalyst 3750X-24P-E Switch
Cisco Catalyst 3560-24PS Switch
Cisco Catalyst 3560E-24PD-E Switch
Cisco Catalyst 3560E-24TD-E Switch
Cisco Catalyst 3750X-48T-L Switch
Cisco Enhanced Layer 2/3 EtherSwitch Service Module
Cisco Catalyst 3560V2-24PS Switch
Cisco Catalyst 2960S-48TS-L Switch
Cisco Catalyst 3560X-48P-L Switch
Cisco Catalyst 3750E-24TD-S Switch
Cisco Catalyst 2960G-8TC-L Compact Switch
Cisco Catalyst 3750X-24T-E Switch
Cisco Catalyst 3750X-48P-L Switch
Cisco Catalyst 2960-8TC-L Compact Switch
Cisco Catalyst 3560E-12SD-S Switch
Cisco ME 3400-24FS-A Switch
Cisco Catalyst 3560X-48T-S Switch
Cisco Catalyst 3750E-48TD-S Switch
Cisco Catalyst 3750E-48PD-SF Switch
Cisco Catalyst 3560-48PS Switch
Cisco Catalyst 3560V2-24DC Switch
Cisco Catalyst 3560V2-24TS Switch
Cisco Catalyst 3750E-24PD-E Switch
Cisco Catalyst 3750G-24T Switch
Cisco Catalyst 3750E-24PD-S Switch
Cisco Catalyst 3750X-24S-E Switch
Cisco Catalyst 2960-8TC-S Compact Switch
Cisco Catalyst 3560X-48P-E Switch
Cisco Catalyst 2975 Switch
Cisco Catalyst 3750X-48T-E Switch
Cisco Catalyst 3750-24TS Switch
Cisco Catalyst 3560V2-48PS Switch
Cisco Catalyst 2960-24LC-S Switch
Cisco Catalyst 3750X-48P-S Switch
Cisco Catalyst 2960-48TT-L Switch
Cisco Enhanced Layer 2 EtherSwitch Service Module
Cisco Catalyst 3750X-12S-E Switch
Cisco Catalyst 3560V2-48TS Switch
Cisco Catalyst 3560X-24T-S Switch
Cisco Catalyst 2960S-48FPD-L Switch
Cisco Catalyst 3750X-24P-L Switch
Cisco Catalyst 3560-8PC Compact Switch
Cisco Catalyst 2960-48TC-S Switch
Cisco Catalyst 3560X-24T-L Switch
Cisco Catalyst 3750X-12S-S Switch
Cisco Catalyst 3560X-24P-S Switch
Cisco Catalyst 3750-24PS Switch
Cisco Catalyst 3560E-48TD-S Switch
Cisco Catalyst 3560E-12SD-E Switch
Cisco Catalyst 2960-24TT-L Switch
Cisco Catalyst 2960S-24TS-S Switch
Cisco Catalyst 3750E-48PD-S Switch
Cisco ME 3400G-12CS-D Switch
Cisco Catalyst 3750E-48TD-E Switch
Cisco Catalyst 3560E-48PD-S Switch
Cisco IE 3000-4TC Industrial Ethernet Switch
Cisco Catalyst 3750X-24P-S Switch
Cisco Catalyst 2960-24LT-L Switch
Cisco ME 3400-24TS-D Switch
Cisco Catalyst 3750G-24PS Switch
Cisco Catalyst 2960-24PC-S Switch
Cisco Catalyst 3560X-48PF-E Switch
Cisco Catalyst 3560E-48TD-E Swtch
Cisco Catalyst 3750G-48PS Switch
Support Cases:
1
Solution
Sadly although we found on bug CSCua52463 seems perfectly matching our case, Cisco does not post any solution and fix for it.
Downgrading to 12.2 or upgrading to 15.2(2)E3 will be our last step to try at this moment. Actually upgrading to 15.2(2)E3 resolved this issue.
Here is the monitoring information from NetMRI. After upgraded IOS to 15.2(2)E3, the free memory is getting stable. It was reducing slowly every minute until upgraded to 15.2(2)E3.
| Type: | Switch (99%) | Vendor: | Cisco |
| O/S Version: | 15.2(2)E3 | Model: | catWsC2960x48tsLL |
| Up Time: | 1d 00h 17m 42s | SNMP Status: | Enabled |
| Last Communication: | 2017-03-27 10:13:30 | MAC Address: | 5C:A4:8A:5C:70:C1 |
| Discovery Blackout: | N/A | Change Blackout: | N/A |
 |
| Free Switch Memory Status in Monitoring Tools |
Reference: