Packet Loss/ High CPU process issue - 3640 12.2(19a)

Hiya,

Got an issue with packet loss on the network that our monitoring box has flagged up. Upon further inspection this doesn't appear to be related to the WAN links but a issue directly on the remote router itself. All interface stats on the device in question are clean.

A ' show proc cpu' shows up the following process as possibly a cause of the issue.

See 5 sec util and Encrypt Proc process. Not 100% if this is the root cause???? Is running 12.2(19a). There are a load of crypto maps on this device as well.



CPU utilization for five seconds: 99%/6%; one minute: 55%; five minutes: 45%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
3 15938092 82828870 192 0.16% 0.06% 0.05% 0 OSPF Hello
11 25174836 54423751 462 0.08% 0.08% 0.08% 0 ARP Input
32 23206075562546334967 0 11.68% 8.96% 7.08% 0 IP Input
61 27366436 57065669 479 2.02% 0.01% 0.36% 0 Crypto Support
70 3752685156 710265501 5283 77.43% 37.40% 28.70% 0 Encrypt Proc
82 25978012 49467466 525 0.16% 0.15% 0.15% 0 IP SNMP
84 45641636 25033859 1823 0.24% 0.20% 0.26% 0 SNMP ENGINE
105 3772 462 8164 0.81% 0.34% 0.32% 131 SSH Process

Normal results below:


CPU utilization for five seconds: 14%/4%; one minute: 14%; five minutes: 16%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
32 23210711722546906950 0 5.89% 5.33% 5.39% 0 IP Input
70 3753381220 710362313 5283 2.37% 3.94% 4.91% 0 Encrypt Proc
95 1540 372 4139 1.06% 0.11% 0.04% 130 SSH Process
84 45661404 25046184 1823 0.49% 0.32% 0.29% 0 SNMP ENGINE
82 25989872 49491863 525 0.32% 0.17% 0.16% 0 IP SNMP
83 9237588 25019361 369 0.24% 0.08% 0.06% 0 PDU DISPATCHER

Cant find much in the way of info on Encrypt Proc though I think this is related to the IPSEC tunnels????


Any ideas.

Thanks all

IP-bod

Hey IP-bod,

Could you provide us some information on the configuration as well as how many VPN connections going through this router and if there were any changes before the high cpu util?

If you have a lot of VPN traffic going through this router, it may be overwhelming it. The 3640 can only handle about 1.8 Mbps of VPN traffic. Reason why is all the VPN stuff is done in software as well as the router's other processes... An add on module, the NM-VPN/MP will increase it about tenfold, but that could be pricey. A quick search on Froogle shows it's about $2000 for the module.

Hi jwj,

Thanks for the response. Yep this is all done in software. Am not to sure i understand the vpn config but will give it a try.

These are the crypto maps configured:
!
crypto ipsec transform-set xxx esp-3des esp-md5-hmac
crypto ipsec transform-set xx esp-3des esp-md5-hmac
crypto ipsec transform-set lxxxx esp-3des esp-md5-hmac
crypto ipsec transform-set xxx esp-3des esp-md5-hmac
crypto ipsec transform-set xxo esp-3des esp-sha-hmac
mode transport
crypto ipsec transform-set hk esp-3des esp-md5-hmac

!
crypto map xxx 10 ipsec-isakmp
set peer x.x.x.x
set transform-set xxx
match address 182
crypto map xxx 20 ipsec-isakmp
set peer x.x.x.x
set transform-set xxxx
match address 184
crypto map xxx 30 ipsec-isakmp
set peer x.x.x.x
set transform-set london
match address 186
crypto map xxx 41 ipsec-isakmp
set peer x.x.x.x
set transform-set ny
match address 185
crypto map xxx 50 ipsec-isakmp
set peer x.x.x.x
set transform-set xxx
set pfs group5
match address 181
crypto map xxx 60 ipsec-isakmp
set peer x.x.x.x
set transform-set xxx
match address 188
!

and its got a handful of tunnels interfaces, though no traffic at all on them - suppose they not being used then?

How would I go about finding the amount of traffic this is generating??

sh crypto engine connections active - perhaps??? Or one of these ?

#sh crypto engine connections ?
active Show all crypto engine active connections
dh Show crypto engine D-H table entries
dropped-packet Show crypto engine dropped packets
flow Show crypto engine flow table entries


I'll have to check a little later as the network is not at its peak at this time in australia. I get the impression though this may not be traffic related as I took thoses process results yesterday same time.

Packet loss is still there 990/1000 packets successful from our link in ny direct to aus but still flagging network alerts throughout the day.

ethernet and serial doing the below respectively. So not really traffic intensive stuff.

5 minute input rate 98000 bits/sec, 48 packets/sec
5 minute output rate 102000 bits/sec, 40 packets/sec

5 minute input rate 10000 bits/sec, 6 packets/sec
5 minute output rate 26000 bits/sec, 6 packets/sec

The box has been up for over year and a half so am thinking about reloading it as well.

Hope this is enough info ?

IP-bod

The tunnel interfaces are usually used for GRE. Probably not being used. Which interface(s) are the crypto maps applied? Here's a couple of show commands that may help you get some more info for troubleshooting:

show crypto engine connections active (yes you are right about this)
show crypto session detail
show crypto isakmp sa
show crypto ipsec sa

Hiya,

I think from this they are being applied to ether0/1:

xxx#show crypto engine connections active

ID Interface IP-Address State Algorithm Encrypt Decrypt
1 <none> <none> set HMAC_MD5+3DES_56_C 0 0
3 <none> <none> set HMAC_MD5+3DES_56_C 0 0
5 <none> <none> set HMAC_MD5+3DES_56_C 0 0
6 <none> <none> set HMAC_MD5+3DES_56_C 0 0
51 <none> <none> set HMAC_MD5+3DES_56_C 0 0
54 Ethernet0/1 x.x.100.1 set HMAC_MD5+3DES_56_C 0 0
2162 Ethernet0/1 x.x.100.1 set HMAC_MD5+3DES_56_C 0 1089
2163 Ethernet0/1 x.x.100.1 set HMAC_MD5+3DES_56_C 1116 0
2164 Ethernet0/1 x.x.100.1 set HMAC_MD5+3DES_56_C 0 10220
2165 Ethernet0/1 x.x.100.1 set HMAC_MD5+3DES_56_C 8567 0
2166 Ethernet0/1 x.x.100.1 set HMAC_SHA+3DES_56_C 0 6528
2167 Ethernet0/1 x.x.100.1 set HMAC_SHA+3DES_56_C 7795 0
2168 Ethernet0/1 x.x.100.1 set HMAC_MD5+3DES_56_C 0 243
2169 Ethernet0/1 x.x.100.1 set HMAC_MD5+3DES_56_C 244 0
2170 Ethernet0/1 x.x.100.1 set HMAC_MD5+3DES_56_C 0 242
2171 Ethernet0/1 x.x.100.1 set HMAC_MD5+3DES_56_C 240 0
2172 Ethernet0/1 x.x.100.1 set HMAC_MD5+3DES_56_C 0 0
2173 Ethernet0/1 x.x.100.1 set HMAC_MD5+3DES_56_C 49 0#



But the good news is, I have reloaded this router and the loss has gone away! Well it was up for over year and a half.


Thanks again JWJ for the help.

That's good. Hopefully you'll have at least another year and a half of uptime.