Dynamic Multipoint VPN (DMVPN)
Troubleshooting
Solutions
Luke Bibby, CCIE #45527
Quick Overview of Scenario Solutions
Scenario #1
Incorrect NHRP mapping on R2
No NHS configuration on R3
Scenario #2
Wrong NRHP authentication key on R2
NHRP Shortcut and redirect on the wrong routers
Scenario #3
Incorrect transform set on R3
Incorrect ISAKMP key on R2
Scenario #4
Incorrect NHRP mappings on R2 and R3
No NHS configuration on R2
Scenario #5
R1, R2 and R3 have the wrong ESP mode
R4 has no NAT configuration for UDP/4500
R2 has the wrong ISAKMP key peer
R3 is missing the IPsec protection profile under the DMVPN interface
Scenario #1 Solutions
When we run the following command we notice that R1 does not know about any of the DMVPN
spokes (this is the same output as from the scenario):
R1#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
Let’s check the NRHP cache on the spokes R2 and R3 to see if they have a mapping to the hub.
R2#show ip nhrp
10.1.1.1/32 via 10.1.1.1
Tunnel100 created 00:04:47, never expire
Type: static, Flags:
NBMA address: 192.168.123.1
Well a mapping exists to R1 so let’s assume that this is not the problem. Can R2 ping R1’s NBMA
address when sourced from its own NBMA address? We will also run a generalised NRHP debug to
see if there are hints on the problem.
R2#debug nhrp
NHRP protocol debugging is on
R2#ping 192.168.123.1 source e0/0 repeat 1
Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 192.168.123.1, timeout is 2 seconds:
Packet sent with a source address of 10.0.12.2
*Dec 7 00:36:08.506: NHRP: NHRP could not map 192.168.123.1 to NBMA, cache entry
not found
*Dec 7 00:36:08.506: NHRP: MACADDR: if_in null netid-in 0 if_out Tunnel100 netid-
out 132
*Dec 7 00:36:08.506: NHRP: Checking for delayed event NULL/192.168.123.1 on list
(Tunnel100).
*Dec 7 00:36:08.506: NHRP: No node found..
Success rate is 0 percent (0/1)
R2#
*Dec 7 00:36:16.207: NHRP: Checking for delayed event NULL/192.168.123.1 on list
(Tunnel100).
*Dec 7 00:36:16.207: NHRP: No node found.
*Dec 7 00:36:16.207: NHRP-ATTR: Requester Ext Len: Total ext_len with NHRP
attribute VPE 20
*Dec 7 00:36:16.207: NHRP: Sending NHRP Resolution Request for dest: 192.168.123.1
to nexthop: 192.168.123.1 using our src: 192.168.123.2
*Dec 7 00:36:16.207: NHRP: Attempting to send packet via DEST 192.168.123.1
*Dec 7 00:36:16.207: NHRP: NHRP could not map 192.168.123.1 to NBMA, cache entry
not found
*Dec 7 00:36:16.207: NHRP: Send Resolution Request via Tunnel100 vrf 0, packet
size: 72
*Dec 7 00:36:16.207: src: 192.168.123.2
R2#, dst: 192.168.123.1
*Dec 7 00:36:16.207: NHRP: Encapsulation failed for destination 192.168.123.1 out
Tunnel100
The above highlighted line is indicating a problem with the mapping to R1. Let’s check the IP
addresses configured on R2 to see if there is an addressing problem.
R2#show ip int br | exc una
Interface IP-Address OK? Method Status
Protocol
Ethernet0/0 10.0.12.2 YES manual up up
Tunnel100 192.168.123.2 YES manual up up
We can now see the problem when we look at the IP address configured on R2’s tunnel address. We
saw earlier that R2 had a mapping to R1’s NBMA address of 192.168.123.1, which is in the same
network as R2’s tunnel address. This is leading us towards thinking that there is a problem with R2’s
NHRP mapping. Let’s see what the configuration on R2’s tunnel looks like.
R2#show run int tun100
Building configuration...
Current configuration : 258 bytes
!
interface Tunnel100
ip address 192.168.123.2 255.255.255.0
no ip redirects
ip nhrp map 10.1.1.1 192.168.123.1
ip nhrp map multicast 10.1.1.1
ip nhrp network-id 132
ip nhrp nhs 192.168.123.1
tunnel source Ethernet0/0
tunnel mode gre multipoint
end
And now we’ve confirmed the NRHP map is the problem. Let’s fix this on R2.
R2(config)#int tun100
R2(config-if)#no ip nhrp map 10.1.1.1 192.168.123.1
R2(config-if)#ip nhrp map 192.168.123.1 10.1.1.1
Let’s now try to ping R1’s tunnel interface.
R2#ping 192.168.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.123.1, timeout is 2 seconds:
..!!!
Success rate is 60 percent (3/5), round-trip min/avg/max = 1/1/1 ms
We should see on R1 that R2 now appears as a DMVPN spoke.
R1#show dmvpn
Legend: Attrb S – Static, D – Dynamic, I – Incomplete
N – NATed, L – Local, X – No Socket
# Ent Number of NHRP entries with same NBMA peer
NHS Status: E Expecting Replies, R Responding, W Waiting
UpDn Time Up or Down Time for a Tunnel
Interface: Tunnel100, Ipv4 NHRP Details
Type:Hub, NHRP Peers:1,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 10.0.12.2 192.168.123.2 UP 00:00:50 D
R3 is still not showing up as DMVPN spoke however so let’s see if the mapping is wrong there too.
R3#show ip nhrp
192.168.123.1/32 via 192.168.123.1
Tunnel100 created 00:20:20, never expire
Type: static, Flags:
NBMA address: 10.1.1.1
This looks fine. Can R3 ping R1’s NBMA address when sourcing the ping from it’s own NBMA
address?
R3#ping 10.1.1.1 source e0/0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 10.0.13.3
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
This proves that there is no underlying routing problem between the tunnel source endpoints. Can
R3 ping R1’s tunnel address?
R3#ping 192.168.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.123.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
As mostly expected this does not work (otherwise R1 would have seen R3 as a DMVPN spoke
earlier). Let’s debug NHRP to see if we can isolate the problem.
R3#ping 192.168.123.1 repeat 1
Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 192.168.123.1, timeout is 2 seconds:
*Dec 7 00:51:49.697: NHRP: Setting ‘used’ flag on cache entry with nhop:
192.168.123.1
*Dec 7 00:51:49.698: NHRP: NHRP successfully mapped ‘192.168.123.1’ to NBMA
‘10.1.1.1’.
Success rate is 0 percent (0/1)
The output seems a lot more quiet than what we saw when we did the same on R2. Notably, there
are no NHRP Registration Request messages bent sent out, which would normally be sent to the
NHRP NHS so that R3 can register itself to the hub router. Let’s see what NHRP NHS is configured on
R3’s tunnel interface.
R3#show run int tun100
Building configuration...
Current configuration : 231 bytes
!
interface Tunnel100
ip address 192.168.123.3 255.255.255.0
no ip redirects
ip nhrp map 192.168.123.1 10.1.1.1
ip nhrp map multicast 10.1.1.1
ip nhrp network-id 123
tunnel source Ethernet0/0
tunnel mode gre multipoint
end
And now we can see the problem: R3 is a spoke but does not have an NHS configured so where is it
supposed to send the NHRP Resolution Request messages to? Let’s fix this.
R3#conf t
R3(config)#int tun100
R3(config-if)#ip nhrp nhs 192.168.123.1
Let’s see now on R1 if R3 has successfully registered itself to the DMVPN.
R1#show dmvpn
Legend: Attrb S – Static, D – Dynamic, I – Incomplete
N – NATed, L – Local, X – No Socket
# Ent Number of NHRP entries with same NBMA peer
NHS Status: E Expecting Replies, R Responding, W Waiting
UpDn Time Up or Down Time for a Tunnel
Interface: Tunnel100, Ipv4 NHRP Details
Type:Hub, NHRP Peers:2,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 10.0.12.2 192.168.123.2 UP 00:13:41 D
1 10.0.13.3 192.168.123.3 UP 00:02:40 D
This looks much better! The scenario told us to verify that R2 could ping R3 so let’s now prove that
our fixes have resolved the original problems.
R2#ping 192.168.123.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.123.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/5 ms
And we should also see an entry for R3 in the NHRP cache on R2.
R2#show ip nhrp
192.168.123.1/32 via 192.168.123.1
Tunnel100 created 00:15:59, never expire
Type: static, Flags: used
NBMA address: 10.1.1.1
192.168.123.2/32 via 192.168.123.2
Tunnel100 created 00:00:04, expire 01:59:55
Type: dynamic, Flags: router unique local
NBMA address: 10.0.12.2
(no-socket)
192.168.123.3/32 via 192.168.123.3
Tunnel100 created 00:00:04, expire 01:59:55
Type: dynamic, Flags: router
NBMA address: 10.0.13.3
Scenario #2 Solutions
The scenario indicated that R2 is having problems registering itself to R1 so let’s first check the NRHP
mapping on R2.
R2#show ip nhrp
192.168.123.1/32 via 192.168.123.1
Tunnel100 created 00:04:40, never expire
Type: static, Flags: used
NBMA address: 10.1.1.1
According to the diagram it looks fine. Let’s see if R2 can ping R1’s NBMA address sourced from it’s
own NBMA address.
R2#ping 192.168.123.1 source e0/0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.123.1, timeout is 2 seconds:
Packet sent with a source address of 10.0.12.2
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/5 ms
No problems there. Let’s run a generalised NHRP debug and try and ping over the tunnel.
R2#debug nhrp
NHRP protocol debugging is on
R2#
R2#ping 192.168.123.1 rep 1
Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 192.168.123.1, timeout is 2 seconds:
*Dec 7 01:16:34.367: NHRP: Setting 'used' flag on cache entry with nhop:
192.168.123.1
*Dec 7 01:16:34.367: NHRP: NHRP successfully mapped '192.168.123.1' to NBMA
'10.1.1.1'
*Dec 7 01:16:35.958: NHRP: Setting retrans delay to 64 for nhs dst 192.168.123.1
*Dec 7 01:16:35.958: NHRP-ATTR: Requester Ext Len: Total ext_len with NHRP
attribute VPE 55
*Dec 7 01:16:35.958: NHRP: Attempting to send packet via DEST 192.168.123.1
*Dec 7 01:16:35.958: NHRP: Setting 'used' flag on cache entry with nhop:
192.168.123.1
*Dec 7 01:16:35.958: NHRP: NHRP successfully mapped '192.168.123.1' to NBMA
'10.1.1.1'
*Dec 7 01:16:35.958: NHRP: Encapsulation succeeded. Sending NHRP Control Packet
NBMA Address: 10.1.1.1
*Dec 7 01:16:35.958: NHRP: Send Registration Request via Tunnel100 vrf 0, packet
size: 107
*Dec 7 01:16:35.958: src: 192.168.123.2.
Success rate is 0 percent (0/1)
R2#, dst: 192.168.123.1
*Dec 7 01:16:35.958: NHRP: 131 bytes out Tunnel100
This confirms that there is no mapping problems, and an NHS must be configured otherwise no
NHRP Registration Request would be sent. Let’s see if R1 has an NHRP cache entry for R2.
R1#show ip nhrp
192.168.123.2/32
Tunnel100 created 00:02:22, expire 00:00:42
Type: incomplete, Flags: negative
Cache hits: 7
It has an entry but it is incomplete. So if R2 can successfully encapsulate packets for the tunnel, and
R1 is receiving them (so it shouldn’t be a filtering issue) but is marking them as complete, what else
could be wrong? Let’s check the configuration of the tunnel on R2’s end.
R2#show run int tun100
Building configuration...
Current configuration : 290 bytes
!
interface Tunnel100
ip address 192.168.123.2 255.255.255.0
no ip redirects
ip nhrp authentication luke123
ip nhrp map multicast 10.1.1.1
ip nhrp map 192.168.123.1 10.1.1.1
ip nhrp network-id 132
ip nhrp nhs 192.168.123.1
tunnel source Ethernet0/0
tunnel mode gre multipoint
end
We can see that authentication is in use for NHRP. If there was an NHRP authentication issue this
could indicate why R1 is marking the NRHP cache as incomplete. Let’s see what key R1 has
configured on the tunnel.
R1#show run int tun100 | inc auth
ip nhrp authentication 1uke123
If you look carefully, R2’s key is “luke123” and R1’s key is “1uke123”. Seeing as we don’t know which
is the correct key, let’s check what R3’s configured key is.
R3#show run int tun100 | inc auth
ip nhrp authentication 1uke123
So R1 and R3 have the correct key, so let’s change the key on R2.
R2(config)#int tun100
R2(config-if)#ip nhrp authentication 1uke123
Soon after configuring this you should see the following message printed to the console indicating
that EIGRP has now formed an adjacency over the tunnel.
R2#
*Dec 7 01:30:30.988: %DUAL-5-NBRCHANGE: EIGRP-IPv4 123: Neighbor 192.168.123.1
(Tunnel100) is up: new adjacency
Now let’s see if we are receiving EIGRP routes from R1 now.
R2#show ip route eigrp
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override
Gateway of last resort is not set
192.168.33.0/32 is subnetted, 1 subnets
D 192.168.33.3 [90/102400640] via 192.168.123.1, 00:00:14, Tunnel100
And let’s see if R3 is receiving an EIGRP route to R2’s loopback over the tunnel.
R3#show ip route eigrp
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override
Gateway of last resort is not set
192.168.22.0/32 is subnetted, 1 subnets
D 192.168.22.2 [90/102400640] via 192.168.123.1, 00:01:31, Tunnel100
So the control plane appears to be setup correctly for the two spokes to be able to ping each other’s
loopbacks, so let’s try it out.
R2#ping 192.168.33.3 source loop192
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.33.3, timeout is 2 seconds:
Packet sent with a source address of 192.168.22.2
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Great! Now let’s see if we can match the verification output from the troubleshooting scenario with
a traceroute.
R2#traceroute 192.168.33.3 source loop192 num
Type escape sequence to abort.
Tracing the route to 192.168.33.3
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.123.1 1 msec 0 msec 1 msec
2 192.168.123.3 1 msec * 1 msec
We are still seeing R1 as a hop, so let’s try the same traceroute again (the original scenario said we
may have to try this a few times).
R2#traceroute 192.168.33.3 source loop192 num
Type escape sequence to abort.
Tracing the route to 192.168.33.3
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.123.1 1 msec 0 msec 0 msec
2 192.168.123.3 1 msec * 1 msec
There are a few ways to fix this:
1. Set “no next-hop-self” under the EIGRP process on R1
2. Use NHRP next-hop override (NHO)
If you look back to the original scenario, the EIGRP route on R2 to R3’s looback had the percent
symbol next to the route, indicating that NHRP has overridden the next-hop. We can confirm that
this is true if we look at the route codes at the top of the routing table output.
R2#show ip route | inc \%
+ - replicated route, % - next hop override
Let’s look at the current next-hop for R3’s loopback from R2’s perspective.
R2#show ip route eigrp
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override
Gateway of last resort is not set
192.168.33.0/32 is subnetted, 1 subnets
D 192.168.33.3 [90/102400640] via 192.168.123.1, 03:04:29, Tunnel100
For NHO to work a few things need to be in place:
1. The hub router, R1 in this case, needs to be configured to send the NHRP Redirect Indicator
when it notices that ingress and egress interface are the same for traffic
2. The spoke routers, R2 and R3 in this case, needs to be configured to install the shortcut, or
rather, the NHO
Let’s use the generalised NHRP debug to see if R1 is actually sending the NHRP redirect indicator
when we ping between the loopbacks on R2 and R3.
R2#debug nhrp
NHRP protocol debugging is on
R2#ping 192.168.33.3 source loopback192 rep 1
Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 192.168.33.3, timeout is 2 seconds:
Packet sent with a source address of 192.168.22.2
!
Success rate is 100 percent (1/1), round-trip min/avg/max = 6/6/6 ms
R2#
*Dec 7 04:42:36.311: NHRP: Setting 'used' flag on cache entry with nhop:
192.168.123.1
*Dec 7 04:42:36.311: NHRP: NHRP successfully mapped '192.168.123.1' to NBMA
'10.1.1.1'
This is not what we want: we need to see R1 send the NHRP redirect so that we can install the NHO.
Let’s see if R1 is configured to send this.
R1#sh run int tun100
Building configuration...
Current configuration : 261 bytes
!
interface Tunnel100
description DMVPN
ip address 192.168.123.1 255.255.255.0
no ip redirects
ip nhrp authentication 1uke123
ip nhrp map multicast dynamic
ip nhrp network-id 123
ip nhrp shortcut
tunnel source Loopback0
tunnel mode gre multipoint
end
In this case, R1 has been configured to install an NHO rather to send the NHRP redirects. It won’t
actually affect the operation if we leave this here and add the configuration to send NHRP redirects,
but on the CCIE lab it is plausible that you could be penalized for having additional unnecessary
configuration so in my opinion is better to take this line out and add in the right configuration.
R1(config)#int tun100
R1(config-if)#no ip nhrp shortcut
R1(config-if)#ip nhrp redirect
Let’s try the traceroute again on R2 again and also leave the generalised NHRP debug on.
R2#trace 192.168.33.3 source loop192 num
Type escape sequence to abort.
Tracing the route to 192.168.33.3
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.123.1 0 msec 1 msec 0 msec
2 192.168.123.3 1 msec
*Dec 7 04:50:33.267: NHRP: Setting 'used' flag on cache entry with nhop:
192.168.123.1
*Dec 7 04:50:33.267: NHRP: NHRP successfully mapped '192.168.123.1' to NBMA
'10.1.1.1'
*Dec 7 04:50:33.267: NHRP: Setting 'used' flag on cache entry with nhop:
192.168.123.1
*Dec 7 04:50:33.267: NHRP: NHRP successfully mapped '192.168.123.1' to NBMA
'10.1.1.1'
*Dec 7 04:50:33.268: NHRP: Setting 'used' flag on cache entry with nhop:
192.168.123.1
*Dec 7 04:50:33.268: NHRP: NHRP successfully mapped '192.168.123.1' to NBMA
'10.1.1.1'
*Dec 7 04:50:33.268: NHRP: Setting 'used' flag on cache entry with nhop:
192.168.123.1
*Dec 7 04:50:33.268: NHRP: NHRP successfully mapped '192.168.123.1' to NBMA
'10.1.1.1'
*Dec 7 04:50:33.269: NHRP-ATTR: ext_type: 32772, ext_len : 0
*Dec 7 04:50:33.269: NHRP-ATTR: ext_type: 32773, ext_len : 0
*Dec 7 04:50:33.269: NHRP-ATTR: ext_type: 32775, ext_len : 11
*Dec 7 04:50:33.269: NHRP-ATTR: ext_type: 9, ext_len : 0
*Dec 7 04:50:33.269: NHRP-ATTR: ext_type: 32768, ext_len : 0
*Dec 7 04:50:33.269: NHRP: Receive Traffic Indication via Tunnel100 vrf 0, packet
size: 99
*Dec 7 04:50:33.269: NHRP: netid_in = 132, to_us = 0
*Dec 7 04:50:33.269: NHRP: nhrp_rtlookup yielded Loopback192
*Dec 7 04:50:33.269: NHRP: netid_out 0, netid_in 132
*Dec 7 04:50:33.269: NHRP: Parsing NHRP Traffic Indication
*Dec 7 04:50:33.269: NHRP: Setting 'used' flag on cache entry with nhop:
192.168.123.1
*Dec 7 04:50:33.269: NHRP: NHRP successfully mapped '192.168.123.1' to NBMA
'10.1.1.1'
This is also not what we want! We should that after receiving the NRHP redirect that R2 sends an
NHRP Resolution Request for 192.168.33.3 to try and identify the correct spoke to send this traffic to
directly. Let’s see if shortcut is configured on R2 and R3.
R2#show run int tun100 | inc shortcut
R3#show run int tun100 | inc shortcut
Well this is our next probem; shortcut is not configured. Let’s configure this on R2 and R3.
R2(config)#int tun100
R2(config-if)#ip nhrp shortcut
R3(config)#int tun100
R3(config-if)#ip nhrp shortcut
Now let’s try our traceroute again.
R2#trace 192.168.33.3 source loop192 num
Type escape sequence to abort.
Tracing the route to 192.168.33.3
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.123.3 1 msec * 5 msec
This is what we want! We are sending traffic directly to R3 now. Let’s also confirm R2’s routing table
for R3’s loopback to see if we have the NHO code next to the route.
R2#show ip route eigrp
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override
Gateway of last resort is not set
192.168.33.0/32 is subnetted, 1 subnets
D % 192.168.33.3 [90/102400640] via 192.168.123.1, 00:09:11, Tunnel100
Scenario #3 Solutions
In the original scenario we were shown something similar to the following output on R1.
R1#show crypto session
Crypto session current status
Interface: Tunnel100
Session status: UP-ACTIVE
Peer: 10.0.12.2 port 500
IKEv1 SA: local 10.1.1.1/500 remote 10.0.12.2/500 Active
IPSEC FLOW: permit 47 host 10.1.1.1 host 10.0.12.2
Active SAs: 4, origin: crypto map
nterface: Ethernet0/1
Session status: UP-IDLE
Peer: 10.0.13.3 port 500
IKEv1 SA: local 10.1.1.1/500 remote 10.0.13.3/500 Active
The first thing we can notice about the output is that R3 has established a pair of IPsec SAs with R2
but not with R3. Let’s run an IPsec debug on R1 to see if that highlights any errors for us.
R1#debug crypto ipsec
Crypto IPSEC debugging is on
Then restart the crypto sessions on R3 using the following command.
R3#clear crypto session
Then observe the debug output on R1.
R1#
*Dec 7 05:10:24.737: IPSEC(validate_proposal_request): proposal part #1
*Dec 7 05:10:24.737: IPSEC(validate_proposal_request): proposal part #1,
(key eng. msg.) INBOUND local= 10.1.1.1:0, remote= 10.0.13.3:0,
local_proxy= 10.1.1.1/255.255.255.255/47/0,
remote_proxy= 10.0.13.3/255.255.255.255/47/0,
protocol= ESP, transform= NONE (Transport),
lifedur= 0s and 0kb,
spi= 0x0(0), conn_id= 0, keysize= 128, flags= 0x0
*Dec 7 05:10:24.738: map_db_find_best did not find matching map
*Dec 7 05:10:24.738: IPSEC(ipsec_process_proposal): transform proposal not
supported for identity:
{esp-aes esp-sha-hmac }
This indicates that R1 and R3 cannot find a common transform set to agree upon, so the IPsec SAs
never get established. Let’s see what transform sets R1 and R3 will attempt to negotiate.
R1#show crypto ipsec transform-set
Transform set default: { esp-aes esp-sha-hmac }
will negotiate = { Tunnel, },
Transform set TS: { esp-192-aes esp-sha-hmac }
will negotiate = { Transport, },
R3#show crypto ipsec transform-set
Transform set default: { esp-aes esp-sha-hmac }
will negotiate = { Tunnel, },
Transform set TS: { esp-aes esp-sha-hmac }
will negotiate = { Transport, },
We can already see that the non-default transform set is different, so let’s see if this is the one
actually being used over the DMVPN.
R1#sh run int tun100 | inc ipsec
tunnel protection ipsec profile IPSEC_PROF
R1#sh crypto ipsec profile IPSEC_PROF
IPSEC profile IPSEC_PROF
Security association lifetime: 4608000 kilobytes/3600 seconds
Responder-Only (Y/N): N
PFS (Y/N): N
Transform sets={
TS: { esp-192-aes esp-sha-hmac } ,
}
And let’s check the same on R3.
R3#show crypto ipsec prof IPSEC_PROF
IPSEC profile IPSEC_PROF
Security association lifetime: 4608000 kilobytes/3600 seconds
Responder-Only (Y/N): N
PFS (Y/N): N
Transform sets={
TS: { esp-aes esp-sha-hmac } ,
}
Because R1 already has an established pair of IPsec SAs with R2, then we can assume that R1 has the
correct configuration and R3 has the incorrect configuration. Specifically, R1 is using ESP-AES 192
and R3 is using ESP-AES 128 for encryption in the supported transform set. Let’s fix this on R3.
R3(config)#crypto ipsec transformset TS esp-aes 192 esp-sha-hmac
Soon after we see the EIGRP adjacency come up on R3 to R1 meaning that the crypto must be
working.
R3#
*Dec 7 05:19:49.705: %DUAL-5-NBRCHANGE: EIGRP-IPv4 123: Neighbor 192.168.123.1
(Tunnel100) is up: new adjacency
We can also confirm this if we look at the crypto sessions on R1.
R1#show cryp session
Crypto session current status
Interface: Tunnel100
Session status: UP-ACTIVE
Peer: 10.0.13.3 port 500
IKEv1 SA: local 10.1.1.1/500 remote 10.0.13.3/500 Active
IPSEC FLOW: permit 47 host 10.1.1.1 host 10.0.13.3
Active SAs: 2, origin: crypto map
Interface: Tunnel100
Session status: UP-ACTIVE
Peer: 10.0.12.2 port 500
IKEv1 SA: local 10.1.1.1/500 remote 10.0.12.2/500 Active
IPSEC FLOW: permit 47 host 10.1.1.1 host 10.0.12.2
Active SAs: 4, origin: crypto map
Now let’s run the traceroute in the verification section of the TS scenario (note that it says to run this
command several times to produce the same output).
R2#trace 192.168.33.3 source loop192 num
Type escape sequence to abort.
Tracing the route to 192.168.33.3
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.123.1 5 msec 5 msec 5 msec
2 192.168.123.3 10 msec * 6 msec
R2#trace 192.168.33.3 source loop192 num
Type escape sequence to abort.
Tracing the route to 192.168.33.3
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.123.1 5 msec 2 msec 5 msec
2 192.168.123.3 5 msec * 6 msec
R2#trace 192.168.33.3 source loop192 num
Type escape sequence to abort.
Tracing the route to 192.168.33.3
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.123.1 5 msec 5 msec 5 msec
2 192.168.123.3 6 msec * 6 msec
About 10 seconds passed between each command run, yet the output never changed. Let’s see what
the route looks like on R2 to R3’s loopback.
R2#show ip route eigrp
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override
Gateway of last resort is not set
192.168.33.0/32 is subnetted, 1 subnets
D 192.168.33.3 [90/102400640] via 192.168.123.1, 00:03:32, Tunnel100
There is no NHO for this route, so maybe there is a problem with the redirect/shortcut configuration.
Let’s check this on each of the routers.
R1#show run int tun100 | inc redirect
no ip redirects
ip nhrp redirect
R2#show run int tun100 | inc shortcut
ip nhrp shortcut
R3#show run int tun100 | inc shortcut
ip nhrp shortcut
That looks fine too so let’s assume that the DMVPN phase 3 configuration is working correctly. The
next thing that should happen is R2 should try and establish an IKE SA with R3 so that they can
communicate directly. Let’s check R2’s crypto sessions.
R2#show crypto session
Crypto session current status
Interface: Tunnel100
Session status: DOWN-NEGOTIATING
Peer: 10.0.13.3 port 500
IKEv1 SA: local 10.0.12.2/500 remote 10.0.13.3/500 Inactive
IPSEC FLOW: permit 47 host 10.0.12.2 host 10.0.13.3
Active SAs: 0, origin: crypto map
Interface: Tunnel100
Session status: UP-ACTIVE
Peer: 10.1.1.1 port 500
IKEv1 SA: local 10.0.12.2/500 remote 10.1.1.1/500 Active
IPSEC FLOW: permit 47 host 10.0.12.2 host 10.1.1.1
Active SAs: 4, origin: crypto map
R2’s doesn’t seem to be able to establish an IKE SA with R3. Let’s see if the following command will
help us isolate the problem.
R2#show crypto isakmp sa detail
Codes: C - IKE configuration mode, D - Dead Peer Detection
K - Keepalives, N - NAT-traversal
T - cTCP encapsulation, X - IKE Extended Authentication
psk - Preshared key, rsig - RSA signature
renc - RSA encryption
IPv4 Crypto ISAKMP SA
C-id Local Remote I-VRF Status Encr Hash Auth DH Lifetime
Cap.
1001 10.0.12.2 10.1.1.1 ACTIVE aes sha256 psk 14 23:35:42
Engine-id:Conn-id = SW:1
0 10.0.12.2 10.0.13.3 ACTIVE 0 0
Engine-id:Conn-id = ???
(deleted)
IPv6 Crypto ISAKMP SA
From this output it looks like R2 and R3 are not able to agree upon the parameters that make up the
IKE SA. Let’s check the ISAKMP policy on both routers.
R2#show crypto isakmp policy
Global IKE policy
Protection suite of priority 1
encryption algorithm: AES - Advanced Encryption Standard (256 bit keys).
hash algorithm: Secure Hash Standard 2 (256 bit)
authentication method: Pre-Shared Key
Diffie-Hellman group: #14 (2048 bit)
lifetime: 86400 seconds, no volume limit
R3#show crypto isakmp policy
Global IKE policy
Protection suite of priority 1
encryption algorithm: AES - Advanced Encryption Standard (256 bit keys).
hash algorithm: Secure Hash Standard 2 (256 bit)
authentication method: Pre-Shared Key
Diffie-Hellman group: #14 (2048 bit)
lifetime: 86400 seconds, no volume limit
Looking at this carefully, all of the values seem to be correct so there is no reason why they should
have trouble negotiating the IKE settings. Let’s check out the ISAKMP peer configurations. Let’s
check the ISAKMP key settings.
R2#show crypto isakmp key
Keyring Hostname/Address Preshared Key
default 10.1.1.1 luke123
R3#show crypto isakmp key
Keyring Hostname/Address Preshared Key
default 10.1.1.1 luke123
10.0.12.2 luke123
There we go! R3 has a key configured for R1 and R2 but R2 only has a key configured for R1. Let’s
confirm this in the configuration on R2.
R2#show run | sec isakmp key|keyring
crypto isakmp key luke123 address 10.1.1.1
If there is no key configured for R3, then the IKE SA will never get established. Let’s fix this on R2.
R2(config)#crypto isakmp key luke123 address 10.0.13.3
Let’s clear the crypto sessions on R2 and then confirm that the IPsec SAs get established between R2
and R3.
R2#clear crypto session
R2#show crypto session
Crypto session current status
Interface: Tunnel100
Session status: UP-ACTIVE
Peer: 10.0.13.3 port 500
IKEv1 SA: local 10.0.12.2/500 remote 10.0.13.3/500 Active
IKEv1 SA: local 10.0.12.2/500 remote 10.0.13.3/500 Active
IPSEC FLOW: permit 47 host 10.0.12.2 host 10.0.13.3
Active SAs: 4, origin: crypto map
Interface: Tunnel100
Session status: UP-ACTIVE
Peer: 10.1.1.1 port 500
IKEv1 SA: local 10.0.12.2/500 remote 10.1.1.1/500 Active
IPSEC FLOW: permit 47 host 10.0.12.2 host 10.1.1.1
Active SAs: 2, origin: crypto map
Now IPsec appears to be working between the spokes, so let’s test the traceroute from the original
scenario.
R2#trace 192.168.33.3 source loop192 num
Type escape sequence to abort.
Tracing the route to 192.168.33.3
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.123.3 5 msec * 5 msec
Now everything looks good!
Scenario #4 Solutions
DMVPN and NAT can make for an interesting TS scenario, especially when the hub is behind the NAT
device. When the spoke is behind a NAT device it is not such a big deal because the hub will notice
that the source IP address in NHRP is different from the source IP address in the IP header, and will
infer that NAT is between the two endpoints. There are some things that need to be taken into
consideration with the hub behind NAT, which we will see in the next two scenarios.
First thing to check is to see if any of the spokes have registered with the hub.
R1#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================
Which matches what we expect from the original scenario anyway. Starting from R2, let’s check the
NRHP map to the hub.
R2#show ip nhrp
192.168.123.1/32 via 192.168.123.1
Tunnel100 created 00:07:50, never expire
Type: static, Flags:
NBMA address: 10.0.14.1
Based on the diagram this looks okay, so let’s see if we can ping between the NBMA addresses.
R2#ping 10.0.14.1 source e0/0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.14.1, timeout is 2 seconds:
Packet sent with a source address of 160.0.24.2
.....
Success rate is 0 percent (0/5)
Do we have a route to 10.0.14.1?
R2#show ip route 10.0.14.1
% Network not in table
R2#show ip route 0.0.0.0
% Network not in table
This makes sense because R1’s address is a private IP address, and is hidden behind R4 which is
performing NAT. Let’s see how the NAT is setup on R4.
R4#show ip nat statistics
Total active translations: 1 (0 static, 1 dynamic; 0 extended)
Peak translations: 1, occurred 00:11:00 ago
Outside interfaces:
Ethernet0/0, Ethernet0/1, Loopback0
Inside interfaces:
Ethernet0/2
Hits: 0 Misses: 0
CEF Translated packets: 0, CEF Punted packets: 0
Expired translations: 0
Dynamic mappings:
-- Inside Source
Total doors: 0
Appl doors: 0
Normal doors: 0
Queued Packets: 0
So there must be something setup to translate inside local addresses out E0/2 to an inside global
address (or addresses) out either E0/0, E0/1, or Loopback0. We can see if static NAT is configured
using the following command.
R4#show ip nat translations verbose
Pro Inside global Inside local Outside local Outside global
--- 160.4.4.4 10.0.14.1 --- ---
create 00:13:15, use 00:13:15 timeout:0, timing-out,
flags:
static, use_count: 0, entry-id: 2, lc_entries: 0
Based on this, we can see that R4 has a static translation between R1’s E0/0 interface to the address
160.4.4.4. Let’s confirm that this is working by pinging R2 from R1 and seeing what address R2 sees
in the ICMP packet.
R2#debug ip icmp
ICMP packet debugging is on
R1#ping 160.0.24.2 repeat 1
Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 160.0.24.2, timeout is 2 seconds:
!
Success rate is 100 percent (1/1), round-trip min/avg/max = 1/1/1 ms
Then looking back at the debug output on R2.
R2#
*Dec 7 06:06:54.007: ICMP: echo reply sent, src 160.0.24.2, dst 160.4.4.4,
topology BASE, dscp 0 topoid 0
So NAT appears to be working okay. Therefore we can assume that if we change the NHRP map on
R2 to point to R4’s loopback 0 (160.4.4.4) as the NBMA address, R1 will actually receive the NRHP
packets. Let’s try this out.
R2(config)#int tun100
R2(config-if)#no ip nhrp map multicast 10.0.14.1
R2(config-if)#no ip nhrp map 192.168.123.1 10.0.14.1
R2(config-if)#ip nhrp map multicast 160.4.4.4
R2(config-if)#ip nhrp map 192.168.123.1 160.4.4.4
We can actually see that R1 has received the traffic because we see the following message printed to
the console.
R1#
*Dec 7 06:10:05.220: %DUAL-5-NBRCHANGE: EIGRP-IPv4 123: Neighbor 192.168.123.2
(Tunnel100) is up: new adjacency
*Dec 7 06:11:24.812: %DUAL-5-NBRCHANGE: EIGRP-IPv4 123: Neighbor 192.168.123.2
(Tunnel100) is down: retry limit exceeded
We also notice that the adjacency is reset soon after. Let’s check the NHRP cache on R1.
R1#show ip nhrp
192.168.123.2/32
Tunnel100 created 00:01:37, expire 00:01:27
Type: incomplete, Flags: negative
Cache hits: 7
However we have an incomplete NHRP mapping to R2. Let’s confirm that R2 is actually registering
itself to the hub.
R2#show ip nhrp nhs
However we can see that R2 has no NHS configured! Let’s now fix this by setting the NHS to
192.168.123.1.
R2(config)#int tun100
R2(config-if)#ip nhrp nhs 192.168.123.3
As soon as we configure this we see the following adjacency come up on R2.
*Dec 7 06:19:42.790: %DUAL-5-NBRCHANGE: EIGRP-IPv4 123: Neighbor 192.168.123.1
(Tunnel100) is up: new adjacency
Now that it appears R2 is working correctly, let’s look at the current status of the DMVPN from R1’s
perspective.
R1#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================
Interface: Tunnel100, IPv4 NHRP Details
Type:Hub, NHRP Peers:1,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 160.0.24.2 192.168.123.2 UP 00:01:44 D
R3 is still not registered to R1. Let’s see if there is an NHRP mapping issue on R3 that is preventing
this registration from occurring.
R3#show ip nhrp
192.168.123.1/32 via 192.168.123.1
Tunnel100 created 00:31:47, never expire
Type: static, Flags: used
NBMA address: 10.0.14.1
In a similar fashion to R2, R3 is configured with an NHRP mapping to the inside local address of R1,
which is not reachable by the spokes. Let’s change this on R3.
R3(config)#int tun100
R3(config-if)#no ip nhrp map multicast 10.0.14.1
R3(config-if)#no ip nhrp map 192.168.123.1 10.0.14.1
R3(config-if)#ip nhrp map multicast 160.4.4.4
R3(config-if)#ip nhrp map 192.168.123.1 160.4.4.4
While we are at it, we may as well see if R3 has no NHS configured like R2.
R3#show ip nhrp nhs
Legend: E=Expecting replies, R=Responding, W=Waiting
Tunnel100:
192.168.123.1 E priority = 0 cluster = 0
R3 is configured correctly here. Soon afterwards, we should see the EIGRP adjacency come up to R1.
R3#
*Dec 7 06:25:03.463: %DUAL-5-NBRCHANGE: EIGRP-IPv4 123: Neighbor 192.168.123.1
(Tunnel100) is up: new adjacency
Let’s now test out the verification from the original scenario to make sure we have solved the TS
scenario correctly.
R2#ping 192.168.33.3 source loop192
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.33.3, timeout is 2 seconds:
Packet sent with a source address of 192.168.22.2
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Which we have.
Scenario #5 Solutions
Similar to scenario #4, DMVPN with NAT can be tricky to troubleshoot especially when the hub is
behind a NAT device. In this scenario, IPsec is configured too to make the scenario even harder. This
should test your understanding of how crypto is handled with NAT and different IPsec modes. The
last debug command from the original scenario actually provides the biggest clue to one of the
problems.
Let’s first see if any of the spoke routers have registered with the hub.
R1#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
Nothing has registered to R1. Let’s check the NHRP mappings and the NHS setting on R2 and R3.
R2#show ip nhrp
192.168.123.1/32 via 192.168.123.1
Tunnel100 created 00:07:37, never expire
Type: static, Flags: used
NBMA address: 160.4.4.4
R2#show ip nhrp nhs
Legend: E=Expecting replies, R=Responding, W=Waiting
Tunnel100:
192.168.123.1 E priority = 0 cluster = 0
R3#show ip nhrp
192.168.123.1/32 via 192.168.123.1
Tunnel100 created 00:06:39, never expire
Type: static, Flags: used
NBMA address: 160.4.4.4
R3#show ip nhrp nhs
Legend: E=Expecting replies, R=Responding, W=Waiting
Tunnel100:
192.168.123.1 E priority = 0 cluster = 0
This all looks fine. If we take a step back let’s see if IKE or IPsec SAs are being established on R1.
R1#show crypto session
Is there a problem with NAT on R4?
R4#show ip nat translations verbose
Pro Inside global Inside local Outside local Outside global
esp 160.4.4.4:0 10.0.14.1:0 --- ---
create 00:08:48, use 00:08:48 timeout:0, timing-out,
flags:
extended, extendable, static, use_count: 0, entry-id: 1, lc_entries: 0
udp 160.4.4.4:500 10.0.14.1:500 --- ---
create 00:08:48, use 00:08:48 timeout:0, timing-out,
flags:
extended, extendable, static, use_count: 0, entry-id: 2, lc_entries: 0
The NAT entries look fine as they allow ISAKMP (UDP/500) and ESP traffic to pass between R1 and
the outside world. Let’s run a debug for ISAKMP on R2 to see if we can isolate any problems with the
ISAKMP negotiation.
R2#debug crypto isakmp
Crypto ISAKMP debugging is on
R2#ping 192.168.123.1 repeat 1
Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 192.168.123.1, timeout is 2 seconds:
*Dec 7 07:36:29.419: ISAKMP:(0): SA request profile is (NULL)
*Dec 7 07:36:29.419: ISAKMP: Created a peer struct for 160.4.4.4, peer port 500
*Dec 7 07:36:29.419: ISAKMP: New peer created peer = 0xB2C81C38 peer_handle =
0x80000014
*Dec 7 07:36:29.419: ISAKMP: Locking peer struct 0xB2C81C38, refcount 1 for
isakmp_initiator
*Dec 7 07:36:29.419: ISAKMP: local port 500, remote port 500
*Dec 7 07:36:29.419: ISAKMP: set new node 0 to QM_IDLE
*Dec 7 07:36:29.419: ISAKMP:(0):insert sa successfully sa = B29F17D8
*Dec 7 07:36:29.419: ISAKMP:(0):Can not start Aggressive mode, trying Main mode.
*Dec 7 07:36:29.419: ISAKMP:(0):No pre-shared key with 160.4.4.4!
*Dec 7 07:36:29.419: ISAKMP:(0): No Cert or pre-shared address key.
*Dec 7 07:36:29.419: ISAKMP:(0): construct_initial_message: Can not start Main
mode
*Dec 7 07:36:29.419: ISAKMP: Unlocking peer struct 0xB2C81C38 for
isadb_unlock_peer_delete_sa(), count 0
*Dec 7 07:36:29.419: ISAKMP: Deleting peer node by peer_reap for 160.4.4.4:
B2C81C38
*Dec 7 07:36:29.419: ISAKMP:(0):purging SA., sa=B29F17D8, delme=B29F17D8
*Dec 7 07:36:29.419: ISAKMP:(0):purging node -1924640252
*Dec 7 07:36:29.419: ISAKMP: Error while processing SA request: Failed to
initialize SA
*Dec 7 07:36:29.419: ISAKMP: Error while processing KMI message 0, error 2..
Success rate is 0 percent (0/1)
Excellent! This debug has told us that there is no pre-shared key with 160.4.4.4 (the inside global
address of the hub). Let’s check this setting on R2.
R2#show crypto isakmp key
Keyring Hostname/Address Preshared Key
default 10.0.14.1 luke123
Okay so we have configured the key to the inside local address rather than the inside global. We will
fix this as follows.
R2(config)#no crypto isakmp key luke123 address 10.0.14.1
R2(config)#crypto isakmp key luke123 address 0.0.0.0 0.0.0.0
It is probable that we are going to need spoke to spoke communications to occur, so we simply use
the key “luke123” for all ISAKMP peers. Let’s now look on R1 to see if the crypto session has come
up.
R1#show crypto session
Crypto session current status
Interface: Ethernet0/0
Session status: DOWN-NEGOTIATING
Peer: 160.0.24.2 port 500
IKEv1 SA: local 10.0.14.1/500 remote 160.0.24.2/500 Inactive
It’s still not coming up so what else could be wrong with IPsec through NAT? This is where the hint
from the original scenario comes into play. Let’s run a debug on R1 to show what is wrong.
R1#debug crypto isakmp
Crypto ISAKMP debugging is on
R1#
*Dec 7 07:43:36.496: ISAKMP (0): received packet from 160.0.24.2 dport 500 sport
500 Global (N) NEW SA
*Dec 7 07:43:36.496: ISAKMP: Created a peer struct for 160.0.24.2, peer port 500
*Dec 7 07:43:36.496: ISAKMP: New peer created peer = 0xB2C2FE70 peer_handle =
0x80000003
<snip>
*Dec 7 07:43:36.527: ISAKMP (1002): NAT found, both nodes inside NAT
*Dec 7 07:43:36.527: ISAKMP:received payload type 20
*Dec 7 07:43:36.527: ISAKMP (1002): My hash no match - this node inside NAT
R1#
There is a lot of output when running the above command, but if you know have ever setup IPsec
through NAT you will have seen this problem before. If it is still not clicking than the debug output
on R2 should highlight the exact problem for you.
R2#debug cry isak
Crypto ISAKMP debugging is on
R2#
*Dec 7 07:48:13.469: ISAKMP:(1004):purging node 1258689276
*Dec 7 07:48:13.469: ISAKMP:(1004):purging node -1579434794
*Dec 7 07:48:14.030: ISAKMP:(0): SA request profile is (NULL)
*Dec 7 07:48:14.030: ISAKMP: Created a peer struct for 160.4.4.4, peer port 500
*Dec 7 07:48:14.030: ISAKMP: New peer created peer = 0xB2C81C38 peer_handle =
0x8000001E
<snip>
*Dec 7 07:48:14.064: ISAKMP (1005): His hash no match - this node outside NAT
*Dec 7 07:48:14.064: ISAKMP:received payload type 20
*Dec 7 07:48:14.064: ISAKMP (1005): His hash no match - this node outside NAT
<snip>
*Dec 7 07:48:14.064: ISAKMP:(1005): sending packet to 160.4.4.4 my_port 4500
peer_port 4500 (I) MM_KEY_EXCH
*Dec 7 07:48:14.064: ISAKMP:(1005):Sending an IKE IPv4 Packet.
*Dec 7 07:48:14.064: ISAKMP:(1005):Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE
*Dec 7 07:48:14.064: ISAKMP:(1005):Old State = IKE_I_MM4 New State = IKE_I_MM5
R2#
*Dec 7 07:48:23.470: ISAKMP:(1004):purging SA., sa=B1053CE8, delme=B1053CE8
*Dec 7 07:48:24.055: ISAKMP (1005): received packet from 160.4.4.4 dport 500 sport
500 Global (I) MM_KEY_EXCH
*Dec 7 07:48:24.055: ISAKMP:(1005): phase 1 packet is a duplicate of a previous
packet.
*Dec 7 07:48:24.055: ISAKMP:(1005): retransmitting due to retransmit phase 1
R2#
*Dec 7 07:48:24.560: ISAKMP:(1005): retransmitting phase 1 MM_KEY_EXCH...
*Dec 7 07:48:24.560: ISAKMP (1005): incrementing error counter on sa, attempt 1 of
5: retransmit phase 1
*Dec 7 07:48:24.560: ISAKMP:(1005): retransmitting phase 1 MM_KEY_EXCH
*Dec 7 07:48:24.560: ISAKMP:(1005): sending packet to 160.4.4.4 my_port 4500
peer_port 4500 (I) MM_KEY_EXCH
*Dec 7 07:48:24.560: ISAKMP:(1005):Sending an IKE IPv4 Packet.
When IPsec is used with NAT, NAT-T is used. NAT-T, discussed in RFC 3947 and RFC 3948, provides a
method to detect a NAT device between the IPsec endpoints and also a way to get around PAT (as
IPsec is not associated with a transport layer port) by encapsulating the ESP tunnel mode payload
with UDP or using transport mode.
So given this new piece of information, we should check whether R4 has a NAT rule for UDP port
4500 (NAT-T).
R4#show ip nat translations
Pro Inside global Inside local Outside local Outside global
esp 160.4.4.4:0 10.0.14.1:0 --- ---
udp 160.4.4.4:500 10.0.14.1:500 --- ---
Which it does not. Let’s add a rule to allow this translation to occur.
R4(config)#ip nat inside source static udp 10.0.14.1 4500 interface Loopback0 4500
Let’s now see if this has fixed the crypto problem between R1 and R2.
R1#show crypto session
Crypto session current status
Interface: Ethernet0/0
Session status: UP-IDLE
Peer: 160.0.24.2 port 4500
IKEv1 SA: local 10.0.14.1/4500 remote 160.0.24.2/4500 Active
Well at least we have the IKE SA now so that’s looking better already. Let’s now see if there is a
problem with the IPsec transform set between R1 and R2.
R1#show crypto ipsec transform-set
Transform set default: { esp-aes esp-sha-hmac }
will negotiate = { Tunnel, },
Transform set TS: { esp-192-aes esp-sha-hmac }
will negotiate = { Tunnel, },
R2#show crypto ipsec transform-set
Transform set default: { esp-aes esp-sha-hmac }
will negotiate = { Tunnel, },
Transform set TS: { esp-192-aes esp-sha-hmac }
will negotiate = { Tunnel, },
The problem has already showed itself above, but we will confirm it with a debug on either R1 or R2.
R1#debug crypto ipsec
Crypto IPSEC debugging is on
R1#
*Dec 7 08:04:39.389: IPSEC(key_engine): got a queue event with 1 KMI message(s)
R1#
*Dec 7 08:04:58.613: IPSEC(key_engine): got a queue event with 1 KMI message(s)
*Dec 7 08:04:58.619: IPSEC(validate_proposal_request): proposal part #1
*Dec 7 08:04:58.619: IPSEC(validate_proposal_request): proposal part #1,
(key eng. msg.) INBOUND local= 10.0.14.1:0, remote= 160.0.24.2:0,
local_proxy= 160.4.4.4/255.255.255.255/47/0,
remote_proxy= 160.0.24.2/255.255.255.255/47/0,
protocol= ESP, transform= NONE (Tunnel-UDP),
lifedur= 0s and 0kb,
spi= 0x0(0), conn_id= 0, keysize= 192, flags= 0x0
*Dec 7 08:04:58.619: map_db_find_best did not find matching map
*Dec 7 08:04:58.619: IPSEC(ipsec_process_proposal): proxy identities not supported
If you read this link you will see that one of the pre-requisites for NAT-T to work in a DMVPN is that
transport mode must be used. Let’s change the transport mode on both R1 and R2 to see if this fixes
our problem.
R1(config)#crypto ipsec transform-set TS esp-aes 192 esp-sha-hmac
R1(cfg-crypto-trans)#mode transport
R2(config)#crypto ipsec transform-set TS esp-aes 192 esp-sha-hmac
R2(cfg-crypto-trans)#mode transport
Shortly after you should see the EIGRP adjacency come up between R1 and R2.
R2#
*Dec 7 08:07:16.456: %DUAL-5-NBRCHANGE: EIGRP-IPv4 123: Neighbor 192.168.123.1
(Tunnel100) is up: new adjacency
R2#
*Dec 7 08:08:36.049: %DUAL-5-NBRCHANGE: EIGRP-IPv4 123: Neighbor 192.168.123.1
(Tunnel100) is down: retry limit exceeded
However the adjacency is flapping, which as we discovered in an earlier task, usually means that
there is a problem with multicast mapping. Let’s check it on R1.
R2#show ip nhrp multicast
I/F NBMA address
Tunnel100 10.0.14.1 Flags: static
We can see the problem straight away. R2 has an NHRP map for multicast to the inside local address
for R1, not the inside global address. Let’s fix this as follows.
R2(config)#int tun100
R2(config-if)#no ip nhrp map multicast 10.0.14.1
R2(config-if)#ip nhrp map multicast 160.4.4.4
Let’s see if this has fixed the flapping EIGRP issue.
R2#show ip eigrp nei
EIGRP-IPv4 VR(CCIE) Address-Family Neighbors for AS(123)
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
0 192.168.123.1 Tu100 10 00:01:05 2 1470 0 3
The queue count here is zero which is good. Let’s see if R1 is receiving routes now.
R1#show ip route eigrp
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override
Gateway of last resort is not set
D 192.168.22.0/24 [90/76800640] via 192.168.123.2, 00:02:05, Tunnel100
This is looking much better! Now we must figure out what is wrong with R3. From earlier we
confirmed that the NHRP mappings and NHS settings looked okay. Let’s see if we also have tunnel
mode ESP configured rather than transport mode.
R3#show crypto ipsec transform-set
Transform set default: { esp-aes esp-sha-hmac }
will negotiate = { Tunnel, },
Transform set TS: { esp-192-aes esp-sha-hmac }
will negotiate = { Tunnel, },
Let’s change this to transport mode now.
R3(config)#crypto ipsec transform-set TS esp-aes 192 esp-sha-hmac
R3(cfg-crypto-trans)#mode transport
But nothing seems to happen. Has any crypto session state been created on R3?
R3#show crypto session
If the DMVPN tunnel is up, then the endpoints should definitely be trying to negotiate an IKE SA.
Let’s check to see if the tunnel is up.
R3#show ip int Tun100 | inc line
Tunnel100 is up, line protocol is up
Has an IPsec crypto profile been created for use on the tunnel?
R3#show crypto ipsec profile
IPSEC profile IPSEC_PROF
Security association lifetime: 4608000 kilobytes/3600 seconds
Responder-Only (Y/N): N
PFS (Y/N): N
Transform sets={
TS: { esp-192-aes esp-sha-hmac } ,
}
IPSEC profile default
Security association lifetime: 4608000 kilobytes/3600 seconds
Responder-Only (Y/N): N
PFS (Y/N): N
Transform sets={
default: { esp-aes esp-sha-hmac } ,
}
This looks good, but has it been enabled on the tunnel?
R3#sh run int tun100
Building configuration...
Current configuration : 329 bytes
!
interface Tunnel100
description DMVPN
ip address 192.168.123.3 255.255.255.0
no ip redirects
ip nhrp authentication 1uke123
ip nhrp map multicast 160.4.4.4
ip nhrp map 192.168.123.1 160.4.4.4
ip nhrp network-id 123
ip nhrp nhs 192.168.123.1
ip nhrp redirect
tunnel source Ethernet0/0
tunnel mode gre multipoint
end
And there we see the problem: the crypto profile has not been configured under the tunnel
interface. Let’s fix this.
R3(config)#int tun100
R3(config-if)#tunnel protection ipsec profile IPSEC_PROF
Straight away we see that this has fixed the problem because the ISAKMP process starts and the
EIGRP adjacency comes up. This is evident from the following messages.
*Dec 7 08:30:51.366: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
R3#
*Dec 7 08:30:51.479: %DUAL-5-NBRCHANGE: EIGRP-IPv4 123: Neighbor 192.168.123.1
(Tunnel100) is up: new adjacency
R3#
Now as a final test, let’s see if R2 and R3 can ping between their loopbacks.
R2#ping 192.168.33.3 source loop192
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.33.3, timeout is 2 seconds:
Packet sent with a source address of 192.168.22.2
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 6/6/7 ms