Tunnels and the Use of Front Door VRFs

Tunnels and the Use of Front Door VRFs

Cisco’s Validated Design (CVD) for IWAN suggests the use of front door VRFs in an IWAN environment.  Front-door VRFs in a tunneled environment are really quite cool.

In the diagram below what this means is that I have the WAN facing physical interface of the DMVPN tunnel end points in a VRF, but the tunnel over it is not in a VRF but is in the global routing table.

iwan

 

 

 

 

 

 

 

 

 

 

 

 

 

 

But let’s not use the above environment to explain this. Let’s go simpler. Let’s strip away IWAN… and strip away DMVPN. Let’s just go with simplest GRE tunnel possible.

Let’s take Foxtrot13 and Foxtrot14.  I hand you 2 tasks ->

  1. create a GRE tunnel between Foxtrot13’s 10.13.2.1 physical interface and Foxtrot14’s 10.4.14.2 physical interface
  2. over this tunnel have an EIGRP neighbor

tunn11_a

 

 

 

 

 

You go and configure Foxtrot13 and Foxtrot14 as below.  Nothing overly exciting in this simplified GRE configuration.

run_tunnels

 

 

 

 

 

 

 

Tunnel Endpoint Connectivity

Of course…. for this Tunnel to come up the physical endpoints of the tunnel (10.13.2.1 & 10.4.14.2) must be able to reach each other.

Let’s look at the physical layer under the tunnel. As you can see, we are going to need some form of  “underlying” routing to be working so these 2 IP addresses can communicate with each other.   As you can see below, I used OSPF.

tunn11_b

 

 

 

 

 

So does Foxtrot13 know how to get to its Tunnel11 GRE tunnel endpoint IP address 10.4.14.2?  Does Foxtrot14 know how to get to its Tunnel11 GRE tunnel endpoint IP address 10.13.2.1?  From what we can see below, the answer is yes.

  • Foxtrot13 has learned about 10.4.14.2 from Echo2 (10.13.2.2) via OSPF.
  • Foxtrot14 has learned about 10.13.2.1 from Echo4 (10.4.14.1) via OSPF.

ospf_dest_ip

 

 

 

 

 

 

 

Are our tunnel11s up?  As you can see below they are.

tunnels

 

 

 

 

 

 

 

 

 

 

 

 

EIGRP

So I have already put the EIGRP configs on Foxtrot14.

eigrp_prep

 

 

 

 

 

 

Let’s put the EIGRP on together on Foxtrot13.

eigrp_tunnel_down_2

 

 

 

 

 

Huh?.. Wait… WHAT?!  Not only did my EIGRP go up and then go down…. but my tunnel also went down. Okay.. so …

  • Tunnel11 was up/up and solid
  • Configure both routers for EIGRP to neighbor up over tunnel11
  • EIGRP neighbor goes up over Tunnel11
  • Seconds later Tunnel11 goes down/down
  • EIGRP (obviously) then goes down

But look at that other error that was reported — “Tunnel temporarily disabled due to recursive routing.”

For those of you who have been “bit” by this one you already know exactly what is happening.  Let’s step it through to see if I can help rope in the rest of you.

flap

 

 

 

 

 

 

 

 

 

 

1) Here we see 2 things

  • the interface Tunnel11 goes up at timestamp 21:41:30.77
  • less than 2 seconds later (21:41:32.32) we see the EIGRP neighbor come up over the Tunnel

2) Foxtrot13 is now learning the IP address (10.4.14.2) used as its destination IP address for the tunnel via EIGRP over the tunnel.  As you can guess… this is just NOT going to work.

3) Not surprisingly we see that at timestamp 21:41:40.777 (just 8 seconds after the EIGRP neighbor came up) we get that error message telling us that “Tunnel11 temporarily disabled due to recursive routing”.  Followed by Tunnel11 getting torn down… which… of course… tears down the EIGRP neighbor going over it.

4) With the EIGRP neighbor torn down we now see the OSPF route to get to 10.4.14.2 is in the RIB (routing information base).  In a few seconds more… if we hung around… the entire process would repeat all over again.  Flap…. flap… flap.

Front-Door VRFs

Let’s say that —

  • Foxtrot13 is a router of yours at a Branch
  • Echo2 and Echo4 are SP routers
  • Foxtrot14 is a router of yours at your Hub Site
  • Tunnel over the SP is what you are going to use exclusively for your routing

This then makes the physical IPs and routing underneath just a “Pipe” the GRE Tunnel uses.  So let’s treat it as a pipe.  🙂   What do I mean?

  • On our routers (Foxtrot13 and Foxtrot14) let’s silo the WAN physical interfaces towards the service providers off into their own VRF (virtual routing forwarding instance).
  • Leave the tunnel interfaces on our routers in the global (default) routing table.
  • Enter a single command on the tunnel 11 interfaces on Foxtrot13 and Foxtrot14 to let them know the physical underneath piping to get us to the other end of the tunnel is in a VRF.

The below 4 steps are how I converted my lab environment over to a front door VRF.  We don’t need to keep “like for like” but to keep things consistent we will.  We will keep the underlying OSPF.

  1. Create a VRF named “Pipe” on both Foxtrot13 and Foxtrot14
  2. Put the “WAN” physicals on both Foxtrot13 and Foxtrot14 into this VRF
  3. Modify the OSPF in Foxtrot13 and Foxtrot14 to be in VRF Pipe
  4. Modify the Tunnel interface to “stitch” the tunnel to the front door VRF

#1: Create a VRF Named “Pipe”

vrf_pipe

 

 

We configure the above in both Foxtrot13 and Foxtrot14. Done.

 

#2: Configure “WAN” Physicals on Foxtrot13 and Foxtrot14 to be in VRF Pipe

Be careful here.  You can see that when I go to configure gig0/0/3 to be in VRF Pipe… my existing IPv4 address is removed from the interface.  Just an FYI here.

So not only will you need to configure the WAN physicals on both Foxtrot13 and Foxtrot14 to be in VRF Pipe… you will also need to enter the WAN physical IP addresses again.

vrf_gig_config

 

 

#3: Modify the OSPF in Foxtrot13 and Foxtrot14 to be in VRF Pipe

This is probably not going to be as simple for you as it was for me in the lab. Since (as you can see from my diagram) I have nothing to the “left” of Foxtrot13 and nothing to the “right” of Foxtrot14… my OSPF on those routers is ONLY running on the “WAN” interfaces.  So for me I just blew away my previous “router ospf 100″ and created a new one tied to vrf Pipe.

PREVIOUS

router ospf 100
router-id 13.13.13.13

REMOVE

no router ospf 100

NEW

router ospf 100 vrf Pipe
router-id 13.13.13.13

#4:  Modify the Tunnel interface to “stitch” the tunnel to the front door VRF

fdv

 

 

 

 

 

 

As you can see we did NOT move the Tunnel11 interface from the global routing table to the routing table for VRF Pipe.  All we did is “stitch” them together.  Via one command “tunnel vrf Pipe” we “told” the router to look for the underlying “piping” not in the global routing table but in the VRF named “Pipe”

 

Putting It All Together

tunVRF

 

 

 

 

 

 

 

 

 

The WAN physical interfaces and the OSPF neighbors and any routing table built out of the OSPF advertisements will ALL remain “removed” and off in their own little “silo’d” world.

It is the single command “tunnel vrf Pipe” under the tunnel11 interface that makes all the magic happen.

tunnel_final

 

 

 

 

 

 

 

What magic? 🙂

The magic of the tunnel being up, functional, and with an EIGRP neighbor across it despite the fact that the routing instance (global/default) that the tunnel11 interface lives in does NOT know how to get to the tunnel end point.

tunn11_a

 

 

 

 

 

magic

 

 

 

 

 

 

 

 

 

 

And that ladies and gentlemen is the absolute beauty of front-door VRFs!  Protects your network and routing over the tunnel from the “transit” you are using underneath.  I am now a HUGE fan of this for DMVPN environments!

Hope you had great fun again playing in the lab with me.  See ya next time!

 

Addl Note:

Darren O’Conner (author of Day One: MPLS for Enterprise Engineers) let me know that Juniper also has a similar feature for tunneled environments. He mentioned this design also being good for security – see tweet below.

twitter

 

 

 

  • Buck Nomenal

    Great article, Denise! Is it also possible to make the EIGRP network statement more specific so that it won’t encompass the physical interface? For example, on Foxtrot13 use “network 10.1.1.1 0.0.0.0” instead of the classful 10.0.0.0. There doesn’t seem to be any recursion there and you can run EIGRP over the tunnel while maintaining OSPF on the physical/NBMA network. Do you lose additional functionality by taking that approach?

    • Denise “Fish” Fishburne

      You can indeed up the granularity. I would just say that obviously… we as humans come along and make tweaks sometimes without looking at the big picture. 🙂 Front Door VRFs would help limit this exposure.

    • Nicolas MICHEL

      That would be fine but Denise “had” to do that in order to make sure the reader why the front door VRF was a nice idea.

      It is a best practice (self) to make the network statement very specific to the interfaces you want EIGRP to run on ….

  • Jeff West

    Wow, what a detailed write up. Thank you

  • Beyers

    Great write up, thank you.

    • Denise “Fish” Fishburne

      🙂 Glad you liked it.

  • Robert Hillcoat

    Using a more specific EIGRP network statement is a good idea to solve the tunnel recursion…………BUT… imagine 1000 DMVPN spokes, you need a default route for NBMA and perhaps a default route for your internal tunneled traffic back to a DC.