MPLS Layer 3 VPNS, a Primer
It’s time for another primer. This week we take a look at MPLS Layer 3 VPNs. Enjoy!
Multiprotocol Label Switching (MPLS) Layer 3 VPNs
MPLS Layer 3 (L3) VPNs are a highly-scalable peer-to-peer solution that provides enterprises with a VPN across a Service Providers network. The Service Provider will act as the intermediary to carry the Customer’s routes between the Customer’s sites. Service providers accomplish this by utilizing MPLS and BGP technologies to create a virtual network for each customer across their core. MPLS Layer 3 VPNs offer the following benefits:
- No static provisioning required
- Adding new sites doesn’t necessarily require reconfiguration of other sites.
- Service Provider keeps separate routing tables per customer
- Allows flexibility in customer addressing, that is, customers RFC 1918 addresses can overlap with each other.
Components
- Virtual Routing/Forwarding (VRF) Tables
- Provides separation of customer routing information
- IGP
- Used between Customer and Service Provider
- Can be static routing, BGP, or any IGP
- MP-BGP
- Used at the Service Provider’s edge network to carry customers routes
- MPLS
- Used in the Service Provider’s core to forward packets towards BGP next hops
MP-BGP (VPNv4)
The first order of business is establishing a MP-BGP VPNv4 adjacency between your PE routers. The VPNv4 address family is used to carry VPN routes between PE routers, that is, it is used to carry VRF routing information between PE routers.
Route Distinguishers
One important feature of MP-BGP VPNv4 is the use of a route distinguisher (RD). An RD is a unique number assigned to each VRF created within the Service Provider’s network. RDs are 8 bytes in length and are defined using the format ASN:NN or ip-address:NN.
A route distinguisher is prepended to each route installed in a VRF; doing this allows for customer address space to remain unique in the Service Provider’s network. To illustrate, let’s say that both, Customer A and Customer B are using 10.0.0.0/24 in their network; what would happen when both customers advertise the 10.0.0.0/24 network to the Service Provider? How would the Service Provider know which particular customer should get traffic destined for 10.0.0.0/24?
This is the exact problem that RDs solve. With the RD prepended on the prefixes that are installed in the VRF, it now allows each of the prefixes to be unique within the Service Providers network. For example, Customer A’s prefix 10.0.0.0/24 with a RD of 1:1 becomes 1:1:10.0.0.0/24 and Customer B’s prefix with a RD of 2:2 becomes 2:2:10.0.0.0/24.
MPBGP VPNv4 Labels
In addition to the RD, each PE router will assign labels for all customer prefixes they receive from directly connected Customer Edge routers. MP-BGP is the protocol that will assign this label and use it for forwarding traffic to the CE router. The MP-BGP assigned label is in addition to the MPLS labels. This implies that packets traversing the Service Provider’s network will carry two labels: one assigned by MP-BGP and the other assigned by MPLS.
Here’s a quick summarization on the labels used throughout MPLS VPNS:
- Outer label (transport label) is used for transport in the SP core; this label is assigned by MPLS.
- Inner label is used for transport to customer prefixes from PE router; this label is assigned by MP-BGP VPNv4
VRF
Next up is VRFs. When defining VRFs there is one additional parameter besides a RD to define, that is, Route Target (RT). The RT is an 8 byte field and the format is exactly the same as the RD, ANS:NN or ip-address:NN. It is possible to use the same number as your RD. The RT will be imported into MP-BGP with the routes as an extended community. Using RTs as extended communities allows for VRFs on different PEs to select which particular routes they will install.
When you specify the route-target import command, you are informing the router which routes will be installed into that VRF, by selecting the routes with the extended community of the RT you configured in the import command. Likewise, when you specify the route-target export command, you are informing the router which extended community should be exported with the routes as they are redistributed into MP-BGP.
Basic Configuration Steps
Before we begin, it is assumed you have MPLS and BGP already configured in your network. Now, these are the steps you need to take in order to configure MPLS L 3 VPN:
- Configure VRF
- Configure routing between CE-PE (BGP or IGP)
- Configure MP-BGP VPNv4 adjececncy between PE routers (Note: You do not need to enable the IPv4-Unicast Address family between PE routers, unless you need to have the Full internet route updates)
- Configure MP-BGP VPNv4 to CE redistribution
Refer to the following topology for the configuration:
Configuring VRFs
First we need to create our customer VRFs on each PE router. The VRFs are defined by a name; the name is local to the router. We will also specify our RD and RTs. I am choosing 1:1 as our RD and RT for simplicity sake.
R2(config)#ip vrf ?
WORD VPN Routing/Forwarding instance name
R2(config)#ip vrf CUSTOMER_A
R2(config-vrf)#?
IP VPN Routing/Forwarding instance configuration commands:
bgp Commands pertaining to BGP
context Associate SNMP context with this vrf
default Set a command to its defaults
description VRF specific description
exit Exit from VRF configuration mode
export VRF export
import VRF import
maximum Set a limit
mdt Backbone Multicast Distribution Tree
no Negate a command or set its defaults
rd Specify Route Distinguisher
route-target Specify Target VPN Extended Communities
vpn Configure VPN ID as specified in rfc2685
R2(config-vrf)#rd 1:1
R2(config-vrf)#route-target ?
ASN:nn or IP-address:nn Target VPN Extended Community
both Both import and export Target-VPN community
export Export Target-VPN community
import Import Target-VPN community
R2(config-vrf)#route-target both 1:1
The last thing we need to do is assign this VRF to f0/0, our interface that connects to our customer. Note that when you assign an interface to a VRF, the router will remove the configured IP address; so I always like to do a sho run int before I add the interface do the VRF.
R2(config-vrf)#int f0/0
R2(config-if)#do sho run int f0/0
!
interface FastEthernet0/0
description TO_R1
ip address 10.1.12.2 255.255.255.252
duplex auto
speed auto
end
R2(config-if)#ip vrf forwarding CUSTOMER_A
% Interface FastEthernet0/0 IP address 10.1.12.2 removed due to enabling VRF CUSTOMER_A
R2(config-if)#ip add 10.1.12.2 255.255.255.252
On R7 we will also create a VRF called CUSTOMER_A and here we will use the same RT (1:1) to import/export and a different RD of 1:2. We are using the same RT which will make this a full-mesh, that is, all routes from each of Customer A’s site will be installed into all the other sites. The following are some other configurations you can have depending how you define the VRF:
- Full Mesh
- Import and Export same everywhere
- Hub and Spoke
- Spokes import only hub’s routes
- Central Services
- Multiple VPNs can import routes from a central site or from a central server
- Management VPNs
- Management loopback on CE routers can be exported into a special management VPN
Configure Routing between PE and CE
I am going to use OSPF for my CE-PE routing protocol. Typically you would want to run BGP, as matter of fact that is what I recommend. There are many reasons why, but that is not in the scope of this writing.
To configure OSPF between the PE and CE routers, you have to enable the process, as well as enable the OSPF process on the interfaces. For our configuration we will configure R7-R8 and R2-R1 as OSPF Area 0 (MPLS L3 VPNs have the ability to extended the OSPF Area between sites)
Realize now that the customer will install the networks they want to distribute across the VPN into OSPF which will then be advertised to the PE routers and from there across PE to PE.
On the CE routers the configuration is very simple, we will add the CE-PE interface into OSPF and also the loopback (used to simulate networks behind the CE routers).
R8(config)#router ospf 1
R8(config-router)#network 10.1.78.0 0.0.0.3 area 0
R8(config-router)#network 8.8.8.8 0.0.0.0 area 1
The PE routers configuration will be slightly different. Remember that the interfaces that connect the PEs to the CEs are installed in a VRF, so the OSPF configuration must take that into consideration:
R7(config)#router ospf 1 vrf CUSTOMER_A
R7(config-router)#network 10.1.78.0 0.0.0.3 area 0
At this point, the PE and CE routers will have an OSPF adjanceny formed. The VRF on the PE router should also contain the routes that are being advertised to the PE router from the CE router.
R2#sho ip route vrf CUSTOMER_A
Routing Table: CUSTOMER_A
< — lines omitted — >
1.0.0.0/32 is subnetted, 1 subnets
O IA 1.1.1.1 [110/11] via 10.1.12.1, 00:09:06, FastEthernet0/0
10.0.0.0/30 is subnetted, 1 subnets
C 10.1.12.0 is directly connected, FastEthernet0/0
MP-BGP VPNv4
Now we need to form the VPNv4 adjacency between the PE routers (R2 and R7) in order for them to exchange the routes between Customer A’s sites.
This will be configured under the BGP process. You must define the neighbors under the vpnv4 address-family. It is very similar to how you would configure normal ipv4 unicast adjacencies, just under a different address family name. It is also important to configure the routers to use extended BGP communities. This is needed for the RT import and exports.
R2(config)#router bgp 1
R2(config-router)#neigh 7.7.7.7 remote-as 1
R2(config-router)#neigh 7.7.7.7 update-source lo0
R2(config-router)#address-family vpnv4
R2(config-router-af)#nei 7.7.7.7 activate
R2(config-router-af)#nei 7.7.7.7 send-community extended
R7(config)#router bgp 1
R7(config-router)#neigh 2.2.2.2 remote-as 1
R7(config-router)#neigh 2.2.2.2 update-source lo0
R7(config-router)#address-family vpnv4
R7(config-router-af)#neigh 2.2.2.2 activate
R7(config-router-af)#nei 2.2.2.2 send-community extended
Next we will verify the adjacency. We can use show ip bgp vpnv4 all su — which is analogous to sho ip bgp su.
R7#sho ip bgp vpnv4 all su
BGP router identifier 7.7.7.7, local AS number 1
BGP table version is 1, main routing table version 1
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
2.2.2.2 4 1 4 4 1 0 0 00:00:14 0
You’ll notice there isn’t any prefixes being advertised (yet). This is because we have not redistributed the routes from Customer A’s VRF into the vpnv4 MP-BGP process yet.
Redistribution into BGP
To redistribute the OSPF learned routes, we need to redistribute OSPF into the ipv4 vrf [VRF] address-family. This is different than the VPNv4 family; remember the VPNv4 family is to define the PE-to-PE communication to carry the customer routes between PEs. The IPv4 VRF address family is used to bring the OSPF routes into MP-BGP. Then through the VPNv4 peering, the routes will be sent to all other PE routers.
R7(config)#router bgp 1
R7(config-router)#address-family ipv4 vrf CUSTOMER_A
R7(config-router-af)#redis ospf 1 vrf CUSTOMER_A
Next, to verify the routes are being redistributed properly, we need to check the IPv4 vrf BGP RIB. You would think the command would be sho ip bgp ipv4 vrf [VRF] or something similar right? Well it’s not. You need to run sho ip bgp vpnv4 command to see which routes are being installed.
R7#sho ip bgp vpnv4 all
BGP table version is 9, local router ID is 7.7.7.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf CUSTOMER_A)
*>i1.1.1.1/32 2.2.2.2 11 100 0 ?
*> 8.8.8.8/32 10.1.78.2 11 32768 ?
*>i10.1.12.0/30 2.2.2.2 0 100 0 ?
*> 10.1.78.0/30 0.0.0.0 0 32768 ?
Notice it lists the tables by the RD. If we have more than one VRF we would see all of them here. You can also run the command for one specific RD or VRF.
From the output we see the routes from R8 and also R2. So at this point the PE routers know about all the site to site prefixes, if we look at the routing table for the VRF from R7’s perspective we should have all the routes installed:
R7#sho ip route vrf CUSTOMER_A
Routing Table: CUSTOMER_A
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route
Gateway of last resort is not set
1.0.0.0/32 is subnetted, 1 subnets
B 1.1.1.1 [200/11] via 2.2.2.2, 00:09:29
8.0.0.0/32 is subnetted, 1 subnets
O IA 8.8.8.8 [110/11] via 10.1.78.2, 00:41:04, FastEthernet0/0
10.0.0.0/30 is subnetted, 2 subnets
B 10.1.12.0 [200/0] via 2.2.2.2, 00:09:29
C 10.1.78.0 is directly connected, FastEthernet0/0
There is one final step to getting full connectivity between sites, we need to redistribute the MP-BGP routes into OSPF so the CE routers. Remember the CE-PE communication only happens within the confines of the OSPF process.
R7(config)#router ospf 1 vrf CUSTOMER_A
R7(config-router)#redistribute bgp 1 subnets
R2(config)#router ospf 1 vrf CUSTOMER_A
R2(config-router)#redist bgp 1 su
At first glance, the configuration might appear to install ALL the MP-BGP routes into Customer A’s OSPF process, but remember all throughout this configuration everything has been done explicitly for the VRF. So only the routes that are in the VRF RIB via MP-BGP will be redistributed into OSPF.
Now we should have full connectivity, let’s try pinging R1 from R8 :
R8#ping 1.1.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 84/92/112 ms
That’s it! There is a lot of moving parts in an MPLS L3 VPN, but all in all it is pretty straight forward once you get comfortable with the configuration.
