Replication Over The Backup WAN, Part 3
“It’s fine to have a backup link, but normally it’s not used and we’re paying a lot for it; could you send our file storage’s replication traffic through it?” says the recurring customer question, immediately followed by the “Just make sure business traffic is preferred in case of a primary link failure.” statement. After describing solutions for branch offices, the final post in this series is about a large campus site.
Scenario 3: Campus site
This setup follows Cisco’s Enterprise Composite Model, only we’re using a collapsed distribution/core:
Diverting traffic for a specific source would be PBR again, so this time we’re sending traffic on the backup link for a given destination. IOU7 has two loopbacks, 7.7.7.7/32 and 77.77.77.77/32; we’re accessing the first one through the backup link. As you can see, the destination is in a different AS which means we’re using BGP for path selection. OSPF is running within the autonomous systems, eBGP peering is formed between IOU3-5 and IOU4-6. The usual IP addressing is used (loopback IPs are H.H.H.H, “physical” interfaces are 192.168.{H1}{H2}.{H1|H2}), except for the two broadcast networks; IP addresses for devices connected to SW2 are in the 192.168.1.0/24 subnet, while SW1’s network is 192.168.2.1/24. Again, QoS is implemented but not discussed in depth here; see Part 1 for details. The only difference is that the destination have to be defined in the ACL, not the source. The question is, how to prefer the backup link for replication traffic in this topology? The answer is:
Setting BGP local preference
BGP Local Preference is used to influence the outbound routing decision of a router running BGP. The local preference has to be set for received routes to influence the outbound routing behavior. A higher local preference is preferred (the default value is 100). Local preference is the strongest attribute in the BGP path selection that is transmitted in the routing updates. There’s one catch though. As I mentioned, local preference values has to be exchanged between the routers. Furthermore, the distribution/core devices have to know which border router they should choose. For this, we’re setting up iBGP sessions between the internal routers. To make sure the primary WAN link is used for normal traffic, we set local-pref to 150 for all routes received by IOU3 from IOU5. IOU4’s received routes have the default value except for 7.7.7.7/32, which is set to 200. This way we forward all external traffic through IOU3 except for the replication, which is going through IOU4.
IOU3
Let’s start with the primary border router. First comes the “usual part”, router-id and the advertised networks:
router bgp 65000
bgp router-id 3.3.3.3
network 1.1.1.1 mask 255.255.255.255
network 2.2.2.2 mask 255.255.255.255
network 3.3.3.3 mask 255.255.255.255
network 4.4.4.4 mask 255.255.255.255
network 192.168.1.0
network 192.168.2.0
Then create the internal neighbors, which will receive the BGP routes along with their local-pref values:
neighbor internal peer-group
neighbor internal remote-as 65000
neighbor internal update-source Loopback0
neighbor internal soft-reconfiguration inbound
neighbor 1.1.1.1 peer-group internal
neighbor 2.2.2.2 peer-group internal
neighbor 4.4.4.4 peer-group internal
Finally, the eBGP connection to IOU5:
neighbor 192.168.35.5 remote-as 65001
neighbor 192.168.35.5 soft-reconfiguration inbound
neighbor 192.168.35.5 route-map Primary in
As you can see, a route-map is defined here; it is used to set the local preference for all incoming routes to 150:
route-map Primary permit 10
set local-preference 150
route-map Primary permit 20
IOU4
The BGP configuration is similar to IOU3’s:
router bgp 65000
bgp router-id 4.4.4.4
network 1.1.1.1 mask 255.255.255.255
network 2.2.2.2 mask 255.255.255.255
network 3.3.3.3 mask 255.255.255.255
network 4.4.4.4 mask 255.255.255.255
network 192.168.1.0
network 192.168.2.0
neighbor internal peer-group
neighbor internal remote-as 65000
neighbor internal update-source Loopback0
neighbor internal soft-reconfiguration inbound
neighbor 1.1.1.1 peer-group internal
neighbor 2.2.2.2 peer-group internal
neighbor 3.3.3.3 peer-group internal
neighbor 192.168.46.6 remote-as 65001
neighbor 192.168.46.6 soft-reconfiguration inbound
neighbor 192.168.46.6 route-map RM_Replication in
The difference is the route-map being used; local preference is only modified for the entries of a given prefix-list (line 10), while other routes are simply accepted (line 20):
route-map RM_Replication permit 10
match ip address prefix-list PL_Replication
set local-preference 200
route-map RM_Replication permit 20
The prefix list for line 10 has a single entry:
ip prefix-list PL_Replication seq 5 permit 7.7.7.7/32
IOU1 & IOU2
These devices' BGP configuration is almost identical, the only difference is the router-id. They’re only receiving the router from the border routers (showing IOU1’s configuration only):
router bgp 65000
bgp router-id 1.1.1.1
bgp log-neighbor-changes
neighbor internal peer-group
neighbor internal remote-as 65000
neighbor internal update-source Loopback0
neighbor internal soft-reconfiguration inbound
neighbor 3.3.3.3 peer-group internal
neighbor 4.4.4.4 peer-group internal
HSRP is configured on the LAN side (interfaces connected to SW2):
IOU1#sh run int e0/0
interface Ethernet0/0
ip address 192.168.1.2 255.255.255.0
standby 1 ip 192.168.1.1
standby 1 priority 150
standby 1 preempt
end
IOU2#sh run int e0/0
interface Ethernet0/0
ip address 192.168.1.3 255.255.255.0
standby 1 ip 192.168.1.1
standby 1 priority 120
standby 1 preempt
end
Tests
Both WAN links are up
Let’s check the routing tables and “show ip bgp” outputs first (note: all the outputs below show the relevant sections only), the we do traceroutes from the connected hosts.
IOU1 & 2
IOU1#sh ip route bgp
[...]
5.0.0.0/32 is subnetted, 1 subnets
B 5.5.5.5 [200/0] via 192.168.35.5, 00:38:56
6.0.0.0/32 is subnetted, 1 subnets
B 6.6.6.6 [200/75] via 192.168.35.5, 00:38:56
7.0.0.0/32 is subnetted, 1 subnets
B 7.7.7.7 [200/65] via 192.168.46.6, 00:38:56
77.0.0.0/32 is subnetted, 1 subnets
B 77.77.77.77 [200/11] via 192.168.35.5, 00:38:56
B 192.168.57.0/24 [200/0] via 192.168.35.5, 00:38:56
B 192.168.67.0/24 [200/74] via 192.168.35.5, 00:38:56
The next hop for all AS-external routes are IOU5, except the designated replication destination (7.7.7.7/32), where the next hop is IOU6. (Note: the links between IOU3-5 and IOU4-6 are included in the OSPF configuration with setting the external interfaces as passive ones. If this is not suitable for you, exclude the external interfaces from the OSPF config and include next-hop-self to the internal peer-group.) The reason behind this routing table can be seen in the “show ip bgp” output:
IOU1#sh ip bgp
BGP table version is 11, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
[...]
*>i 5.5.5.5/32 192.168.35.5 0 150 0 65001 i
*>i 6.6.6.6/32 192.168.35.5 75 150 0 65001 i
*>i 7.7.7.7/32 192.168.46.6 65 200 0 65001 i
*>i 77.77.77.77/32 192.168.35.5 11 150 0 65001 i
[...]
*>i 192.168.57.0 192.168.35.5 0 150 0 65001 i
*>i 192.168.67.0 192.168.35.5 74 150 0 65001 i
The highlighted local-pref value makes the desired route selection possible.
IOU3
All BGP routes except 7.7.7.7/32 in the routing table are from IOU5, because IOU4 is sending its local-pref value for that prefix through the iBGP peering:
IOU3#sh ip route bgp
[...]
5.0.0.0/32 is subnetted, 1 subnets
B 5.5.5.5 [20/0] via 192.168.35.5, 02:56:31
6.0.0.0/32 is subnetted, 1 subnets
B 6.6.6.6 [20/75] via 192.168.35.5, 02:56:31
7.0.0.0/32 is subnetted, 1 subnets
B 7.7.7.7 [200/65] via 192.168.46.6, 02:56:28
77.0.0.0/32 is subnetted, 1 subnets
B 77.77.77.77 [20/11] via 192.168.35.5, 02:56:31
B 192.168.57.0/24 [20/0] via 192.168.35.5, 02:56:31
B 192.168.67.0/24 [20/74] via 192.168.35.5, 02:56:31
IOU3#sh ip bgp
[...]
Network Next Hop Metric LocPrf Weight Path
*> 5.5.5.5/32 192.168.35.5 0 150 0 65001 i
*> 6.6.6.6/32 192.168.35.5 75 150 0 65001 i
*>i 7.7.7.7/32 192.168.46.6 65 200 0 65001 i
* 192.168.35.5 11 150 0 65001 i
*> 77.77.77.77/32 192.168.35.5 11 150 0 65001 i
*> 192.168.57.0 192.168.35.5 0 150 0 65001 i
*> 192.168.67.0 192.168.35.5 74 150 0 65001 i
The highlighted rows show that 7.7.7.7/32 is received from two sources; IOU5 has a lower local-pref value, therefore it’s only installed in the routing table if the link between IOU4 and IOU6 goes down. IOU4 is similar to IOU3; I’m not showing it here, you can download the config files and recreate the setup in your lab to check it.
Traceroutes
Since we’re influencing the path for a remote destination (not the source like in the PBR labs), the traces made on PC10 and PC99 are identical. Traffic to 77.77.77.77/32 goes via the primary link:
PC10> ping 77.77.77.77
84 bytes from 77.77.77.77 icmp_seq=1 ttl=252 time=0.995 ms
84 bytes from 77.77.77.77 icmp_seq=2 ttl=252 time=0.823 ms
84 bytes from 77.77.77.77 icmp_seq=3 ttl=252 time=0.781 ms
84 bytes from 77.77.77.77 icmp_seq=4 ttl=252 time=0.658 ms
84 bytes from 77.77.77.77 icmp_seq=5 ttl=252 time=0.644 ms
PC10> trace 77.77.77.77
trace to 77.77.77.77, 8 hops max, press Ctrl+C to stop
1 192.168.1.2 0.463 ms 0.378 ms 0.329 ms <<< IOU1
2 192.168.2.3 0.549 ms 0.441 ms 0.472 ms <<< IOU3
3 192.168.35.5 0.677 ms 0.547 ms 0.347 ms <<< IOU5
4 *192.168.57.7 0.429 ms (ICMP type:3, code:3, Destination port unreachable) <<< IOU7
Traffic to 7.7.7.7/32 goes via the backup WAN link:
PC10> ping 7.7.7.7
84 bytes from 7.7.7.7 icmp_seq=1 ttl=252 time=9.069 ms
84 bytes from 7.7.7.7 icmp_seq=2 ttl=252 time=10.258 ms
84 bytes from 7.7.7.7 icmp_seq=3 ttl=252 time=9.606 ms
84 bytes from 7.7.7.7 icmp_seq=4 ttl=252 time=8.908 ms
84 bytes from 7.7.7.7 icmp_seq=5 ttl=252 time=9.154 ms
PC10> trace 7.7.7.7
trace to 7.7.7.7, 8 hops max, press Ctrl+C to stop
1 192.168.1.2 0.350 ms 0.228 ms 0.164 ms <<< IOU1
2 192.168.2.4 0.385 ms 0.254 ms 0.264 ms <<< IOU4
3 192.168.46.6 8.710 ms 8.718 ms 8.817 ms <<< IOU6
4 *192.168.67.7 8.904 ms (ICMP type:3, code:3, Destination port unreachable) <<< IOU7
Backup link down
If the link between IOU4 and IOU6 goes down, the route for 7.7.7.7/32 is simply removed from IOU4’s routing table, and all routers in the AS learn it from IOU3:
IOU1#sh ip route bgp
[...]
5.0.0.0/32 is subnetted, 1 subnets
B 5.5.5.5 [200/0] via 192.168.35.5, 03:35:01
6.0.0.0/32 is subnetted, 1 subnets
B 6.6.6.6 [200/75] via 192.168.35.5, 03:35:01
7.0.0.0/32 is subnetted, 1 subnets
B 7.7.7.7 [200/11] via 192.168.35.5, 00:03:44
77.0.0.0/32 is subnetted, 1 subnets
B 77.77.77.77 [200/11] via 192.168.35.5, 03:35:01
B 192.168.57.0/24 [200/0] via 192.168.35.5, 03:35:01
B 192.168.67.0/24 [200/74] via 192.168.35.5, 03:35:01
IOU1#sh ip bgp
[...]
Network Next Hop Metric LocPrf Weight Path
*>i 5.5.5.5/32 192.168.35.5 0 150 0 65001 i
*>i 6.6.6.6/32 192.168.35.5 75 150 0 65001 i
*>i 7.7.7.7/32 192.168.35.5 11 150 0 65001 i
*>i 77.77.77.77/32 192.168.35.5 11 150 0 65001 i
*>i 192.168.57.0 192.168.35.5 0 150 0 65001 i
*>i 192.168.67.0 192.168.35.5 74 150 0 65001 i
Traceroutes go via the primary path, regardless of the destination:
PC10> trace 7.7.7.7
trace to 7.7.7.7, 8 hops max, press Ctrl+C to stop
1 192.168.1.2 0.295 ms 0.240 ms 0.244 ms
2 192.168.2.3 0.365 ms 0.292 ms 0.293 ms
3 192.168.35.5 0.372 ms 0.389 ms 0.368 ms
4 *192.168.57.7 0.466 ms (ICMP type:3, code:3, Destination port unreachable)
PC10> trace 77.77.77.77
trace to 77.77.77.77, 8 hops max, press Ctrl+C to stop
1 192.168.1.2 1.129 ms 0.849 ms 0.665 ms
2 192.168.2.3 0.681 ms 0.640 ms 0.632 ms
3 192.168.35.5 0.907 ms 0.801 ms 0.814 ms
4 *192.168.57.7 1.235 ms (ICMP type:3, code:3, Destination port unreachable)
Primary link down
Similar to the previous failure scenario, the only difference is that the routes are advertised by IOU4 now:
IOU3#sh ip route bgp
[...]
5.0.0.0/32 is subnetted, 1 subnets
B 5.5.5.5 [200/75] via 192.168.46.6, 00:00:16
6.0.0.0/32 is subnetted, 1 subnets
B 6.6.6.6 [200/0] via 192.168.46.6, 00:00:16
7.0.0.0/32 is subnetted, 1 subnets
B 7.7.7.7 [200/65] via 192.168.46.6, 00:00:41
77.0.0.0/32 is subnetted, 1 subnets
B 77.77.77.77 [200/65] via 192.168.46.6, 00:00:16
B 192.168.57.0/24 [200/74] via 192.168.46.6, 00:00:16
B 192.168.67.0/24 [200/0] via 192.168.46.6, 00:00:16
IOU3#sh ip bgp
[...]
Network Next Hop Metric LocPrf Weight Path
*>i 5.5.5.5/32 192.168.46.6 75 100 0 65001 i
*>i 6.6.6.6/32 192.168.46.6 0 100 0 65001 i
*>i 7.7.7.7/32 192.168.46.6 65 200 0 65001 i
*>i 77.77.77.77/32 192.168.46.6 65 100 0 65001 i
*>i 192.168.57.0 192.168.46.6 74 100 0 65001 i
*>i 192.168.67.0 192.168.46.6 0 100 0 65001 i
All prefixes are received with a local preference of 100, except for 7.7.7.7/32 which is still altered by the route-map. Traceroutes go via the backup path:
PC10> trace 7.7.7.7
trace to 7.7.7.7, 8 hops max, press Ctrl+C to stop
1 192.168.1.2 0.361 ms 0.223 ms 0.205 ms
2 192.168.2.4 0.364 ms 0.283 ms 0.317 ms
3 192.168.46.6 8.694 ms 8.628 ms 8.821 ms
4 *192.168.67.7 17.380 ms (ICMP type:3, code:3, Destination port unreachable)
PC10> trace 77.77.77.77
trace to 77.77.77.77, 8 hops max, press Ctrl+C to stop
1 192.168.1.2 0.437 ms 0.213 ms 0.286 ms
2 192.168.2.4 0.482 ms 0.353 ms 0.297 ms
3 192.168.46.6 8.782 ms 8.632 ms 8.766 ms
4 *192.168.67.7 16.009 ms (ICMP type:3, code:3, Destination port unreachable)
FAQ
Is there any other way to divert the replication traffic?
Sure, there are several other ways like AS Path Prepending, Policy Based Routing, or even static routes (though that would be awful); use whichever fits into your topology better.
I want to run BGP only on my edge routers, but not in the core. Is it possible?
Yes. Set up iBGP between IOU3 and IOU4, and configure redistribution to your IGP. Only the one with the better metric will redistribute the route.