Discontiguous BGP Autonomous Systems
Let’s imagine a scenario where your campus network has its own AS and isconnected to the service provider(s) using eBGP. Business grows, your company hires more and more people, and suddenly your office space is not enough. Your managers decide to open a new site nearby, however you could only connect the two using dark fiber, which costs too much, and the management is not willing to pay for it. An additional AS also costs too much so you have to find a workaround to use the same ASN on the new site too, without having your own interconnects. Important note: the solution described in this post is only possible if the two sites are connected to the same provider AS.
Layer 1 topology
The following lab was created to solve the stated problem:
Layer 3 topology
Assigned subnets and addresses:
Loopback interfaces are highlighted with red background.
Initial setup
Site 1 runs OSPF internally and advertises its subnets to PE1 using BGP. The route map is used to accept the default route only (originated from Provider 2) to simplify the routing tables.
CE1#sh run | se router
router ospf 1
router-id 5.5.5.5
passive-interface Ethernet1/0
network 5.5.5.5 0.0.0.0 area 0
network 10.0.4.0 0.0.0.0 area 0
network 10.1.0.0 0.0.255.255 area 0
default-information originate
router bgp 65001
bgp router-id 5.5.5.5
bgp log-neighbor-changes
network 5.5.5.5 mask 255.255.255.255
network 10.1.0.0 mask 255.255.255.0
network 10.1.1.0 mask 255.255.255.0
network 10.1.2.0 mask 255.255.255.0
neighbor 10.0.4.1 remote-as 65000
neighbor 10.0.4.1 soft-reconfiguration inbound
neighbor 10.0.4.1 route-map RM-Default_only in
CE1#sh route-map RM-Default_only
route-map RM-Default_only, permit, sequence 10
Match clauses:
ip address prefix-lists: PL-Default_only
Set clauses:
Policy routing matches: 0 packets, 0 bytes
route-map RM-Default_only, deny, sequence 20
Match clauses:
Set clauses:
Policy routing matches: 0 packets, 0 bytes
CE1#sh ip prefix-list PL-Default_only
ip prefix-list PL-Default_only: 1 entries
seq 5 permit 0.0.0.0/0
Connecting the 2nd site
First, let’s create the eBGP peering on CE2 with the provider using the same settings:
router bgp 65001
bgp router-id 7.7.7.7
network 7.7.7.7 mask 255.255.255.255
network 10.2.1.0 mask 255.255.255.0
neighbor 10.0.5.1 remote-as 65000
neighbor 10.0.5.1 soft-reconfiguration inbound
neighbor 10.0.5.1 route-map RM-Default_only in
end
The next step is adding static routes on the CE routers to each other’s loopbacks. The reason for this is that the routers are not taking the default route into account when looking for their configured iBGP peer’s IP address.
CE1(config)#ip route 7.7.7.7 255.255.255.255 10.0.4.1
CE2(config)#ip route 5.5.5.5 255.255.255.255 10.0.5.1
Then let’s create the iBGP peering between the CE routers:
CE1
router bgp 65001
neighbor 7.7.7.7 remote-as 65001
neighbor 7.7.7.7 update-source Loopback0
neighbor 7.7.7.7 soft-reconfiguration inbound
end
CE2
router bgp 65001
neighbor 5.5.5.5 remote-as 65001
neighbor 5.5.5.5 update-source Loopback0
neighbor 5.5.5.5 soft-reconfiguration inbound
end
Troubleshooting the flapping neighbors
At this point the peering is up; but then you see something strange. The following messages appear on the console:
*Aug 30 13:36:52.699: %BGP-5-NBR_RESET: Neighbor 7.7.7.7 reset (BGP Notification sent)
*Aug 30 13:36:52.699: %BGP-5-ADJCHANGE: neighbor 7.7.7.7 Down BGP Notification sent
*Aug 30 13:36:52.699: %BGP_SESSION-5-ADJCHANGE: neighbor 7.7.7.7 IPv4 Unicast topology base removed from session BGP Notification sent
The iBGP neighbors are flapping constantly. Why is it happening? Let’s check the routing table and the received advertisements on PE1:
PE1#sh ip route 7.7.7.7
Routing entry for 7.7.7.7/32
Known via "bgp 65000", distance 20, metric 0
Tag 65001, type external
Last update from 10.0.4.2 00:01:44 ago
Routing Descriptor Blocks:
* 10.0.4.2, from 10.0.4.2, 00:01:44 ago
Route metric is 0, traffic share count is 1
AS Hops 1
Route tag 65001
MPLS label: none
PE1#sh ip bgp neighbors 10.0.4.2 received-routes
BGP table version is 166, local router ID is 3.3.3.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 5.5.5.5/32 10.0.4.2 0 0 65001 i
*> 7.7.7.7/32 10.0.4.2 0 65001 i
*> 10.1.0.0/24 10.0.4.2 0 0 65001 i
* 10.1.1.0/24 10.0.4.2 20 0 65001 i
*> 10.1.2.0/24 10.0.4.2 0 0 65001 i
*> 10.2.1.0/24 10.0.4.2 0 65001 i
Total number of prefixes 6
If you examine the output, you’ll see that CE1 start advertising the loopback of CE2 to PE1. PE1 then learns it, and sets CE1 as the next hop to 7.7.7.7. When PE1 sends the packet for 7.7.7.7 to CE1, it sends it back to PE1 because of the created static route entry. Yes, we created a routing loop. Here’s how it looks when capturing the traffic:
Since CE1 can’t reach CE2, the BGP keepalive messages are not received. After the BGP hold timer times out, the entries are cleared, and the CEs can reach each other again until the BGP peering is formed again. I put together a small animation that describes this behavior:
Solving the routing loop
We have to find out how we can tackle this issue. Let’s start by looking at the advertised BGP routes on CE1:
CE1#sh ip bgp neighbors 10.0.4.1 advertised-routes
BGP table version is 122, local router ID is 5.5.5.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 5.5.5.5/32 0.0.0.0 0 32768 i
r>i 7.7.7.7/32 7.7.7.7 0 100 0 i
*> 10.1.0.0/24 0.0.0.0 0 32768 i
*> 10.1.1.0/24 10.1.0.2 20 32768 i
*> 10.1.2.0/24 0.0.0.0 0 32768 i
*>i 10.2.1.0/24 7.7.7.7 0 100 0 i
Total number of prefixes 6
There are two networks CE1 shouldn’t advertise: 7.7.7.7/32 and 10.2.1.0/24. It’s doesn’t take too much to spot the common in these two, the next hop is 7.7.7.7 for both.
Solving the issue is easy from here: we just have to filter the other site’s next hop address(es) from our BGP advertisements using these commands:
CE1
conf t
ip prefix-list PL-IBGP_neighbors permit 7.7.7.7/32
route-map RM-EBGP_to_provider deny 10
match ip next-hop prefix-list PL-IBGP_neighbors
route-map RM-EBGP_to_provider permit 20
exit
router bgp 65001
neighbor 10.0.4.1 route-map RM-EBGP_to_provider out
end
CE2
conf t
ip prefix-list PL-IBGP_neighbors permit 5.5.5.5/32
ip prefix-list PL-IBGP_neighbors permit 10.1.0.2/32
route-map RM-EBGP_to_provider deny 10
match ip next-hop prefix-list PL-IBGP_neighbors
route-map RM-EBGP_to_provider permit 20
exit
router bgp 65001
neighbor 10.0.5.1 route-map RM-EBGP_to_provider out
end
State after solving the issue
After applying the configuration above, the flapping stops and only the local subnets are advertised:
CE1#sh ip bgp neighbors 10.0.4.1 advertised-routes
BGP table version is 166, local router ID is 5.5.5.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 5.5.5.5/32 0.0.0.0 0 32768 i
*> 10.1.0.0/24 0.0.0.0 0 32768 i
*> 10.1.1.0/24 10.1.0.2 20 32768 i
*> 10.1.2.0/24 0.0.0.0 0 32768 i
Total number of prefixes 4
CE2#sh ip bgp neighbors 10.0.5.1 advertised-routes
BGP table version is 320, local router ID is 7.7.7.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 7.7.7.7/32 0.0.0.0 0 32768 i
*> 10.2.1.0/24 0.0.0.0 0 32768 i
Total number of prefixes 2
The AS path is the same for all subnets from a third AS:
P2#sh ip bgp | i 10.1.1.0|10.2.1.0
*> 10.1.1.0/24 172.16.1.2 0 65000 65001 i
*> 10.2.1.0/24 172.16.1.2 0 65000 65001 i