VyOS (Quagga, Vyatta) Route Selection
Cisco’s OSPF path selection, that is different from RFC 2328 and 3101, has been discussed in many articles (here’s a nice one from INE). I was wondering how VyOS, running Quagga, handles the same prefixes received with different LSAs. I put together two nasty topologies in GNS3 (you should never ever do the same in a real world environment 😄) that will be used to observe LSA processing; one for a VyOS, one for a Cisco only environment.
VyOS
Topology, addressing
This is the topology that was used for the test:
IP addresses were assigned using these formulas (H = hostname sequence number, i.e. 4 for IOU4):
- Loopbacks: H.H.H.H/32
- Interfaces: 192.168.{H1}{H2}.{H1|2} (e.g. 192.168.25.2 for eth1 of VyOS2, 192.168.13.3 for e0/2 of IOU3)
Routing
Protocols, areas
This is where it gets messy. 😄
- IOU6 advertises it’s loopbacks using RIP to IOU3.
- IOU3 redistributes R6’s advertisement to OSPF NSSA no-summary area 1 as E1 with a cost of 50. It also advertises its own loopback.
- The OSPF cost of the link between R3 and VyOS1 is 100.
- VyOS1 has its eth2 interface in area 1; eth0 and eth1 are in area 0.
- All other routers and interfaces belong to area, except for the link between IOU4 and IOU5; they’re exchange routing information for 6.0.0.0/8 using BGP.
- IOU5 redistributes 6.0.0.0/8 to OSPF with a cost of 1.
A picture to make it clear:
Advertisements of 6.0.0.0/8
- IOU6 advertises the route using RIP to IOU3.
- IOU3 redistributes it to OSPF and sends the prefix 6.0.0.0/8 to VyOS1 as LSA 7.
- VyOS propagates the advertisement to IOU4 and VyOS2 as LSA5. VyOS2 also advertises it to IOU5.
- IOU4 advertises the prefix using BGP to IOU5.
- IOU5 installs the route to its routing table from IOU4 because BGP’s administrative distance is lower than OSPF’s. Then it redistributes the prefix to OSPF and advertises to VyOS2 as E1 with a cost of 1.
- VyOS2 installs the route coming from IOU5 because both advertisements are LSA5 but IOU5’s cost is lower than VyOS1’s.
- VyOS2 advertises the prefix to VyOS1 with LSA5.
- Now VyOS1 has the advertisement from two sources: IOU3 as LSA7 with a cost of 151 and VyOS2 as LSA5 with a cost of 21. VyOS1 will install the route from IOU3 because it prefers LSA7 over LSA5 even if the cost of the latter is lower.
- There’s one catch though. IOU4 learns VyOS1 as next hop but the cost of the route is only 31 instead of 161. The reason for this is it prefers the LSA originated from IOU5 and initially would send the packet there, but VyOS1 “overwrites” this path.
This figure visualizes what’s been described above; the routers are marked with the source where they install the route from:
CLI outputs
IOU3 learns 6.0.0.0/8 from RIP:
IOU3#sh ip route | i [^6]6.0.0.0
R 6.0.0.0/8 [120/1] via 192.168.36.6, 00:00:05, Ethernet0/0
VyOS1 gets the prefix from two sources but installs the one coming from IOU3:
vyos@vyos:~$ sh ip ospf database
OSPF Router with ID (1.1.1.1)
[...]
NSSA-external Link States (Area 0.0.0.1 [NSSA])
Link ID ADV Router Age Seq# CkSum Route
6.0.0.0 3.3.3.3 1448 0x80000004 0xe4f2 E1 6.0.0.0/8 [0x0]
AS External Link States
Link ID ADV Router Age Seq# CkSum Route
6.0.0.0 1.1.1.1 1447 0x80000004 0x4cd9 E1 6.0.0.0/8 [0x0]
6.0.0.0 5.5.5.5 1858 0x80000003 0xfa36 E1 6.0.0.0/8 [0xfde8]
vyos@vyos:~$ sh ip route | match [^6]6.0.0.0
O>* 6.0.0.0/8 [110/151] via 192.168.13.3, eth2, 02:03:05
R4 also receives two LSAs, uses IOU5’s because of the lower cost:
IOU4#sh ip ospf database
OSPF Router with ID (4.4.4.4) (Process ID 1)
[...]
Type-5 AS External Link States
Link ID ADV Router Age Seq# Checksum Tag
6.0.0.0 1.1.1.1 1684 0x80000004 0x004CD9 0
6.0.0.0 5.5.5.5 81 0x80000004 0x00F837 65000
IOU4#sh ip route | i [^6]6.0.0.0
O E1 6.0.0.0/8 [110/31] via 192.168.14.1, 01:42:56, Ethernet0/1
IOU4#sh ip route 6.0.0.0
Routing entry for 6.0.0.0/8
Known via "ospf 1", distance 110, metric 31
Tag 65000, type extern 1
Advertised by bgp 65000
Last update from 192.168.14.1 on Ethernet0/1, 00:05:23 ago
Routing Descriptor Blocks:
* 192.168.14.1, from 5.5.5.5, 00:05:23 ago, via Ethernet0/1
Route metric is 31, traffic share count is 1
Route tag 65000
IOU5 has the prefix in its OSPF database but installs the route learnt via BGP:
IOU5#sh ip ospf database
OSPF Router with ID (5.5.5.5) (Process ID 1)
[...]
Type-5 AS External Link States
Link ID ADV Router Age Seq# Checksum Tag
6.0.0.0 1.1.1.1 1873 0x80000004 0x004CD9 0
6.0.0.0 5.5.5.5 266 0x80000004 0x00F837 65000
IOU5#sh ip route | i [^6]6.0.0.0
B 6.0.0.0/8 [20/31] via 192.168.45.4, 01:45:38
VyOS2 receives two LSA5 advertisements, chooses the lower cost:
vyos@vyos:~$ sh ip ospf database
[...]
AS External Link States
Link ID ADV Router Age Seq# CkSum Route
6.0.0.0 1.1.1.1 636 0x80000005 0x4ada E1 6.0.0.0/8 [0x0]
6.0.0.0 5.5.5.5 1053 0x80000004 0xf837 E1 6.0.0.0/8 [0xfde8]
vyos@vyos:~$ sh ip route | match [^6]6.0.0.0
O>* 6.0.0.0/8 [110/11] via 192.168.25.5, eth1, 01:59:02
A traceroute from VyOS1 reaches IOU6:
vyos@vyos1:~$ traceroute 6.6.6.6
traceroute to 6.6.6.6 (6.6.6.6), 30 hops max, 60 byte packets
1 192.168.13.3 (192.168.13.3) 0.264 ms 0.350 ms 0.170 ms
2 192.168.36.6 (192.168.36.6) 0.314 ms 0.210 ms 0.206 ms
Cisco
Topology, addressing, routing
The same as with VyOS except routers 1 and 2 are also Cisco routers.
Advertisements of 6.0.0.0/8
This is where the two become different.
- IOU6 advertises the route using RIP to IOU3.
- IOU3 redistributes it to OSPF and sends the prefix 6.0.0.0/8 to IOU1 as LSA 7.
- VyOS propagates the advertisement to IOU4 and IOU2 as LSA5. IOU2 also advertises it to IOU5.
- IOU4 advertises the prefix using BGP to IOU5.
- IOU5 installs the route to its routing table from IOU4 because BGP’s administrative distance is lower than OSPF’s. Then it redistributes the prefix to OSPF and advertises to IOU2 as E1 with a cost of 1.
- IOU2 installs the route coming from IOU5 because both advertisements are LSA5 but IOU5’s cost is lower than IOU1’s.
- IOU2 advertises the prefix to IOU1 with LSA5.
- Now IOU1 has the advertisement from two sources: IOU3 as LSA7 with a cost of 151 and IOU2 as LSA5 with a cost of 21. IOU1 will install the route from IOU2 because it prefers LSA5 over LSA7 (also, the cost is lower).
- Even the prefix originally came from R3, a routing loop has been created between IOU1-4-5-2.
This figure visualizes what’s been described above; the routers are marked with the source where they install the route from:
CLI outputs
Outputs are almost the same except for IOU1 which, as mentioned before, prefers LSA5 and installs the route pointing to IOU2.
IOU1#sh ip ospf database
[...]
Type-7 AS External Link States (Area 1)
Link ID ADV Router Age Seq# Checksum Tag
6.0.0.0 3.3.3.3 100 0x80000001 0x00EAEF 0
Type-5 AS External Link States
Link ID ADV Router Age Seq# Checksum Tag
6.0.0.0 5.5.5.5 39 0x80000001 0x00FE34 65000
IOU1#sh ip route | i [^6]6.0.0.0
O E1 6.0.0.0/8 [110/21] via 192.168.12.2, 00:01:23, Ethernet0/0
If we start a traceroute, it goes on a loop until TTL expires:
IOU1#trace 6.6.6.6
Type escape sequence to abort.
Tracing the route to 6.6.6.6
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.12.2 0 msec 4 msec 5 msec
2 192.168.25.5 0 msec 1 msec 4 msec
3 192.168.45.4 4 msec 0 msec 4 msec
4 192.168.14.1 4 msec 4 msec 4 msec
5 192.168.12.2 0 msec 1 msec 0 msec
6 192.168.25.5 0 msec 1 msec 0 msec
7 192.168.45.4 0 msec 0 msec 0 msec
8 192.168.14.1 0 msec 0 msec 0 msec
9 192.168.12.2 0 msec 0 msec 0 msec
10 192.168.25.5 0 msec 0 msec 0 msec
11 192.168.45.4 0 msec 1 msec 0 msec
12 192.168.14.1 0 msec 5 msec 0 msec
13 192.168.12.2 5 msec 0 msec 1 msec
14 192.168.25.5 0 msec 1 msec 0 msec
15 192.168.45.4 1 msec 0 msec 1 msec
16 192.168.14.1 0 msec 5 msec 4 msec
17 192.168.12.2 1 msec 0 msec 1 msec
18 192.168.25.5 0 msec 0 msec 0 msec
19 192.168.45.4 0 msec 1 msec 0 msec
20 192.168.14.1 1 msec 1 msec 0 msec
21 192.168.12.2 1 msec 0 msec 1 msec
22 192.168.25.5 0 msec 1 msec 0 msec
23 192.168.45.4 1 msec 0 msec 1 msec
24 192.168.14.1 0 msec 5 msec 1 msec
25 192.168.12.2 0 msec 1 msec 0 msec
26 192.168.25.5 0 msec 1 msec 0 msec
27 192.168.45.4 1 msec 1 msec 0 msec
28 192.168.14.1 1 msec 0 msec 1 msec
29 192.168.12.2 1 msec 0 msec 1 msec
30 192.168.25.5 0 msec 0 msec 1 msec
FAQ
Could you summarize all of this?
VyOS works as described in the RFCs.
Does it mean VyOS is better than IOS?
Not better, not worse, only different. What might be handy in one situation, can be a disadvantage in another one.
Why is it important? No one sets up a topology like that.
I’m glad to hear that. 😄 Though a properly planned network shouldn’t look like this, it might be a step during a migration, like it happened to me a few years ago (on another platform, see NetIron below).
Mentioning NetIron, does it behave the same way as VyOS?
No, at least 5.3.0 didn’t. I’m afraid I can’t recreate the setup in the lab anymore, but I know that NetIron only cared about the cost of a route; whether the advertisement was LSA5 or LSA7, it didn’t care.