I have been working with BGP for some time, and have used it often in my deployments. While I feel comfortable utilizing BGP, I haven’t really sat down to consider BGP convergence until recently. In online learning and books, it is often said that the internet never “converges” and thus it has been put in the back of my mind.
When studying RFC3345, I re-read some books and listened to a video series of Russ White on the PacketPushers youtube channel and finally understood how BGP truly converges. I thought it was important to share what I had learned.
What is convergence
A network is said to be converged when all routers participating in a routing domain are said to have received all network information. This is the state where all routers have a consistent view of the network and when there are no routing updates being sent. In terms of BGP, this would mean that all routers have received all BGP Update packets and the BGP RIB is not processing any new information.
This is the opposite of network “churn” which is state where the routing table is constantly changing due to routing updates. The BGP table on the internet is never in a state of convergence as there are constant updates due to routers going offline, interfaces changes, carrier fiber outages ect.
In any non-internet network, it is important to ensure that the routing protocols are converged. As people, including myself, rely on BGP more and more then understanding how quickly BGP can converge becomes important.
Minimum Route Advertisement Interval
RFC4271, section 9.2.1.1, defines the the minimum amount of time that can pass between BGP Updates. This is known as MinRouteAdvertisementInterval or MRAI. This RFC recommends the default timer be 5 seconds for an iBGP peering and 30 seconds for an eBGP peering. This was intended as a way to minimize the number of BGP messages that would be sent in a network, but can have undesirable affects.
Lets look at the following topology.
In this topology, there are 5 different BGP ASNs all representing separate companies. AS500 is advertising the subnet of 7.7.7.0/24 to its peer R1 in AS100. When R7 sends the advertisement to R1, R1 will in turn, advertise that prefix to its peers in AS200 and AS300.
Based on RFC4271, AS100 will wait the default 30 seconds before advertising the routes to AS200 and AS300. AS200 and AS300 will also be subject to the MRAI and wait for 30 seconds before advertising the route to their peers in AS400.
In total, the convergence time was 90 seconds from when AS500 advertised the route to when AS400 was able to see it. 90 seconds isn’t horrendous when advertising a prefix, but it is a problem if the route was being withdrawn.
If the prefix of 7.7.7.0/24 were to stopped being advertise by AS500, that could take as long as 90 seconds for that to propagate to AS400. All traffic that was sent to 7.7.7.0/24 in that timeframe would be lost before AS400 would realize that network was unreachable.
BGP Path Hunting
Another problem that can arise with MRAI timers is something called BGP Path Hunting. THis is the state the a BGP speaker is in when it is looking for alternate routes to a prefix that has stopped being advertise by a particular peer. The BGP speaker will look in its table, and install the next best route. If a route that is being advertise to multiple peers stops being advertised, each BGP speaker will seek alternate routes even though those alternate routes will not be able to reach the destination.
Lets change the topology so that AS100 would have multiple ways to reach to reach AS500.
In this topology, there are three different paths that AS100 can take to reach the prefix of 7.7.7.0/24.
- The Path AS400 –> AS500
- The path AS400 –> AS200 –> AS500
- The path AS200 –> AS300 –> AS100 –> AS500
If default BGP values are used, AS400 will prefer the shortest AS Path which would be the path directly to AS500. There is a situation where BGP will start “path hunting” for a route that has been withdrawn from BGP. Lets say R7 in AS500 were to stop advertising the 7.7.7.0/24 prefix to all of its peers. BGP MRAI would kick in and it could take 30 seconds before AS500 sends a BGP Update message telling its peers that the 7.7.7.0/24 prefix is unreachable. R7 in AS500 would send out three BGP Updates, one directly to R4 in AS500, one to R2 in AS200, and one to R1 in AS100.
The diagram is showing the BGP Update messages that contain the withdraw message for the 7.7.7.0/24 prefix. R4 would remove the BGP route directly to R7, and look at its BGP table to install the next best route. The next best route it would select is the route via AS200 and it would update its route table.
This is all well and dandy until 30 seconds later. R2 would notify AS400 that the prefix 7.7.7.0/24 is unreachable, and R1 in AS100 would send the BGP Update to R3 in AS300.
Once again, R4 in AS400 would have to recompute the best path to reach 7.7.7.0/24 prefix. It would look in its BGP table and realize the best and only route it has to reach 7.7.7.0/24 is via AS300. It would install that route from its BGP table into it’s RIB table.
We all know what is coming next. 30 seconds later R3 in AS300 would send the BGP update message to AS400. R4 would receive that update and finally remove the 7.7.7.0/24 prefix from its route table. It would take 90 seconds from the time the 7.7.7.0/24 stopped being advertise to R4 removing the route from its BGP table. This updating of the route table to other paths is known as BGP Path hunting.
This path hunting and MRAI may not be an issue on the wider Internet, but it is an issue with certain topologies. For instance, it is common to change the MRAI timers to 0 from 30 seconds in Spine/Leaf architectures. This comes with the offset of more frequent BGP updates, but modern network equipment is exponential faster than equipment from the 90s when MRAI was first described.
