The Three Tier network design has been recommended by vendors and used by customers for many years. It is taught in the CCNA course and used in the following certification tracks as a model for how to design a large campus network. Cisco is still showing a 3 Tier or Collapsed Core as part of their Cisco Validated Designs.
It is a design that I am familiar and have used many times when talking to customers. The layout is easy, the problems are mostly understood, and this design did help grow many enterprises. There are problems with the design that have been widely talked about; issues with the reliance on Spanning-Tree and issues with VLAN and broadcast domains are the two mostly widely spoken about.
3 Tier Design
The Three-Tier design is a way to setup the network infrastructure like building blocks. There are, as the name suggests, three tiers in the design: Core, Distribution, and Access.
Core
The core layer is at the top of most diagrams. These are typically beefy switches that logically present as a single device through the use of Virtual Chassis. This allows for MC-LAG’d interfaces between devices and minimizes the use of Spanning-Tree within an environment. Most external services like Internet, SDWAN, DMZ, are connected directly to this layer as seen here.
Distribution
The distribution layer is the layer that aggregates the access switches to the core layer. The core layer switches are often expensive, and each port on the Core layer is important. As a way to make sure that there are enough ports and bandwidth available, often there will be a distribution layer that takes x number of uplinks from the access switches, and uses only y number of ports to connect to the core layer in a way that x > y.
The distribution layer is often where the IP gateways live for the hosts that hang off of the access layer and will run some form of First Hop Redundancy so that clients are able to connect to services in the event a distribution switch were to go offline.
Access
The access layer is the simplest layer in terms of what it does. The purpose of the access layer is connect end devices, whether those are servers, hosts, or phones to the rest of the network. There are often multiple access switches within a closet depending on how large the office or floor is.
Bandwidth Problem
As mentioned, there is an issue with the 3 Tier design as campuses or datacenters grow. Often people will talk about the scale of VLANs or how broadcast domains become large. Another issue that I don’t see talked about often is the issue of bandwidth. There are often many more access switches than there are distribution switches, and more distribution switches than there are core switches. This can create a serious oversubscription ratio that often isn’t immediately seen.
Lets see this example again.
In this topology, there are 8 access switches. Lets assume that each switch is a 48 1g port switch with 2 10g uplinks. This would total 48gbps down and 10gbps up due to spanning tree disabling the secondary uplink interface. This is about a 4:1 ratio on an individual access switch basis. In here, there are 8 switches times the 48 1g ports which comes out to 384gbps of available bandwidth downstream on the access switches and only 30gbps up.
There is the same problem to the distribution layer. If each uplink is using link aggregation and they are 40gbps links, that is only 160gbps up from the distribution to the core while there is the 384gbps that could potentially be required by the devices off of the access switches.
The bandwidth issue only grows the farther away the devices are from each other. Devices that are connected to each other through a single access switch will have full 1g bandwidth available to each other, but if servers are attempting to reach each other from different access switches, then there is a greater chance that their traffic could be dropped or delayed due to the oversubscription.
Server placement then becomes a concern. If servers are connected to the access layer and do large backups or large file shares then the closer those servers are physically, the better performance they will have.
This issue prompted this wonderful article who faced this issue when building a datacenter. While it is from 2010, the problem with Three Tier network is the same.
Alternatives
This bandwidth constraint, along with the broadcast and vlan creep, is what prompted the design of Spine/Leaf fabrics which provide consistent latency and bandwidth for east to west traffic. While there may still be an oversubscription issue, it should not be at the scale of a 3 Tier network.
Now this doesn’t mean that no one should use a the Three Tier network, but it does mean that what application or services one is rendering their customer needs to be taken into account. If the network is mostly local server to local server, then maybe spine/leaf designs should be considered.
