How to Deploy Top-of-Rack Switches
As servers shrink to 1U and 2U sizes, network managers are moving switching infrastructure directly into server racks. These top-of-rack switches dramatically reduce cabling, dropping the number of between-rack cables from more than 100 to just a handful.
To configure a successful top-of-rack switching architecture, follow these suggestions.
1. Be consistent.
Customized configurations form the first step on a path of confusion and human error. Wherever possible, use extremely consistent and generic configurations in top-of-rack switches. This reduces time to replace or repair and provides a no-surprises environment that is also tolerant of minor cabling errors.
For example, configure VLAN-tagged ports identically, with any restrictions on allowed VLANs enforced upstream at a distribution layer switch. Use VLAN-tagged ports to configure all virtualized servers to provide maximum flexibility.
If untagged ports are required for dedicated servers, some sort of visual marking, such as colored electrical tape, can help reduce the chance of someone plugging the wrong server into the wrong VLAN. But this type of visual hint is expensive and difficult to maintain. To simplify configuration, omit other switch security and management features unless they're absolutely required at the top-of-rack level.
2. Be economical.
Most servers will have at least three connections: redundant data connections to two different top-of-rack switches and a separate "lights-out" connection for management. Savvy network managers will recycle their older switches for lights-out connections.
There's a strong argument for running all Intelligent Platform Management Interface connections using a separate network because this reduces the dependencies when something goes very wrong with a server or the network and there's a need for IPMI access.
3. Be careful.
Oversubscription can be a problem with top-of-rack switches, especially in a virtualized environment. If 400 virtual servers are all packed into a single rack, those servers can generate an immense amount of traffic.
Top-of-rack switches don't require 10-gigabit-per-second uplinks in every data center, but network managers planningfor growth should push for at least two 10Gbps connections out of every rack, and more if traffic justifies. Keep a close watch on inter-rack uplink speeds to be sure the network has capacity to move traffic between server racks without bottlenecks.
In some environments, such as with blade servers, a 10Gbps server connection may be justified. Have at least two different top-of-rack data switch models: one for a rack full of 1Gbps servers and another for a rack of 10Gbps servers. Carefully mixing the two types in one rack sounds like a good way to balance traffic, but it's difficult to track how many ports of what speed are available. Unless the data center is limited to a handful of racks, a simpler configuration that doesn't mix server speeds will be easier to manage.
4. Be reliable.
While good switches are generally reliable, software updates and failures are guaranteed to occur over the life of a network. Designing for reliability by using two top-of-rack switches in every cabinet and backhauling those to different distribution-layer devices is a good way to ensure nonstop networking. In general, all designs should allow for any networking element — switch or router — to be rebooted, upgraded or replaced without causing connectivity failures.
Connect servers to the two switches using link aggregation groups based on IEEE standard protocols, which allow for much faster failure detection and better management of the combined link. Simply plugging two ports in between a server and two switches doesn't add much reliability, and creates more failure scenarios without a controlling protocol for link aggregation.
