While working with a customer last week on a multi-site, production virtualization design using the 700-series Vblocks, a question came up over the scalability of the Cisco MDS switches. The customer has MDS 9222i switches in use now, and each Vblock included a pair of MDS 9506 switches for FC connectivity. Specifically, the customer is using multiple 8Gbps FC connections to connect the blades to block storage, in addition to multiple 10Gbps connections for NAS and CIFS. They asked if there were any architectural caveats that needed to be taken into account, and based on my experience as a customer of the MDS switches, I knew that a discussion about oversubscription was going to be needed.
First a little recap if you aren’t that familiar with the Cisco storage networking product set. Cisco has two separate lines of fabric switches, the “Director” line which includes the MDS 9500-series of modular switches and the “Fabric and Blade Switch” line which includes the 9100- and 9200-series devices as well as the blade switches for the HP c-Class BladeSystem. Each device which is capable of using an 8Gbps SFP (with one exception which we’ll get to later), whether it is a line card used in the MDS 9500-series products or a stand-alone fabric switch, has some level of oversubscription in place either locally on the card or on the backplane that connects the cards in the case of the Directors. Knowing where and how this oversubscription takes place is a key part of architecting a design, so here are the basics.
The first thing you need to know is how the individual ports are “grouped” on the switch. I assume (although I haven’t been able to find any documentation to confirm it) that each port group corresponds to a physical hardware stack, maybe an individual fabric services ASIC, which is what actually creates the oversubscription matrix. For example, on the MDS 9222i, which is a 4Gbps FC switch, there are 6 ports per group. On the 4/44-port 8-Gbps Host-Optimized Fibre Channel switching module for the MDS 9500-series there are 12 ports per group.
The next thing to know is how much bandwidth is available per port group. For both of the afore-mentioned devices, the total is 12.8Gbps. In fact, with the exceptions of the MDS 9124 and the MDS 9148, every 8Gbps-enabled MDS device has a total of 12.8Gbps of bandwidth available.
By using the total number of 8Gb ports, the number of ports in a group and the amount of bandwidth per group, you can figure out how far you are going to be able to scale the switch at full line rate, and at which point the over-subscription is going to kick in. If you’d like a handy chart to reference all of the values we just discussed, you can find it here:
On the chart, you’ll notice the exceptions I referenced earlier: The MDS 9124/9134 switches (4Gb FC only) and the MDS 9148 (8Gb FC) all require no oversubscription to run full bandwidth through each port on the device. Incidentally, the MDS 9148 is the FC switch included with all 300-series Vblocks in part for this specific reason.
Now you know what your switches/blades are capable of, so what? Is this really a big deal? To me, yes and no. Yes, it’s a design consideration that needs to be accounted for, or at least communicated as part of an architecture since there’s always the possibility that a customer could ramp up FC usage to the point where the oversubscription comes into play. No, because the chances of saturating 100Gbps of backplane with FC traffic is remote at best. Add in that Cisco has included an entire set of NX-OS/SAN-OS commands specifically to be able to control where and how oversubscription happens on a switch/blade and you have a lot of tools at your disposal to prevent an issue. Worst case, Cisco offers both 4Gb and 8Gb FC switches that require no oversubscription at all, so if your design is looking to stress FC bandwidth to the max, that’s the best direction to take. Even before I came to VCE I’ve been a big fan of the MDS line of products, and hopefully this information will make it that much easier for you to utilize them in your own environment.
Thanks to Jason Nash (@nashj) and Tyler Britten (@VMTyler) for keeping me straight with this one. Tyler in particular confirmed that my dusty, hazy memory of bandwidth oversubscription was accurate, and he and Jason both helped me dig up the Cisco documentation to confirm it. I wish I was one of those guys who could keep all of this information in his head, but I’m definitely no Andy Sholomon. It’s good to have friends out there who can help me when I need it. Thank goodness (again) for Twitter!