Our review Supermicro MicroBlade chassis came with a 10Gb Ethernet switch installed, the Supermicro MBM-XEM-001. In terms of function, this switch is similar to out SuperBlade switch as it provides both internal connectivity between the nodes as well as external up-links. In our discussions with various software providers, hosting providers and partners over the past year, there seems to be a consensus that the MicroBlade networking is a strong point.
Supermicro MicroBlade MBM-XEM-001 10GbE Networking
We had access to a 10GbE switch module, the MBM-XEM-001. The unit can be used in both the 3U and 6U chassis. There are a total of 56x 10GbE internal ports and 4x 40GbE uplink ports in the switch. The Intel FM6348 that the switch is built around offers up to 480Gbps of port bandwidth. Supermicro now offers a Broadcom BCM56846 (Trident +) based switch, the MBM-XEM-002 which came out after we installed our review system. With the 3U MicroBlade, both of these switches offer ample bandwidth for the chassis. They are also part of what makes the 3U MicroBlade so flexible as they can accommodate not just single system per blade configurations but also multiple systems per blade.
The switch itself is installed in the rear of the chassis above the power supplies. In the 3U MicroBlade, there is room for two high speed switches but our review unit only had one. A second switch can be used for redundancy or extra bandwidth. Since the chassis management module provides management networking, there is no need to install cables other than for the uplink ports.
Here is the massive heatsink that covers the Intel FM6348 and related components that make the switch work.
The rear edge connector that connects to the MicroBlade’s backplane is very dense as it can provide 10Gb networking for up to 56 nodes.
Networking Management
There are a few options for network management on the Supermicro MBM-XEM-001 chassis switch we tested. One option is using a traditional serial console port which is available on the switch. The second option is a standard CLI interface. The third is a Web GUI which we are going to focus on in our review.
Logging onto the Supermicro MBM-XEM-001 we see a more modern UI than we saw on previous Supermicro switches. We found this interface easy to use even on an iPad and Android phone.
Overall the web interface allows even SMB users easily manage networking. Simple tasks like changing the management IP for the out-of-band management requires a few clicks in the Web management.
One can use the web interface to check on networking for each of the nodes.
Beyond simple administrative tasks, one can also dig deeper into the switch. For example, one can setup VLANS and other features on a port by port basis or in groups of ports using the web interface. Here is a look at a screen to manage port based VLANs:
Overall, the 10Gb level 2 switch was very easy to manage with a few clicks of the Web UI. One can see the work that has been done to make menus and links easier to interact with via a touchscreen. This is an excellent feature that we wish more switch vendors would adopt.
Each of the Intel Xeon D nodes we tested is using the Intel X552 MAC from the SoC.
The Intel X552 is now part of the Intel ixgbe driver set. You will need to use the latest OSes and/or drivers to utilize these network controllers.
SMB-XEM-X10SM Network Performance
We used iperf3 to validate our networking performance. We wanted to check a few different scenarios:
- Blade-to-blade bandwidth
- Blade-to-external node bandwidth (connected to the chassis switch)
- Blade-to-external node bandwidth (through external infrastructure)
- 6x blades-to-6x external nodes bandwidth (through external infrastructure)
Here is a summary of our results.
Blade-to-Blade
This scenario tests bandwidth from one blade to another in the chassis. Here is an example but doing even simple blade-to-blade testing clearly showed that we have a 10GbE link.
We did test compute to storage and compute to compute node and the results were all similar since they are going through the chassis switch.
Blade-to-external node bandwidth (using a chassis switch)
We tested the setup from a blade through the chassis switch to a QSFP+ Xeon E5 node in the lab using a 1M QSFP+ DAC.
This is as we would expect given the switch specs.
Blade-to-external node bandwidth (through external infrastructure)
We tested the setup from a blade through the chassis switch to our QCT T3048-LY8 switch and to a SFP+ Xeon D node in the lab.
As one can see, performance remained identical within a standard margin of error.
6x blades-to-6x external nodes bandwidth (through external infrastructure)
In our final test we took the six MicroBlade nodes we had and simultaneously tested against six different targets using 2 QSFP+ 40GbE up links to our lab switch to get enough bandwidth from our chassis switch. This worked since we only had 6 nodes total to work with making our upstream bandwidth higher than what we could generate in the chassis.
There was very little variation and we were able to drive full bandwidth from the six nodes out of the chassis.
Networking closing thoughts
Networking is a bright spot for the 3U MicroBlade and it is one of the areas where we have heard a lot of positive comments. Using single node MicroBlades you have 140Gbps (in each direction) of NICs and uplinks that can total 160Gbps. Even with the dual node Intel Xeon D option, the chassis still has 280Gbps of machine traffic and 160Gbps of uplink, below the switch limits. The fact that there is so little over-subscription makes the MicroBlade platform very popular Xeon D platform. When you are working with applications such as Mirantis OpenStack or similar bandwidth hungry applications, the MicroBlade platform works well. We would have liked to have seen Level 3 features on the chassis switch but we understand features like Inter-VLAN routing were added in the updated MBM-XEM-002 switch that we did not test.
The Supermicro MicroBlade 3U system is quite complex. As such we have a multi-part article for the review. See the other portions of the review here:
- Supermicro MicroBlade Review Part 1: Overview
- Supermicro MicroBlade Review Part 2: Management
- Supermicro MicroBlade Review Part 3: Networking
- Supermicro MicroBlade Review Part 4: Dual node Xeon D compute blade
- Supermicro MicroBlade Review Part 5: Xeon D storage Blade
Check out the other parts of this review to explore more aspects of the system.
“4x 40GbE uplink ports”
Theres another switch like this: “XEM-002”. There are 2x40G and 4x10G uplinks.
A question: If one of these uplink ports not used – can I use for a direct connection to another single server or more(NIC)?
Or this switches cannot handle external devices just other (for xmple top rack) switches/routers on uplink ports?
Tibor, yes you can use this as a normal switch port if that is what you are asking.
Yes! Thank you Patrick!
This is so helpful!