Inside a Supermicro Intel Omni-Path 48x 100Gbps Switch

With ISC 2019 coming this week, we wanted to do two things. We wanted to kick off our “Cool Server Hardware Series” and at the same time, highlight something that may be found in supercomputer centers. We have a quick tour of what is inside a Supermicro SSH-C48Q which is a 48-port 100Gbps Omni-Path switch that has been running in our lab for almost two years.

Inside a Supermicro Intel Omni-Path 48x 100Gbps Switch

The switch we are using here is a Supermicro SSH-C48Q which is a 48-port Intel Omni-Path 100Gbps (OPA100) switch. Intel Omni-Path 100 was a major focus for the company around 2017 with the Intel Xeon Phi x200 series and the initial Intel Xeon Scalable “Skylake-SP” series of CPUs. Over the past few years we mostly have worked with the Intel Skylake on-package Omni-Path Fabric using Intel Xeon Gold 6148F CPUs and even PCIe cards. We wanted to take an opportunity to look inside the switch we were using in our lab.

Here is a quick video tour of inside the switch:

Although this is in many ways a first generation product, it is a relatively simple reference design that was reasonably elegant. With 48 ports, one needed fewer switches in a fabric topology lowering latency and cost versus smaller 32 and 36 port switch fabrics.

Final Words

My feedback to Intel is that it needs an Ethernet fabric solution. If one could buy chips with 100GbE connectivity for $200 more per SKU, that would be almost a category killer for networking. As momentum for Intel Omni-Path 100 slowed, and the OPA200 200Gbps generation teased a year ago did not make it to 2nd gen Intel Xeon Scalable CPUs, this may be a technology that Intel plans to reboot. Still, this switch ran demos beyond typical HPC applications including NVMeoF and even Optane NVMeoF using the low cost, and high-performance 100Gbps fabric in our lab.

One of the great parts of the server industry is that there is constant innovation. Sometimes products are blockbusters. Sometimes they sell slowly until there is a next-generation product tweak. With our Cool Server Hardware Series, we are going to highlight some of the hardware you may not see every day but we get to see in our lab.

There is one thing that seems almost certain. As large HPC clusters get decommissioned with OPA100, the “F” series Intel Xeon Scalable CPUs with onboard Omni-Path are likely to have an excellent second life because that 100Gbps low latency fabric is very useful. We also think that our recent coverage of Intel to Acquire Barefoot Networks for Ethernet Switch Silicon and the recent Intel Ethernet 800 Series 100GbE NIC Launch show that Intel is more serious about upgrading its networking fabric portfolio.