Cisco Nexus 9000 adds 400G and 800G Options

Cisco today announced its new 400GbE and 800GbE options in the Nexus line. Today, we have the new Cisco Nexus 9800 and Nexus 9400 chassis switches and two new Nexus 9300 fixed switches all with support for 100GbE and 400GbE for next-generation networks. Later this year, we will have servers and DPUs/ IPUs with 400GbE as we transition to PCIe Gen5. As a result, infrastructure providers are looking at the 400GbE not just for aggregation layers, but also even down to individual nodes on high-end servers.

Cisco Nexus 9800 Series

Cisco has a new chassis switch that is designed for 100GbE/ 400GbE today and 800GbE in the future. Ranging in capacities from 57 Tbps to 115 Tbps the new Cisco Nexus 9800 series can either take four or eight line cards, two Supervisor modules, and two or three power supply trays per chassis.

The line new 400GbE cards are 36-port 400G QSFP-DD cards. These also support splitting into 4x 100GbE per port for higher capacity 100GbE networking. There are line cards available for lower-speed 10/25/50/100GbE as well. On the 400GbE card, Cisco is highlighting MACSec on all 36 ports.

As one can see, the Cisco Nexus 9800 series also has two control modules called the Cisco Nexus 9800 Supervisor modules. Cisco is not consistent on what it shares on its data sheets, but the Nexus 9400 series and 9300 series utilize Intel Broadwell-DE processors. In this configuration, we can see the large CPU heatsink with two memory modules and four DIMM slots which would match the Intel Xeon D-1500/ D-1600 series, and also the new Xeon D-1700 series. Our sense given the other models is that Cisco is still using the 2015-era Broadwell-DE here as with the other products announced today, but we hope to be wrong.

Next, let us get to the Nexus 9400 series.

Cisco Nexus 9400 Series

The Cisco Nexus 9400 series is designed with up to eight modules. This is a more compact but also a lower density. This is designed with a 25.6T Cisco switch ASIC and can have various 400GbE, 200GbE, and 100GbE ports, including MACSec for the 400GbE ports.

Here is the front of the chassis half-filled with modules plus a Supervisor node.

Here is the rear of the chassis. We can see the power supplies on top and the fan modules on the bottom. This looks a lot like the Facebook Minipack Switch that debuted in 2019 but was only half the 400GbE density at 64 ports using a 12.6T Broadcom Tomahawk 3 ASIC.

In the specs, Cisco lists a 4-core Broadwell-DE control plane in the Supervisor. It is somewhat interesting that they are doing this after we covered the new Xeon D-1700 series that replaces the 2015 generation in Welcome to the Intel Ice Lake D Era with the Xeon D-2700 and D-1700 series.

Cisco customers generally care less about this, but between Broadwell-DE and Ice Lake-D Intel introduced a number of new features including significant Spectre and Meltdown mitigations, as well as other encryption and AI improvements, so it is interesting that Cisco is not using the more modern part that has been out for over a quarter.

Next, let us get to the Nexus 9300 series.

Cisco Nexus 9300 Series

Cisco also has new fixed-configuration switches. The Cisco Nexus 9300 series added the N9K-C9364D-GX2A 64x 400GbE switch and the N9K-C9348D-GX2A 48x 400GbE switch. Interestingly here, the MACSec on all ports across the three 400GbE offerings is only on the 48-port model.

The N9K-C9332D-GX2B is the 1U 32x 400GbE model that is similar to the Inside an Innovium Teralynx 7-based 32x 400GbE Switch we looked at prior to Innovuim being purchased by Marvell. You can see inside that 32x 400GbE switch here:

This level of networking is becoming harder to fit in a 1U or 2U chassis just due to the power consumption requirements. Eventually, we will see these switches need to swap to co-packaged optics. As we have gone over previously, there are a lot of operational challenges with co-packaged optics so companies like Cisco have innovated to push the boundaries to switch a few more generations.

Still, it is great to see Cisco add new options here.

Final Words

While Cisco is focusing on the new 48 and 64-port Nexus 9300 today as well as the Nexus 9800 series and next quarter the Nexus 9400 series, which is the 400GbE generation. In the not too distant future, Cisco also showed its plans by building the Nexus 9800 series capable of future upgrades to the 800GbE generation.

200GbE enabled by PCIe Gen4 in servers was a relatively short-lived standard. 400GbE is where the market is looking next as it allows for breakouts to 100GbE or even enormous pipes into servers. For those looking at running a modern composable infrastructure, having a huge low latency pipe to an in-server or storage shelf DPU/ IPU is going to be a huge win. For others, it will just make for easier aggregation from lower speed 25/100GbE layers in the network.

Hopefully, we will bring you more 400GbE switches on STH later in 2022. These are certainly several steps up from the low-cost 32x 100GbE switch we reviewed just yesterday. Next though, we will be looking at the Facebook/ Meta Wedge100 to show you what the hyper-scalers are using in terms of switches. My part of the review is done, Patrick, our Editor-in-Chief, and the rest of the team just need to finish the video.