Recently, we moved one of our Arista 32x 100GbE lab switches. Patrick, our Editor-in-Chief, pulled it out of the rack and wanted to do a little feature on it, so we put together a short “inside the switch” look of the Arista DCS-7060CX-32S 32x 100GbE switch. Unlike servers that we get to do weekly looks at hardware, switches are much harder to take a look at. As a result, Patrick did the video, and I am doing a web overview so our readers can see how these are made.
Inside the Arista DCS-7060CX-32S 32x 100GbE Switch
If you want to see the video version, you can see it here.
As always, we suggest opening it in a YouTube tab. There are more angles there than we can show in pictures.
Arista 7060CX-32S Hardware Overview
The switch itself is a 1U form factor for standard 19″ racks. The front of the switch has 32x 100GbE ports. One will notice that there are also two SFP+ ports on the left side of the switch.
We also have a management, RJ45 COM port, and USB port on the side to manage the switch.
The rear of the switch has four hot-swappable fans and two power supplies. One can see these have red handles so we have a port to PSU airflow switch.
While the external part of the switch one can see often, we wanted to show what is inside.
Behind the QSFP cages, we have the Broadcom Tomahawk (1) switch ASIC in the middle of the switching PCB. This is a 6.4T switch chip for bi-directional 32x 100GbE port traffic and is capable of around 3.3Bpps.
We will note that Broadcom has iterated on this original Tomahawk chip several times but when it first arrived in late 2015/ early 2016 this was a high-end 1U switch platform.
The switch cooling solution is interesting. There is a center section heatsink with a few fins. That center heatsink has heat pipes that dart to either side with much denser fin arrays. The total heatsink length is around 15″ including the tips of the heat pipes.
The heatsink wings are positioned well above the PCB. Underneath, we can see that there are lower-power components. Here we have an Intel-Altera Cyclone FPGA just below one of the wings.
Behind the main switch PCB we get the control plane PCB. Here is a look at the switch from the rear with the power supplies and fans installed.
Here is another look without the power supplies.
Many of the switches we see have power supplies that connect to the control PCB rather than the switch PCB. This Arista switch is designed to directly connect the power supplies to the higher-power switch PCB.
Powering the system we get an AMD embedded SoC. This is the AMD GX-424CC from what we can see in the switch. Arista has an x86 platform for managing the switch. A fun fact is that this chip technically has Radeon 5E graphics onboard. Although there are no display outputs, technically there is AMD GPU IP in this switch.
Something else that is very important here is that by using an AMD embedded CPU instead of the common Intel Atom C2000 series. As a result, we do not have to worry about the Intel Atom C2000 Series Bug (AVR54). We have had the Intel Atom C2000 AVR54 Bug Strike STH. Atom 2000 switches, without the fix or new chip stepping can suddenly fail to boot. Using the AMD embedded CPU, this particular bug is not a concern unlike many switches such as the Celestica Seastone DX010 32x 100GbE Switch we looked at a few weeks ago.
Memory is very interesting. There are two DDR3 slots and one is populated. Per the spec sheet, we were expecting to see a 4GB DIMM here. Instead, this is clearly an 8GB DIMM.
Another small feature that is not populated here is an M.2 SSD slot. We did not see this in the spec sheet, but it appears to be designed to take up to an 80mm SSD.
The fans have nice hot-swap connectors where tension from red clips is used to keep the fans in place and guide them to the connector.
The power supplies were another area where the switch did not match the spec sheet. The spec sheet says 500W power supplies however these are 495W 80Plus Platinum Delta power supplies.
In the lab, this switch runs under 200W for us and the specs state a maximum power draw of 333W, so to us, the 5W difference is more of a curiosity rather than a cause for concern.
One other quick note that we wanted to mention about this switch. If you want to run EOS, then that is fairly straightforward on this platform. If you want to venture into the world of open networking, then this switch supports SONiC. You can check out Arista SONiC on GitHub.
We took a snapshot of the list above because at STH we had a guide years ago on getting started with 40GbE SDN with Microsoft Azure SONiC for under $1K using the 7050QX 40GBE generation of this part.
Final Words
Overall, we wanted to show one of Arista’s 100GbE switches that you can use to get started with SONiC. If you either currently or hope to, work in the networking space, it is likely wise to learn about SONiC since that is a huge force in the industry. Also, 100GbE is a rapidly growing standard that will be bolstered by the transition to PCIe Gen4 this year and PCIe Gen5 in 2022 and 2023. In the last year or so, our test platforms at STH have been adopting 25GbE and we use 100GbE switches with breakout cables in our lab racks for testing. With PCIe Gen4 we can get dual 100GbE ports in a single PCIe Gen4 x16 slot, and there are new lower-cost 100GbE NICs such as the Supermicro AOC-S100GC-i2C 100GbE Intel 800 Series NIC hitting the market.
Hopefully, you enjoyed this quick look at the Arista 7060CX-32S.
I wonder if someone could get a video output by using that M.2 slot to convert to a PCIe slot and put a GPU in. It’ll be interesting to see if it actually makes a video output.
What is your take on switch air flow, a a vendor’s rep told me that you can do it either way, even thought they have models for front-to-back and back-to-front. Does matter in terms of heat?
I’ve never understood why vendors don’t just include a switch on the front or back that allows you to flip between front-to-back or back-to-front airflow. The hot-plug fans and power supplies would obviously need a connector to detect the airflow setting, but it could be done. And the fans would need 45 degree blades, not curved.
Some switches have to be ordered one way or the other, and heaven help you if you want to re-use the switch somewhere else with different airflow needs.
Ahh cool, interesting about the m.2 and gpu!
we use these at work, plus multiple other models!
Wasn’t aware about the sonic stuff pretty cool.
Omar that’s correct, their colour coded red or blue showing which way the air flow is going, which should be based on the rack configuration of hot vs cold aisles in the DC
Enterprise switches: it would be nice if STH could add one more test to the switch reviews. Are the power supplies “string attached” to the main board meaning is there “logic” between the main board and the PSU so that the switch main board only works is the PSU replies through a particular protocol, which obviously is proprietary and unpublished. This has two consequences:
1) You can only buy replacement PSU from the vendor.
2) Years later, when you buy a used switch you cannot replace the PSU if the vendor stopped supporting them, rendering the switch useless if the PSU is not working.
I’ve got around 200 of the slightly more uprevved CX2 model in production at a few site around the world, and I love these things. Solid as a rock, excellent software, and amazing support.
So the embedded AMD chip has graphics built onto it?
I guess that’s better than spinning up a special SoC just for this product.
But I have to ask… Will it play Crysis?!