Edgecore AS7712-32X Switch Overview A 32x 100GbE Switch

In this article, we are going to take a look inside a bare metal switch, the Edgecore AS7712-32X. This is a bare-metal switch with ONIE and broad network OS support so you can use it with most major NOS distributions. It features 32x 100GbE ports that can break out to 25GbE ports in a rack making the switch extremely flexible. In our overview, we are going to look at the switch and see how it was built.

Edgecore may not be an overly well-known brand if you have always used Cisco hardware. The company is a subsidiary of Accton which is one of the big networking ODMs. When you read reports about hyper-scale companies like AWS and Facebook leaving their Cisco gear and instead of going to merchant silicon, that is the market that Accton has excelled in. An example of this collaboration between hyper-scale and Edgecore is the Facebook Minipack which is a switch Facebook is using for 100GbE and 400GbE networking. While you may have never heard of Edgecore or Accton, their hardware is running many of the hyper-scale services you use every day.

Edgecore AS7712-32X Video

We took some video of the switch if you are on the go and prefer listening to learn about the switch. Check that out here:

That video gives a few more angles than we can show in photos just due to being video. Of course, we have more detail in our piece here so keep reading to dive in.

Edgecore AS7712-32X Switch External Overview

Looking at the front of the switch, we can see the 32x QSFP28 100GbE ports. Where we used QSFP+ for 40GbE, we now use QSFP28 for 100GbE switching. Those 32 ports dominate the front of the switch.

We wanted to show three important features if you use this switch. There is a console, out-of-band management, and a USB port. All three are important. The serial console port is a standard feature in the industry. One can also see the management port and USB port. An essential part of the bare metal switch and ONIE is loading your own operating system. As a result, these features are often used to accomplish that first step in using the switch. After the NOS is installed, that management port is also the out-of-band management port for the switch.

Moving to the rear of the switch we can see a fairly simple, but designed to be an easily serviceable switch. There are six numbered hot-swap fans that run in a 5+1 redundant manner so even in the unlikely event that a fan fails, the switch can operate while that fan is being replaced.

One will also see two power supplies. These are also hot-swappable units and have internal fans to keep them cool in the switch as well.

For most users, this is all they will see of the Edgecore AS7712-32X. We, however, wanted to share a look inside.

Edgecore AS7712-32X Switch Internal Overview

Looking at the internal overview of the switch we can see a layout where we see the ports and the main switch chip PCB, we have the management portion, then we have the cooling control in the rear of the switch.

One item we wanted to note here is just how many fairly large ICs there are in a switch like this. We are going to talk about them next, but we just wanted to point out that there are chips from Intel x86 and Altera CPLDs/ FPGAs which is now part of Intel. There is the Broadcom switch chip. One can even see the SmartFusion Coretex-M3/ FPGA SoC onboard. There is a ton going on here.

The main switch chip here is the Broadcom Tomahawk BCM56960 which is a 3.2Tbps switch chip that can do 6.4Tbps full-duplex. That fits nicely with 32x 100GbE ports. The Broadcom Tomahawk was perhaps the first widely-adopted 100GbE switch chip. Although there are new generations and Broadcom is pushing 12.8Tbps versions of newer Tomahawk chips for 400GbE switches, this is an extremely popular switch chip.

We wanted to quickly point out that this PCB has more than just the switch chip and ports. One will also see three low-power Altera (now Intel) Max V CPLDs, There is one more of these chips on the fan PCB for a total of four that we saw.

That is not the only Intel-Altera product. The switch also has a Cyclone IV FPGA next to the primary switch chip.

Perhaps the most interesting feature is the Intel Atom C2538 chip onboard serving as management for the switch. This is a low power quad-core SoC that is part of Intel’s Rangely line. It runs at 2.4GHz and has a 15W TDP. You will notice the small heatsink on the SoC. Edgecore can get away with this since the switch is moving so much air.

An item we wanted to note is that since these are Intel Atom C2000 series products, one needs to be aware of the Atom C2000 Series Bug. In 2016 about three years after they launched, Cisco identified an issue where the Atom C2000 series had a chance to cease working after years in operation. There is a board-level workaround and in 2017 an Intel Atom C2000 C0 stepping came out fixing the AVR54 bug. The problem is that devices such as switches and firewalls are not the easiest items to replace and the chips are everywhere. At STH, we even fell victim to the Intel Atom C2000 AVR54 Bug. Datacenter power is reliable, however, we suggest that you ensure that your Edgecore AS7712-32X has a newer chip or the workaround. If not, plan accordingly.

Since the Atom C2000 series was launched in 2013, it utilized DDR3 memory, not the newer DDR4 that current chips all use. We can see two 8GB SODIMMs for a total of 16GB of memory onboard.

Since a management controller in this type of switch looks a lot like an embedded x86 server, we also have a 64GB mSATA SSD.

Since this is a switch, we also have an Actel SmartFusion A2F200M3F chip with FPGA logic and an Arm Cortex-M3 processor.

Going back to our original diagram, one can see just how many good size logic ICs there are in this switch and how it has been designed for flexibility. One way to think about these bare metal switches is as low-power servers that control high-speed networking subsystems.

You can see that the switch has airflow guides to keep PSU airflow and main switch airflow separate. Airflow is specifically directed around the Broadcom Tomahawk switch chip and the QSFP28 ports to ensure the higher-power components get enough cooling.

One can see internally the hot-swap fans that have their own PCB to monitor fan operation and ensure that failures are diagnosed and replacement is relatively easy. If you were wondering, this switch can run well over 60dba like many servers and other data center equipment.

One nice feature is that the Edgecore AS7712-32X has LED light channels so one can easily see which fan has failed.

At this point, you have probably noticed that there is a fan PCB, a switching PCB, and a management controller PCB with the Atom SoC. Edgecore sells the AS7712-32X with a different CPU subsystem. Specifically, the AS7716-32X has an Intel Xeon D-1518 chip onboard. Building switches in this modular fashion allows Edgecore to gain efficiencies when creating switch lines with different port configurations and different CPU options.

The power supplies are 80Plus Platinum rated units. The cooling for these is not redundant, but there are two power supplies each of which can run the entire switch leading to a 1+1 redundant setup.

Edgecore AS7712-32X Switch NOS Options

The Edgecore AS7712-32X comes pre-installed with Open Network Install Environment (ONIE) so you can load your network OS of choice onto the switch. From Edgecore, here are the options one has:

• Open Network Linux (ONL), an open-source, OCP reference NOS.
• Netvisor ONE Network operating system from Pluribus Networks.
• Big Monitoring Fabric 5.6.0 and later version from Big Switch Networks.
• Cumulus Linux 3.1 and later versions.
• SONiC open-source network software.
• PicOS 2.7.1 and later version from Pica8 Inc.
• OcNOS 1.2 and later version from IP Infusion.
• Ixia Vision Edge OS 4.7.3 and later.
• SnapRoute CN-NOS, Cloud Native – NOS software.
• Infinera CNOS.
• ArcOS 3.3.1 and later versions.

In our next lab build-out, this switch will be used with SONiC, however, the idea has been tossed around using ONL.

While the Edgecore AS7716 has better hardware with the onboard Xeon D, it only is compatible with a subset of the above. At some point, we want to see the open networking world becomes more CPU agnostic. These days if you want to use many of these OSes, you need not just an x86 CPU, but also a specific model or line. This is happening over time, but there are still some hardware constraints. The Edgecore AS7712 is popular because it works with many OSes that have been tested and validated on the switch.

We wanted to take a quick second to point out that this is an OCP Accepted design. That means that the switch complies 100% with an OCP approved specification and the design files are open-sourced and available. This has become a popular switch for that reason.

Final Words

Hopefully, you enjoyed this tour of the Edgecore AS7712-32X switch. We thought this is an important piece since the AS7712-32X is extremely popular and runs many of the popular networking distributions out there. For some of our readers looking at next-generation open networking, we wanted to show how a switch like this is designed at a high-level and some of the reasons why. For others, this is just a look at a cool piece of hardware that you will know more about it should you ever encounter it in a data center.