Juniper Express 5 28.8Tbps Network Routing ASIC at HC34

At Hot Chips 34, we got an insight into the new Juniper Express 5. This is a cool modular chiplet approach to building an ASIC for routing and switching devices. With two chiplets, the company can make a number of ASICs for its portfolio.

Note: We are doing this piece live at HC34 during the presentation so please excuse typos.

Juniper Express 5 28.8Tbps Network Routing ASIC at HC34

In the old days, there were routers or switches. There was a question if routers would still exist or if we would just have smart switches. These days, routers can switch and switches can route.

Juniper has a number of ASICs that it makes for routing. The new one here is the Express ASIC for Juniper’s PTX line.

There are two main ways that people build big networks. Either 1U-2U boxes and scaling out or scaling up using chassis switches.

On the scale-out side, we normally get 32-port Radix for smaller networks but 36-ports is for networks with deeper buffers. 36 pluggable cages is what fits in a standard 1U rack so we get a 28.8Tbps switch as 36x 800Gbps.

Juniper also discussed the scale-up modular chassis.

When one looks at the scale up and scale out, they can look similar in some ways, but in practice, they are not.

Juniper Express 5 has a cell-based fabric.

Making its new chips is the X-chiplet. This has networking interfaces, packet forwarding, packet buffer (on-die), and XSR interfaces. It is made on TSMC 7nm. Size-wise, this is similar to a modern GPU in terms of TSMC 7nm transistor count.

The F-chiplet is the fabric and switching chiplet.

Between the chips there are 112G XSR PAM4 SerDes. One reason for this is that Juniper is looking to the co-packaged optics future.

32Tbps is RX+TX bandwidth in the below slide. This is a giant chip with two X-chiplets and multiple HBM stacks. The HBM2e gives massive capacity and a lot of memory bandwidth to help the device perform routing functions. Note here that there are no F-chiplets because this is the routing device.

Here are the features of the big ASIC.

Here are the packet forwarding pipelines.

That big ASIC is not the only one. It can also have a X and a F chiplet configuration called ASIC 2. This also has HBM2e.

Removing HBM and the X-chips there are ASICs for different size chips based only on one or two F-chiplets.

ASIC 5 is a X-chiplet with HBM2e and no F-chiplets.

ASICs 6 and 7 are X-chiplets without HBM in either 2 or 1 X-chiplet configurations.

Here is the co-packaged optics (CPO) being attached using XSR.

In the end, Juniper gets seven different types of ASICs with only two chiplets and an eighth with the CPO option.

This leads to building a whole portfolio of scale-out and scale-up solutions with just the two chiplets.

Final Words

This is a really interesting design as it is flexible and is different from what many in the industry are doing. Juniper offers some support for things like SONiC and P4 across its portfolio but this is largely going to be innovation for Juniper hardware and software customers unless Juniper decides to put more emphasis on SONiC / P4. During the conference, someone from ByteDance (known widely for TikTok) asked about SONiC support so hopefully, that helps push towards more open networking. Then again, with Juniper, you are getting the hardware, software, and support so its model makes sense.