The New Intel Mesh Interconnect Architecture and Platform Implications

Skylake The Hidden Gem: Integrated Omni-Path Fabric

One of the biggest features is going to be Skylake-F, or Skylake CPUs with integrated fabric. Although it will likely be limited in sales, the fabric is underestimated in its potential impact.

Skylake-F is the codename for the Skylake package with integrated Omni-Path fabric. Much like the Intel Xeon Phi x200 generation has. From a price perspective, Intel charges under $200 premium for a dual port Omni-Path 100Gbps Xeon Phi x200 part. For comparison purposes, a Mellanox EDR Infiniband card costs around $1200.

With the new mesh and more memory bandwidth, Omni-Path sees a significant speedup over previous generation CPUs.

The actual implementation of Skylake-F looks something like this:

Each package has a protrusion where a cable is fed that connects to a backplane QSFP28 cage. That means each CPU has a 100Gbps RDMA network connection. In a four-socket system, that is 400Gbps of network capacity without using an add-in card or a standard PCIe lane.

If you adopt Omni-Path, you essentially get ultra low-cost 100Gbps RDMA enabled networking without using a PCIe 3.0 x16 slot. As a result, your effective PCIe lanes versus competitive products (e.g. AMD EPYC) skyrockets by 16 PCIe lanes per CPU. The downside is that you have to use Omni-Path. Unlike Mellanox VPI solutions, you cannot put these cards into Ethernet mode.

Just how low cost? Integrated 100Gbps OPA is about $155 more than the SKUs without it.

If Intel ever needs to change the game in this generation competing with EPYC in the scale-out storage/ hyper-converged storage space, adding an inexpensive 100GbE fabric option instead of OPA would be a complete win.

In the meantime, the bigger implication is that Intel is killing off the advantage that Xeon Phi has over the mainstream Xeons. In conjunction with adding AVX-512 we are seeing Intel move HPC back to its mainstream CPUs.

Final Words

Overall, the new Intel mesh interconnect architecture is required so the company can scale CPU cores and add more I/O without seeing ever increasing ring latencies. Each path on the mesh has fewer hops than the old rings had. If Intel gets serious about pushing fabric on package, then it is going to provide a huge value proposition for users.