Mellanox BlueField NVMeoF SoC Solution

Perhaps one of the more logical and exciting announcements from Flash Memory Summit 2017 was the Mellanox BlueField announcement. Mellanox BlueField is an amazingly logical solution to a rapidly developing market. Vendors today are trying to build new NVMe over Fabrics (NVMeoF) solutions. Mellanox is seizing this opportunity to utilize its networking leadership position to bring together a solution for the NVMeoF marketplace.

Mellanox BlueField NVMeoF SoC Solution

Here is the diagram of what Mellanox is trying to solve for:

If one conceptualizes NVMeoF to a list of components, what is really there? There is a high speed RDMA fabric (Infiniband, Ethernet, Omni-Path or similar) to connect nodes together and expose storage to compute devices. There are PCIe switches because you will generally have more devices than fabric bandwidth. Finally, there is a CPU to provide services atop of the raw infrastructure and attach DRAM for a caching tier. Mellanox essentially wraps all of that into its SoC solution:

There are a few key highlights. First and foremost it is using PCIe Gen 4. Both AMD EPYC and Intel Xeon Scalable are using PCIe Gen 3. For 100Gbps network adapters and even NVMe storage, this is becoming a bottleneck. As an example, dual port 100GbE adapters cannot operate at full bandwidth using a PCIe 3.0 x16 link. The answer is PCIe 4 and a few years later PCIe Gen 5, but for now, x86 architectures are not supporting the bandwidth needed for next-gen NVMeoF designs.

Mellanox’s answer is to use a 16 core ARM CPU for DDR4 attach as well as allowing vendors to provide network services. It then can use its ConnectX-5 IP for 100Gbps Infiniband or 100GbE fabrics. Finally, it adds a 32-port (external) PCIe switch with a PCIe x4 x16 backhaul to the rest of the SoC. What that means is that Mellanox has all one may need to run a NVMeoF storage server in a single SoC. That SoC is also a generation of PCIe 4 which means companies using BlueField can get a technology generation beyond what AMD and Intel are offering.

Here is the keynote from Mellanox at Flash Memory Summit 2017: