Kioxia CM6 Review PCIe Gen4 SSDs for the Data Center

Market Perspective

A market perspective by STH’s Editor-in-Chief Patrick Kennedy

If you have arrays of SATA or SAS2 SSDs that are 4-5 years old, let us put the Kioxia CM6 level of performance into perspective. One of the reasons this article is getting published today is that four years ago, to this day, we published Testing the Phison S10DC data center SSD reference platform. If you have not heard of Phison, they are the controller company that many SSD vendors use. That was a 24-bay SATA SSD platform with three SAS HBAs. If you look at where we are today, only 1-2 Kioxia CM6 SSDs provide a similar, and a sometimes better level of performance than that 24-bay server solution.

PCIe Gen4 has the opportunity to provide major IT refresh performance gains that go well beyond a 2x performance improvement. In this example, we are effectively looking at a 1-2 2.5″ bays providing the performance of a 24-bay 2U dual Xeon server. If you are still using legacy SATA or SAS SSDs, the PCIe Gen3 NVMe SSDs often offered a compelling reason to switch. With PCIe Gen4 SSDs like the Kioxia CM6, the numbers get wild. Performance is one aspect, but in terms of capacity those 960GB SSDs were large four years ago but the CM6 mixed-use SSDs, such as the one we tested, scale to 12.8TB each. Even using capacity as the limiter, we could get a similar capacity in two drive slots instead of twenty-four.

PCIe Gen4 makes SATA SSDs utterly obsolete for performance applications. SATA SSDs still have useful applications for boot devices and similar use cases, but not in the performance realm.

The next question is in terms of platforms. That is, perhaps, the bigger obstacle at this point. PCIe lanes that SSDs connect generally terminate in one of two places. Either a PCIe switch or, the more likely scenario, directly to a CPU. There are some companies doing PCIe HBAs, but ultimately those add another device in the chain between the CPU and drives so those are more for backward-looking organizations who want to retain old enterprise models rather than for large organizations and more agile organizations.

The Kioxia CM6 is a data center product. The majority of server CPU vendors including AMD on the x86 side with their EPYC 7002 series or “Rome” processors, as well as companies such as IBM (POWER9 and soon POWER10), and the cadre of Arm players such as Ampere (Altra), Huawei (Kunpeng 920), Annapurna Labs/ Amazon AWS (Graviton 2), NVIDIA-Mellanox Bluefield, and soon Marvell with its ThunderX3 all support PCIe Gen4. Still, all of those players, combined, do not sell as many servers as Intel Xeons power. AMD, even with this feature out for over a year only has managed to claim around 6% data center market share in terms of units shipped and the others combined are not that larger.

While PCIe Gen4 will be a factor pushing organizations to Intel Xeon alternative platforms, Intel will finally have its PCIe Gen4 “Ice Lake” Xeons available later in 2020. Most OEMs are expecting volume shipments to customers to start in later Q1 2021 with some SKUs pushing into early Q2 2022. Those dates have shifted later already since the beginning of 2020.

That puts PCIe Gen4 in a “pickle” (another name for a baseball rundown) as a standard caught between two opposing forces. PCIe Gen3 which current Xeons use and that have been dominant for devices in the data center, and PCIe Gen5 where server architecture will begin a fundamental shift with technologies such as CXL. Intel’s “Sapphire Rapids” Xeon generation is scheduled to start initial production shipments in late 2021 and volume shipments in 2022. AMD and Arm players will also have PCIe Gen5 and CXL offerings, potentially before Intel. That means that we have only about six to eight quarters of PCIe Gen4 SSDs being dominant where PCIe Gen3 SSDs were used for more than seven years.

For those looking for maximum SSD performance today, the Kioxia CM6 and CD6 (review of that coming soon) are something new and very fast. You will need to explore alternative architectures such as the AMD EPYC 7002 (and EPYC 7003 Milan later in 2020), Ampere Altra, IBM Power9/ Power10, Marvell ThunderX3, NVIDIA Bluefield, the Huawei Kinpeng 920, or one of the other Arm-based processors out there. The benefit to that exploration can be lower costs, higher CPU and networking performance, and significantly faster storage performance.

Also, as a quick thanks to the team for having the fortitude to do the PCIe testing. That tied a test platform up for ~6 weeks to get enough test run data to validate the hypothesis that we could not compare results on Xeon platforms and that we needed to constrain our testing to a specific AMD EPYC 7002 PCIe lane set to get usable results. The simple answer would have been to just say a PCIe lane is a PCIe lane and not go to that level of detail. On the plus side, now that this validation is done, we have our EPYC 7002 CPU testing platform available for CPU reviews so expect that series to resume. /Patrick

Final Words

Pushing the performance boundaries of the Kioxia CM6 requires a look beyond just the PCIe Gen4 interface. Getting maximum performance from these drives requires knowledge of platforms and software stacks. Beyond just what we are showing here, there are other limitations. For example, PCIe Gen4 bandwidth in the AMD EPYC 7002 platform can saturate such a large portion of the CPU memory bandwidth and interconnect bandwidth that can limit the performance of higher-end arrays. Further, software stacks designed to service hard drives and SATA/ SAS2 SSDs may need to be re-tuned for this next-level of performance, and that tuning may not just be for x86 architectures in the future. Companies such as Amazon, Microsoft, and Google are aggressively pursuing PCIe Gen4 Arm infrastructure.

Looking back at the exercise, we probably went too far into looking at EPYC v. Xeon PCIe Gen4 v. Gen3 performance. Still, if those variables are not taken into account then it is hard to do a decent comparison. This is going to be a bigger aspect of storage performance as we get away from Xeons having a virtual monopoly on the market. It was time to start that discussion.

In our testing, the Kioxia CM6 performed extremely well. It powered through our testing clearly showing the benefits of the drive itself. For those still clinging to SATA III SSD arrays, the time has come to retire them. Consolidation ratios exceeding 10:1 for performance mean that the operational cost savings can be enormous. When we see market forecasters foretell the sharp decline in SATA SSD sales, the Kioxia CM6, as a first-generation PCIe Gen4 data center drive, clearly shows us why not just 2.5″ hard drives are endangered in the data center, but also SATA and SAS SSDs.

Kioxia is the first major storage vendor to have PCIe Gen4 SSDs available. If you are purchasing AMD EPYC servers, or Arm/ Power-based servers, then there is a tangible benefit to getting the Kioxia CM6 over PCIe Gen3 NAND SSDs. When Intel Ice Lake Xeons with PCIe Gen4 finally arrive, the picture may change, but for now, Kioxia has a leading product in the market.