At SC24, we learned something really neat. The NVIDIA BlueField-3 has a special version, and we are not talking about the BlueField-3 SuperNIC. Instead, there is a self-hosted version for primarily storage applications.
NVIDIA BlueField-3 Self-Hosted Version Necessary
At SC24, there were a few new storage servers with a twist. Instead of sporting an x86 or traditional Arm CPU, these servers had a NVIDIA BlueField-3 DPU installed as the host processor. Some of them included not just storage, but through the magic of a PCIe switch, were able to host a GPU alongside the NIC and storage.
It turns out that NVIDIA has “SH” or self-hosted version of the BlueField-3 DPU. These are able to expose the PCIe root and then have other PCIe devices like NVMe SSDs and GPUs attached. That is a bit strange since NVIDIA has been using the BlueField-3 DPU downstream ports for storage within servers. The best way I have had it explained is that it is for the gold finger ports.
One of the big benefits of the BlueField-3 generation is its significantly higher memory bandwidth since BlueField-2 was a single-channel design. That BlueField-2 actually had less memory bandwidth than the BlueField-1 as a result. With the BlueField-3 generation, we get faster everything making it a big upgrade. With its own 16-core Arm A78 infrastructure, it has a capable host processor for applications that do not have high compute requirements like storage. The BlueField-3 has become NVIDIA’s lower-power network-attached CPU solution as well.
Final Words
While the standard NVIDIA BlueField-3 is the B3220, the self-hosted models add “SH” and are thus B3220SH. It is interesting that NVIDIA has distinct cards here given the onboard PCIe switch they are using. The self-hosted version is in the NVIDIA docs, but it is something that not many folks talk about in the industry unless they happen to be building a storage appliance that uses the cards.
Hopefully this is a fun one for our readers.
I like the bottom right port with just the label “Black Cable”
BF2500 also provided a root complex. In the STH BF2 ZFS article (with the JBOF backplane) it was very vague about the configuration and mentioned using BF2 516A parts. I always assumed it was possible Nvidia provided firmware or some enablement for root complex (turning 516A into 516B).
I have some BF1 boards and they are useless – no way to get software for it, or a BSP (board support package) to build an own distro – despite having much more capability than a regular desktop system. I really hate NV for producing new and new hw, and dropping support for previous generations so quickly. That is a brutally forced short update cycle, not seen anywhere else. Same goes with Tegra chips – they never achieved complete support of announced features before getting EOLed.