At Flash Memory Summit 2015 we saw a number of interesting designs where companies were trying to cram as much PCIe/ NVMe flash in a single system as possible. Practically, there are only so many PCIe lanes that Intel provides on each CPU, up to 40 PCIe 3.0 lanes on each Xeon E5 CPU. With x4 devices, that allows for an easy expansion path to 10 drives per processor. If one wants to go higher, a different architecture is needed. Avago recently purchased not just LSI but also PLX. With the LSI acquisition Avago has the ability to expand SAS implementations. With the PLX acquisition, Avago has the ability to expand PCIe lanes.
The key component is the Avago/ PLX PCIe switch line. At a very high-level, these chips operate similar to a network switch where one can have one or more uplink ports and hang a number of devices off of the switch. Here is a nice shot of what this practically looks like from a company called Serial Cables. One can see four SFF-8644 external PCIe cables in the rear each carrying 4x lanes for a total of 16 lanes uplinked to a system. Convieniently, the same number of lanes in a standard PCIe x16 slot. The front of the board has 12x SFF-8643 connectors. So we have 48 lanes to use to connect drives. The Avago switch chip is therefore likely a 64 PCIe port switch.
The benefits of this are large. One can use these chips to increase capacity (on this board) by a factor of 3x. Taking this to a larger level, there were a few systems on display from One Stop Systems. William recently reviewed a One Stop Systems PCIe expansion appliance focused on desktops. The company had a system build to OpenCompute specs that could handle m.2 SSDs:
One can see no less than seven Avago/ PLX chips there. The company also showed off expansion chassis that could handle 2.5″ top loading PCIe/ NVMe SSDs using special adapters:
Here is the same system full of add-in-card type SSDs:
A number of vendors did show off traditional looking 2U and 4U appliances that had 24+ PCIe drives installed. Although we could not take pictures of the chips due to the display cases, simple math deduced and we confirmed with company reps that, these systems were indeed using Avago PCIe switch chips.
Of course, SAS has been using expanders and large topologies for years. One of the more interesting SAS arrays was the SanDisk IF100 which we got to see in person for the first time:
The IF100 does not use Avago PCIe chips but is certainly a possibility where these earlier stage PCIe designs are heading.
Conclusion
Expect to see more PCIe switch chips from Avago in larger storage systems in the near future. The PLX technology has been around for years and using Avago switches does provide some unique capabilities e.g. dual ported PCIe/ NVMe drives are available that when combined with Avago switches and the right chassis can provide Active-Active controller NVMe drive systems.
Now, you have excited a huge e-woody….. OMG
Hi Patrick. You wrote>”… With the PLX acquisition, Avago has the ability to expand PCIe lanes. …”
Will PLX allow expanding the number of PCI-e lanes or only re-arrange say a x16 connections to four x4 connections?
I don’t think board containing a PLX chip can do more than divide an EXISTING x8 or x16 slot in to two x8 or four x4 connections?
Is there an inexpensive motherboard yet with on board NVME 8643/8644 mini SAS-HD connectors that would accept the Intel 750 SSD SFF-8639 to SFF-8643 cable?
What about an inexpensive 5.25 inch bay that supports four SFF-8639 connectors or four SFF-8643 connectors to four hot direct connect swap SFF-8639 connectors on the SSD?
Sorry for so many questions! Hope you, or others, have time to respond on target.
Thanks,
Paul
>Patrick wrote in another thread: “… Supermicro AOC-SLG3-2E4 has a PLX chip that allows bifurcation down to x4 devices.
Apparently the lower cost R board works with some motherboards. I guess the ones that already od bifurcation?
Is bifurcation software/bios or hardware? Do we know what allows the R to work with some motherboards. Why is the PLX chip sometimes needed and sometimes not?
Thanks,
Paul
The salad looks great!! I haven't really had any outings lately been working a lot and my girls have been sick:( Thanks for visiting my blog!!!
Good rant, Howie. Not the first time I've seen (or run into, for that matter) conservatives posing as progressives who tried to rope me in–even after giving me a good whiff of their slimy corporatist values.