External SAS/ SATA Disk Chassis Wiring – Part 2

In External SAS/SATA Disk Chassis Wiring – Part 1 we looked at wiring a server with the cables and adapters needed to connect to an external SAS/SATA disk chassis. Now we can move on to wiring up the JBOD disk chassis itself. This article is for those who plan to assemble their own disk chassis, JBOD or RAID, or those who just want to better understand the cabling involved.

Our external SAS/SATA disk chassis starts with disks. The disks themselves are not the subject of the article, so we’ll gloss over your options: 2.5″ versus 3.5″, SAS or SATA, single or dual ported. For your home storage server you’ll probably select 3.5″ SATA drives. 2.5″ drives and both single and dual ported SAS drives are much more expensive. Let’s assume that you have done so unless otherwise noted. Whatever disks you select will be mounted into drive sleds and inserted into hot-swap backplanes. SAS/SATA backplanes are preferable, but you might find a good deal on SATA backplanes so we’ll talk about those as well.

Wiring for SAS Backplanes

If your backplane is a standard non-expander SAS backplane then it will have one SFF-8087 port for every four disks. We’ll talk about other backplanes a bit later in this article. We remember SFF-8087 ports and cables from Part 1 of the article and the SAS/ SATA cable guide. As a reminder, the photo below is of one of six SFF-8087 ports on the back of a Supermicro 24-bay backplane. The port shown has a SFF-8087 cable already attached. A second SFF-8087 connector is visible at the right of the photo, partially hidden under the large blue SFF-8087 cable.

We need to convert this SFF-8087 backplane port to a SFF-8088 female port at the rear of the disk chassis. As with our server in External SAS/SATA Disk Chassis Wiring – Part 1, we accomplish this with a PCI SAS adapter bracket and a SFF-8087 cable.  The adapter bracket mounts in the back of the disk chassis in a PCI opening. Since we have done exactly this in our server already, we can skip the details and show the results. The photo below shows a 24-bay external disk chassis with six SFF-8087 ports (one port per four drives of course.) It is wired using three Supermicro pre-wired dual-port PCI SAS adapter brackets model CBL-0168L. The small green board is the Supermicro power board from the earlier article on that component. This chassis is ready to connect to a server with six SFF-8088 ports of its own – or to connect to up to six different servers, since each disk port is an entirely separate connection to a group of four drives. A database server in my environment uses three dual-port LSI SAS cards to connect to this chassis. Running six SFF-8088 cables from a server to a disk chassis is a lot of wires, but the resulting system provides real-world throughput of over 8 Gigabytes per second using 24 SSD drives. Theoretical bandwidth for this setup is an enormous 144 Gigabits per second.

At the rear of the chassis you can see the SFF-8088 ports. The photo below is from a similar chassis, but one with eight ports instead of six.

Wiring for SATA Disk Backplanes

If your JBOD disk backplane is a SATA backplane instead, or if you are using SATA drives without a backplane at all, then the wiring is just a little bit different. In that scenario we need to convert your SATA disk connectors to SFF-8088. We use a PCI SFF-8088 to SFF-8087 adapter bracket as before, but instead of a SFF-8087 cable, we use a SFF-8087 to SATA breakout cable, also known as a SAS to SATA “Forward” breakout cable. Sometimes these connectors are called 7-pin SATA or SAS connectors. One such cable is shown below, attached to a PCI SAS adapter bracket. You’ll also see “Reverse” breakout cables sold. These are used to convert four motherboard SATA ports to a SFF-8087 connector – something I advised you not do to in Part 1 due to cable length and signal strength limitations. Other than the use of the forward breakout cables, the wiring is identical to our earlier chassis.

Basic Wiring for SAS Expanders

The external disk chassis above, with multiple SFF-8088 ports, uses the least complex wiring setup and is perfect if you are using SSD drives and need maximum bandwidth from your disk chassis. Surprisingly, it can also be the lowest-cost solution for more general-purpose use if you have an inexpensive way of outfitting your server with multiple HBAs. Often, however, a different solution is desired – a chassis-mounted expander. SAS expanders allow you to connect many more than four drives to each SFF-8088 port on your server. Expanders add another tier to the wiring setup, and we’ll explore that now.

A SAS expander mounts within your external disk chassis and connects to your server with one or two SFF-8088 ports and then connects to multiple disk backplanes with other SFF-8087 ports. The most commonly available expanders come in two physical formats: PCI and what we’ll call “Rectangular” for the lack of a better word. The PCI form expanders are designed to mount in PCI slot. While they can draw power from the slot, they do not send any data at all via that port. In other words, this format of SAS expander cards look like SAS adapter cards, but they aren’t. The photo below shows a Chenbro SAS expander. HP offers a similar PCI-based SAS expander with a single external port. Patrick used this HP SAS Expander in his early DIY SAS Expander chassis articles years ago.

A PCI format SAS expander like the one above simply replaces the PCI adapter bracket in our earlier wiring architecture. We mount the expander in a PCI slot within our disks chassis and connect the SFF-8087 cables from our disk backplanes to the SFF-8087 ports on the expander. The result looks like the diagram below.

The other form of SAS expander is the “Rectangular” form like the Intel expander below. These are designed to mount entirely within a server chassis and use SFF-8087 connectors for all connections.

In our setup we’ll need to convert one of these SFF-8087 ports to a SFF-8088 connector for connection to our server. We’ll do this with the same kind of PCI SAS adapter bracket as before. Interestingly, Intel also sells a SAS expander that uses the PCI format and yet does not have external SFF-8088 ports. We’ll treat this like the rectangular format, using a PCI adapter bracket to convert one of the ports to SFF-8088. The image below shows the wiring for the rectangular type of SAS expander.

More Speed? Better Fault Tolerance?

The SAS expander setups above feature very simple single-cable wiring between server and disk chassis, and with the appropriate RAID card in the server they allow us to create a RAID volumes containing up to all of the disks in our disk chassis. Daisy chaining, a key feature of SAS expanders, won’t be discussed in this article, but enables even larger numbers of disks per RAID or HBA card. The downside of this simple expander wiring is that we have limited bandwidth – around 2GB/Second with current generation hardware – and several possible “single points of failure”. We can address the bandwidth limitations by using a feature of some SAS expanders sometimes known as x8 wide porting – combining two x4 ports. With a compatible RAID card and SAS expander, you can have dual SFF-8088 connections between the server and disk chassis, doubling the theoretical bandwidth. It is also possible, with the right hardware and the right operating system drivers, to achieve both increased bandwidth and failure tolerance by using wide porting and multi pathing. Most such setups also utilize dual-ported SAS drives for additional speed and redundancy. The details of such a setup are outside of our scope, except that I’ll show one possible wiring diagram for reference. See below.

Other Notes

This article covers the most common disk chassis cabling issues, but is not comprehensive. Fire off a message if you have a corner-case and would like an opinion from me and/or the talented STH community. Since someone will probably ask, I will share these additional tips now:

• Some JBOD disk chassis lack PCI bracket openings and have smallish rectangular openings instead. Look for Centronics SAS adapter brackets for these instead of PCI SAS adapter brackets – the guts are the same but the brackets are shaped differently.
• As in Part 1, we are purposely ignoring the new and still rare new style SAS connectors for the sake of simplicity. Wherever you see “SFF-8088” in this article, you can read it as “SFF-8088 or SFF-8644”. You can also translate “SFF-8087” to “SFF-8087 or SFF-8643”. The new connectors are smaller than the old ones, which is good news for high density servers.
• SAS expander cabling is quite simple, but SAS expanders quickly get complex when you factor in compatibility, performance, and features beyond the basics. Do your homework before buying an expander.
• You do not need to mount a PCI SAS expander in a PCI port. These cards include a Molex port that can also be used to power them. The HP SAS Expander does not have a 4-pin Molex power connector. Instead it must be placed in a powered PCIe slot. Patrick showed an option in his JBOD SAS Expander chassis v2.
• SAS expander chips are sometimes built in to the disk backplane. Many Supermicro chassis are available with either standard SAS backplanes or SAS expander backplanes, and you can replace one with the other if your budget allows. Wiring an expander backplane is easy – SFF-8087 cable from the backplane to yet another SFF-8087 to SFF-8088 PCI adapter bracket.