QCT QuantaMesh T3048-LY9 Review – Our 10Gbase-T lab switch

For the STH Sunnyvale colo lab, we purchased the Quanta/ QCT QuantaMesh T3048-LY9. After about eight months with the switch, it is finally time for our review. Unfortunately due to timeline constraints we could not open the switch up as we normally do with review units. We have had the time to put the switch though its paces under actual workloads. The switch is certainly a bit higher-end than some of the offerings we have tested recently as it has a street price in $8000 range. What we can say about this switch is that it has performed flawlessly with the 40GbE and 10Gbase-T servers and switches we have utilized in our lab.

The QCT QuantaMesh T3048-LY9

The main feature of the QCT QuantaMesh T3048-LY9 are the I/O ports. There are 48x 10Gbase-T ports and 6x QSFP+ 40GbE ports. Based on the Broadcom Trident II chipset the switch promises 1.44Tbps of switching capacity, 1071Mpps forwarding rate. If you are looking at 48 port 10Gb switches, this is a generation newer than the 48x 10Gb plus 4x 40Gb configurations we see with Trident I based switches.

There is a USB port for adding updates or other OSes via OINE. These switches have an option to purchase as a BMS (bare metal switch). That lowers the switch purchase price and allows one to use switch software from SDN vendors such as Cumulus Networks. One can also choose between a Freescale CPU or an Intel Rangeley CPU board. At this point, we likely would have gone with the Rangeley and BMS option so if you are looking for a similar switch, that combination should be on your radar. In the event you were wondering how this looks in-rack, we did take a picture during initial installation, when you could still see the switch in the rack. It is now populated with cables but here is the older picture:

There is a standard RJ-45 serial console port but also a RJ-45 OOB management port. For those that build specific OOB management networks, one can add a dedicated interface on the port into that infrastructure. We rely on this feature for the STH data center lab.

As with other higher-end TOR switches there are a number of power and cooling options. One can have front-to-back airflow, back-to-front airflow (what we purchased) and either AC or DC power (we use 208V AC.) The net impact is that one can keep airflow in the rack moving the proper direction. The trade-off is that one does need to specify these options at the time of ordering. The redundant PSUs and the fans are all hot swappable. There are also two QSFP+ ports at the rear of the switch for uplink ports. We also have the QCT QuantaMesh T3048-LY8, SFP+ version of this switch and that has all front mounted QSFP+ ports. The RJ-45 ports on the front of the unit seem to dictate that two of the QSFP+ ports had to move to the rear on the QCT T3048-LY9. We have been using the rear uplink ports since day 1 with the switches.

The QCT QuantaMesh T3048-LY9 also accepts a wide variety of QSFP+ cables. We have tried both Mellanox DACs and Avago lamps and they have worked in the QSFP+ slots. Compared to QSFP+ pricing from other vendors this can be a large savings. For example, after discounts we were quoted ~$450 for a 3m QSFP+ DAC from another vendor. The prices pre-discount were in the four figure range. We instead used $75 new Mellanox cables saving $375 per cable or over $2200 on the set of 6 cables. On a switch that costs between $7000-8000 and under $6000 for the BMS version, that is a huge savings. We did have to buy rails for the switch but these were around $100 which is reasonable.

Management

The switch has a dedicated management (Ethernet) interface along with a standard serial console port. This is much like the popular trend of adding dedicated management interfaces to servers. We took a few screen grabs of the management interfaces with the switch. The traditional way to manage the switch is via a console port. One can also get to a SSH console over the dedicated management IP address.

The CLI is fairly standard in terms of syntax for configuring other brands of switches. The manual’s 1000+ pages of reference materials is excellent for those times when you need to look up a how to. Unfortunately, there is nowhere near the amount of online documentation on commands as one would find with a Cisco, Juniper, Dell or HP switch. Having a well documented manual is essential for those getting into the CLI side. If you are looking for how-to guides and videos, the breadth of options for other offering is staggering in comparison.

One item we wish this switch had was a web GUI much like the QCT QuantaMesh T1048-LY4A we purchased at the same time. The lack of a web GUI limits the potential of this switch to be used in the SMB segment. There is a $1800 or so street price premium for the QNOS switch software that QCT provides over the bare metal switch (BMS) models. For that much of a differential, we would have liked to have seen a web GUI like we saw on other QCT offerings.

One area where the Quanta OS on these switches stands out is in terms of included features. We got the Advanced Layer 3 package on our switch, and we understand that as of August/ September 2015 this was standard on all of the switches. Here is a quick look at the software. [Update September 2016: we were asked by QCT to update this slide with the latest revision to reflect the latest Quanta OS feature set]

With the addition of the Advanced package on the switches included by default, this is a fairly nice feature set that does not require additional licensing.

Uptime thus far

So far our switch has been rock solid. We do add/ remove gear weekly but this is the latest up time chart:

As you can see, it was just turning over to 139 days of up time. The last time it was down we flashed new firmware on the device after moving it to a different rack.

Power Consumption

Specs place the power consumption at around 63W. We have been using the switch in our data center rack with per-outlet metered APC PDUs and 208V electrical. As a result, we are able to observe the actual power consumption of the switch over some time. This switch handles our 10Gb and some 40Gb traffic in the Sunnyvale, California colocation lab. As such, it is not being hammered with a “hero” load. Here is what we are seeing in terms of max and typical power consumption with all ports active:

• Idle: 179w
• STH lab average: 197w
• STH lab maximum: 276w

The QCT T3048-LY9 is rated at up to 330w. We have not seen the switch hit higher than 276w. We are using relatively short cable lengths and our workloads generally involve many connected but mostly idle ports. If you compare this power consumption to other Broadcom Trident II switches this is about what we would expect. If you were to compare these to older generation switches, there are significant power savings. Our power consumption is similar to the 24-port 10Gbase-T Dell PowerConnect 8132 which has half of the 10Gbase-T ports and no QSFP+ uplink ports.

Performance

The QCT QuantaMesh T3048-LY9 We did a quick burn-in test when we had a chance just to see if we could push close to the rated bandwidth across each port. We recently had a short period where we were able to use the entire lab and fill all of the switches with I/O to generate some figures for various switches we use. We used iperf3 scripted across 29 nodes to get some idea of the switches performance.

Given, we did not spend hours tuning results, this is a fairly awesome chart showing over 630Gbps in aggregate bandwidth across the switch. Total setup was under 15 minutes but one can clearly see that we were getting solid bandwidth across the switch.

Conclusion

Overall this switch has served our needs well over the past few months. Digging in manuals for commands is not overly difficult as the documentation from QCT is extensive. If you are looking for a switch to run Cumulus Networks or other SDN software on, this is a great switch although we would suggest the BMS with Rangeley variants. Power consumption was great. In terms of performance we were able to get a lot of performance without much tuning. Our only concern with this switch is the lack of a web management GUI will likely preclude SMB operations from using this switch. Switch CLI is more difficult than modern switch web interfaces for the inexperienced network admin. This is clearly a top of rack switch focused on larger data center operators. We know we have seen large cloud vendors use the QCT switch line with their own custom software. As a result, the SMB market and those companies without an experienced network admin on staff are not really the target of the switch.

The big selling point beyond the technical features of the switch is price. At under $8,000 this is still one of the more affordable 48 port 10Gbase-T switches with 6x QSFP+ ports on the market. If you are looking for the BMS option and instead want to use SDN software, the price is under $6,000. We were quoted by four vendors when buying switches for the lab and the QCT switches were less expensive by a large margin. In addition, the fact that the QSFP+ ports appear to not be vendor locked can save thousands more. If you were looking to build a web hosting infrastructure for 10Gbase-T nodes (e.g. popular Xeon D nodes STH has reviewed), then this switch should be high on your shopping list.