For the STH Sunnyvale colo lab, we purchased the Quanta/ QCT QuantaMesh T3048-LY8. After about eight months with the switch, it is finally time for our review. Unfortunately due to timeline constraints we could not open the switch up as we normally do with review units. We have had the time to put the switch though its paces under actual workloads. The switch is certainly a bit higher-end than some of the offerings we have tested recently as it has a street price in $6000 range. What we can say about this switch is that it has performed flawlessly with the 40GbE and 10Gbase-T servers and switches we have utilized in our lab.
The QCT QuantaMesh T3048-LY8
The main feature of the QCT QuantaMesh T3048-LY8 are the I/O ports. There are 48x 10Gb SFP+ports and 6x QSFP+ 40GbE ports. Based on the Broadcom Trident II chip set the switch promises 1.44Tbps of switching capacity, 1071Mpps forwarding rate. If you are looking at 48 port 10Gb switches, this is a generation newer than the 48x 10Gb plus 4x 40Gb configurations we see with Trident I based switches. These days, the additional two QSFP+ 40GbE ports will be utilized as we see more gear with 40GbE ports.
There is a USB port for adding updates or other OSes via ONIE. These switches have an option to purchase as a BMS (bare metal switch). That lowers the switch purchase price and allows one to use switch software from SDN vendors such as Cumulus Networks. One can also choose between a Freescale CPU or an Intel Rangeley CPU board. At this point, we likely would have gone with the Rangeley and BMS option so if you are looking for a similar switch, that combination should be on your radar. In the event you were wondering how this looks in-rack, here is a snap when we were quickly trying the switch.
There is a standard RJ-45 serial console port but also a RJ-45 OOB management port. For those that build specific OOB management networks, one can add a dedicated interface on the port into that infrastructure. We rely on this feature for the STH data center lab.
As with other higher-end TOR switches there are a number of power and cooling options. One can have front-to-back airflow, back-to-front airflow (what we purchased) and either AC or DC power (we use 208V AC.) The net impact is that one can keep airflow in the rack moving the proper direction. The trade-off is that one does need to specify these options at the time of ordering. The redundant PSUs and the fans are all hot swappable. There are also two QSFP+ ports at the rear of the switch for uplink ports. We also have the QCT QuantaMesh T3048-LY8, SFP+ version of this switch and that has all front mounted QSFP+ ports. The RJ-45 ports on the front of the unit seem to dictate that two of the QSFP+ ports had to move to the rear on the QCT T3048-LY8. We have been using the rear uplink ports since day 1 with the switches.
Perhaps the single biggest feature of the switch is optics and the lack of vendor locking. Unlike offerings from Cisco, Juniper and others, the QCT T3048-LY8 is compatible with different SFP+ and QSFP+ modules. Without the vendor premium, cabling these systems is significantly less expensive. We are actually using Mellanox brand QSFP+ 40GbE direct attach cables in the lab. Aside from the initial purchase price being lower than the bigger brands, in practice this switch is much less expensive to implement because it can utilize cable and optics that are not vendor specific. This feature’s usefulness cannot be understated.
Management
The switch has a dedicated management (Ethernet) interface along with a standard serial console port. This is much like the popular trend of adding dedicated management interfaces to servers. We took a few screen grabs of the management interfaces with the switch. The traditional way to manage the switch is via a console port. One can also get to a SSH console over the dedicated management IP address.
The CLI is fairly standard in terms of syntax for configuring other brands of switches. The manual’s 1000+ pages of reference materials is excellent for those times when you need to look up a how to. Unfortunately, there is nowhere near the amount of online documentation on commands as one would find with a Cisco, Juniper, Dell or HP switch. Having a well documented manual is essential for those getting into the CLI side. If you are looking for how-to guides and videos, the breadth of options for other offering is staggering in comparison.
One item we wish this switch had was a web GUI much like the QCT QuantaMesh T1048-LY4A we purchased at the same time. The lack of a web GUI limits the potential of this switch to be used in the SMB segment. There is a $1800 or so street price premium for the QNOS switch software that QCT provides over the bare metal switch (BMS) models. For that much of a differential, we would have liked to have seen a web GUI like we saw on other QCT offerings.
One area where the Quanta OS on these switches stands out is in terms of included features. We got the Advanced Layer 3 package on our switch, and we understand that as of August/ September 2015 this was standard on all of the switches. Here is a quick look at the software. [Update September 2016: we were asked by QCT to update this slide with the latest revision to reflect the latest Quanta OS feature set]
With the addition of the Advanced package on the switches included by default, this is a fairly nice feature set that does not require additional licensing.
Uptime thus far
So far our switch has been rock solid. We do add/ remove gear weekly but this is the latest up time chart:
As you can see, it was just turning over to 250 days of up time. The last time it was down we were installing new metered by outlet PDUs.
Power Consumption
We have been using the switch in our data center rack with per-outlet metered APC PDUs and 208V electrical. As a result, we are able to observe the actual power consumption of the switch over some time. This switch handles our 10Gb SFP+ and some 40Gb traffic in the Sunnyvale, California colocation lab. As such, it is not being hammered with a “hero” load. Here is what we are seeing in terms of max and typical power consumption with all ports active:
- Idle: 113w
- STH lab average: 146w
- STH lab maximum: 191w
We are using relatively short DAC’s and several short-range optics in the switch but we also have a number of idle ports. We were pleasantly surprised with how low the power consumption of the system is in our racks.
The QCT QuantaMesh T3048-LY9 We did a quick burn-in test when we had a chance just to see if we could push close to the rated bandwidth across each port. We recently had a short period where we were able to use the entire lab and fill all of the switches with I/O to generate some figures for various switches we use. We used iperf3 scripted across 29 nodes to get some idea of the switches performance.
Given, we did not spend hours tuning results, this is a fairly awesome chart showing over 640Gbps in aggregate bandwidth across the switch. Total setup was under 15 minutes but one can clearly see that we were getting solid bandwidth across the switch.
Conclusion
Overall this switch has served our needs well over the past few months. Digging in manuals for commands is not overly difficult as the documentation from QCT is extensive. If you are looking for a switch to run Cumulus Networks or other SDN software on, this is a great switch although we would suggest the BMS with Rangeley variants. Power consumption was great. In terms of performance, we were able to get a lot of performance without much tuning. Our only concern with this switch is the lack of a web management GUI will likely preclude SMB operations from using this switch. Switch CLI is more difficult than modern switch web interfaces for the inexperienced network admin. This is clearly a top of rack switch focused on larger data center operators. We know we have seen large cloud vendors use the QCT switch line with their own custom software. As a result, the SMB market and those companies without an experienced network admin on staff are not really the target of the switch.
The big selling point beyond the technical features of the switch is its price. At under $8,000 this is still one of the more affordable 48 port 10Gbase-T switches with 6x QSFP+ ports on the market. If you are looking for the BMS option and instead want to use SDN software, the price is under $6,000. We were quoted by four vendors when buying switches for the lab and the QCT switches were less expensive by a large margin. In addition, the fact that the QSFP+ ports appear to not be vendor locked can save thousands more. If you were looking to build a web hosting infrastructure for 10Gbase-T nodes (e.g. popular Xeon D nodes STH has reviewed), then this switch should be high on your shopping list.
I believe its ONIE instead of OINE