Ubiquiti EdgeRouter ER-12P 12-port Router Review

The Ubiquiti EdgeRouter 12P (ER-12P) is a big brother of the Ubiquiti ER-6P device we reviewed earlier. This unit is based on the same platform but offers an additional onboard Ethernet switch chip. We wanted to take a look at what Ubiquiti is doing with this device and whether or not we are getting better value as a result.

Ubiquiti EdgeRouter ER-12P Hardware Overview

The device measures 268.1 x 136.5 x 31.1 mm (10.55 x 5.37 x 1.22″), 700 g (1.54 lb), and has a maximum power consumption of 20W (not including PoE). Ubiquiti went with an all-metal, clean, and minimalistic design, consistent with the rest of the ER- line. The ER-12P is a fanless unit. To help with cooling, most of the surfaces of this device are mesh. On the front of the device, we find ten (10) RJ-45 Gigabit Ethernet ports with support for 24V 2-pair/ 24V 4-pair passive PoE output, two (2) SFP port, serial console, reset button, a USB 3.0 port is reserved for future use, and status LEDs for device and ports.

Please note that “Passive” PoE is not part of 802.3af or 802.3at standards. While the ER-12P will work with most Ubiquiti PoE devices, we encourage users to read the ER-12P QSG to get a better idea of the power budget, requirements, and limitations for PoE devices connected to ER-12P.

At the back, one can see a 24V DC input and ground point.

At the bottom of the device, we find integrated keyholes for wall mounting and (4) rubber feet.

The ER-12P can be rack-mounted with an optional kit. And at $15 the price is reasonable, but it would have been nice if Ubiquiti included this.

Just like the EdgeRouter 6P, the 12P version is based on a Marvell (Cavium) CN7130 SoC containing a 1Ghz MIPS64 quad-core CPU. It has a variety of connectivity options including a packet processing engine, QSGMII, RXAUI, SATA3.0, USB 3.0, and PCIe Gen 2, although not all of these interfaces are utilized by ER-12P. This Marvell platform has started on the carrier-grade market and as time passed, made its way to consumer and SOHO markets.

The Marvell CN7130 SoC in this device is coupled with 1Gb of DDR3 RAM and 4GB eMMC storage.

Unlike the ER-6P, the ER-12P version has a switch chip connected to the first eight Ethernet ports (LAN). Ubiquiti does not disclose technical details in the datasheet, but according to reports from the community, it is a Qualcomm Atheros QCA8511 or similar. The advantages of such a design include low latency and line-rate throughput between clients on LAN ports configured for L2/L2+ switching. On the other hand, combined throughput for clients connected to LAN port configured for L3 operation will be limited by the interface between switch chip and SoC. While schematics and/or block diagrams are not publicly available, we would be surprised if Ubiquiti does not utilize one of the SerDes/QSGMII blocks (up to 5Gbit/s). It is also worth mentioning that hardware offloading for IPv4 forwarding is not available (or does not work) for communication between LAN ports. At this point it’s not clear if this is a hardware limitation or an issue with firmware, there is nothing in the release notes that specifically calls out the issue with LAN ports, but it does mention “[Offloading] – VLAN traffic is not being offloaded on ER-12,” which may be related.

EdgeRouter ER-12P Management

EdgeOS is the default firmware for EdgeRouter X, which we briefly covered in the EdgeRouter X piece. At the time of this review, the latest release of 2.x version of EdgeOS (2.0.9-hotfix.1) has improved IPv4 forwarding performance with and without HW offloading. Taking into account that the 1.x version has not been updated for almost a year and may miss critical security fixes, we believe it’s time to switch to 2.x. The UI for 2.x firmware remains almost the same with some cosmetic changes, like the addition of a button to check for firmware updates in the top right corner, or a button to perform a factory reset in the system menu.

The CLI also remains the same; we can get access to a packet processor engine to get information about the status of various modules, per-packet and per-flow statistics, and the ability to manipulate parameters. We can also turn on/off these modules and adjust the size of the lookup tables.

Next, let us look at the performance.

Performance evaluation

Test Bench Setup

Our testing bench is based on a Cisco T-Rex project which in turn is based on the DPDK framework which we are going to cover in future articles and consists of:

You can read our Dell Precision T7920 Dual Intel Xeon Workstation Review for more on the platform itself.

T-Rex modes of operation

For our tests, we primarily use stateful (STF) and advanced stateful (ASTF) modes of operation. You can get a full overview of each mode on the T-Rex home page. In a nutshell, STF scales up to Mpps per core, and the number of flows can easily go up to 100k and above. In this mode, T-Rex emulates stateful traffic, and while it allows one to verify some devices under test (DUTs) with NAT enabled, it does not implement a TCP/IP stack, and as a result, will not handle any TCP/IP errors properly and will not work with DUTs that terminate traffic. For the same reason, we allow a rather large percent of packets to be dropped (up to 1%) by DUT, understanding that in a real-world scenario, the TCP/IP stack on DUT or client/server-side may be able to handle the error gracefully (I.E transparent for end-user). Primary metrics collected with this mode are %of packets lost and latency. We will continue to use this mode to evaluate the basic (raw) routing capacity of DUT, but for NAT/VPN/DPI we will switch to ASTF.

ASTF mode has TCP/IP implementation based on the BSD stack. While it is more demanding in terms of resources and CPU power, in addition to raw performance, it allows us to evaluate the quality of services by inspecting the number of packets that were re-transmuted, out-of-order, duplicated, etc., allowing us to review a wider range of DUTs. Last but not least ASTF comes with a Python API, which helps a lot with automation.

Using Case Driven Benchmarks

While synthetic benchmarks are good for marketing, and when used properly give a high-level overview of device potential, they do not make it easier to evaluate the performance of the device for a particular use or compare performance across devices due to different boundary conditions. Such boundary conditions may result in a significant error in the final numbers.

T-Rex gives us the freedom to define any workflow we like, or even create one based on real traffic captured from a production system. In order to see how the device under test performs in a more realistic scenario, we will use the SFR profile. This profile includes a combination of traffic templates such are:

• http_get / http_post / https
• mail-related traffic flows
• SIP
• DNS
• and etc.

Below we can find a graphical representation of SFR profile:

EdgeOS v2.x.x

There were several reasons why some users stayed on the v1 version of firmware for Cavium based on ER devices in the past. To name a few, early versions of v2 firmware had a number of stability issues with GUI and performance degradation compared to v1. Early versions of v2 firmware did include the following statement in the release notes

Known issues:
Performance - Throughput degradation by 5-10% when comparing with v1.10.9 
  firmware with older kernel

Things have changed since our ER-6P review; in the latest version (2.0.9/2.0.9-hotfix.1) of firmware, the “Throughput degradation” warning disappeared and firmware appears to be more stable. What is more important, the v1 version had no updates for over a year, and it may miss important security fixes. For these reasons, we encourage users who are still on the v1 version to re-evaluate the latest firmware.

ER-12P Routing performance (STF)

While we are not going to test the ER-12P with the v1 version of firmware for the reasons stated above, we want to give our readers some reference points to set the right expectations about potential performance degradation when switching to the v2 version of the firmware. The ER-12P is based on the same platform as the ER-6P, so we will use it as a reference point to spot-check routing performance with a multi-WAN scenario. To recap, our STF routing performance test ER-6P with firmware version v1.10.11 had virtually 0 packets lost at speeds up to 400Mbps and less than 0.003% up to an upper limit of our tests at 2.3Gbps.

ER-12p 4ports RTE Packet DropThe ER-12P with firmware version 2.0.9-hotfix.1 in the same scenario has virtually 0 packets lost up 300Mbps after 300Mbps percent of packet lost slowly raises reaching 0.8% at 2.3Gbps, which is still acceptable.

ER-12P Routing performance (ASTF)

Note: We would like to highlight that percent of packets dropped should not be compared between STF and ASTF modes.

The test is executed with the firewall disabled and hardware offloading enabled for IPv4 forwarding and VLAN. We use two (2) pairs of ports connected directly to our bench. Each pair gets its own pool of clients(255) and servers (64k). The client-side of a pair is connected, to LAN while the server-side is connected to one of the WAN ports.

In this test, we will be looking for maximum combined throughput (based on SFR profile) the DUT can handle without losing data, which for ASTF we define as a combined packet drop below 0.1%. The ER-12P has no problem handling this scenario with 0 packets lost up to ~1.2Gbps and reaching a maximum of 0.03% at 2.3Gbps

Another point we are curious about is a statement in the release notes claiming to fix the following issue:

 Routing - Added Ethernet driver patch from Cavium that fixes packet reordering 
            with 4.x kernel

For reference, a previous version of the firmware had the following statement under known issues in the release notes:

Offloading - On Cavium-based routers (ER, ER-Pro, ER-Lite, ER-PoE, ER-4, 
  ER-6P, ER-12, ER-Infinity) small percentage of packets are randomly 
  reordered. This issue was fixed in v1.10.0 firmware but it reappeared 
  since v2.0.0 because of new ethernet driver.

So let us look how many out-of-order (OOO) packets we get:

Looking at the results, I personally do not believe the issue is resolved, or that it is a small number. With that said, it is important to mention that out-of-order packets by themselves are not a problem, they just make the network stack (in case of TCP) or Application level (in case of UDP) work a little bit harder, but users with more or less recent devices are unlikely to notice anything.

ER-12P NAT performance (ASTF)

The setup described in the previous test is modified to add masquerading rules based on outgoing interface (WAN) for each pair of ports. With NAT enabled, we hit our limit of 0.1% at ~2Gbps (combined throughput).

And checking the % of OOO packets, we see that it’s still quite high.

Ubiquiti ER-12P Power Consumption

We saw an average idle power consumption for the device around  ~10w. During the test execution, power consumption was ~13w.

Final Words

Just like the ER-6P, we think the Ubiquiti EdgeRouter 12p (ER-12P) can be an excellent choice for most home and SOHO users who are not locked to a particular ecosystem. The solution is built on a hardware platform that is still in use by big names for entry-level enterprise devices. Here, the ER-12P delivers excellent results and we expect it to be able to reliably handle multi-WAN connections up to 2Gbps based on our testing.

PoE support is limited to “Passive mode” and the power budget is quite small. Last year when we reviewed the ER-6P, we praised Ubiquiti for this feature, as it would allow users to connect some low-power Access Points, cameras, and phones directly. However, as of today, it would not be sufficient to power most recent APs even within the Ubiquiti ecosystem, so the value of “Passive POE” is diminishing very fast. New WiFi 6E APs are using even more power so this is a trend we see continuing.

Special consideration should be made towards which version of the firmware to use, especially taking into account known VLAN-related issues in the latest firmware, which could hurt the ER-12P primary use case. We hope it will be resolved with the next update. We also recommend that anyone who is still using v1.10.x firmware on devices with some public services to re-evaluate v2.0.x firmware due to a number of security fixes added through the year.

Support for license-free cloud or on-premises management through UNMS could potentially turn into significant TCO savings as well. At $299 list price ($249 street price) this makes this device extremely competitive.