MikroTik CRS520-4XS-16XQ-RM Block Diagrams
The block diagram for this switch is fascinating.
Here we can see that the Annapurna Labs AL32400 has dual 25GbE links to the Marvell 98CX8410 switch chip. We can then see the two 10Gbase-T management links also connect through the Annapurna Labs CPU not the Marvell switch chip.
We did some digging since this was hard to find on Marvell’s website, but here are some of the features of the Marvell 98CX8400 line:
Note, these are chip features, not necessarily what MikroTik has fully implemented. We also found a block diagram for the switch chip:
This is a fairly big upgrade over the lower-end Prestera chips used in some of MikroTik’s other switches.
Next, let us get to management.
MikroTik CRS520-4XS-16XQ-RM Management
Management for these switches is fairly standard. We get a CLI, a WebFig web-management solution, and WinBox desktop app. WebFig and WinBox tend to be very close to one another in the feel.
Something that is really interesting with this switch is that there is enough power to start using some of the more advanced features in the interface because it has a fairly high-performance CPU. With dual 25GbE links between CPU and switch chip, we are not limited to a maximum of 1GbE that we find on many of the other switches as well.
This is a huge feature for MikroTik. Many of the used enterprise switches that are alternatives rely on CLI only. Most in the SMB space buying 1-2 switches do not want to learn a CLI. The GUI options allow for 16-port 100GbE solutions to be deployed by non-networking people. That is a huge deal.
What will be interesting to see is how far MikroTik goes into enabling hardware offloads and features using this CPU and switch combination. It feels like MikroTik put this combination together with room to grow.
If you have not purchased a MikroTik switch in some time, be warned, they now use random management passwords. That is probably a good thing given how many people connected MikroTik devices with management interfaces to the open Internet with default passwords.
Next, let us talk performance.
MikroTik CRS520-4XS-16XQ-RM Performance
On the performance side, we saw fairly decent results. We were below 170Gbps, but this is also a harder setup for use to generate traffic for.
MikroTik has its performance results using bigger testing tools:
We thought we would at least show this for folks as well.
It feels like you could almost use this switch as a WAN router if you had a lower-speed WAN connection. Maybe that was the idea behind using such a large management processor.
Next, let us get to the power consumption and noise.
If you want to use a full 100gbe you kinda need RDMA for windows. I have a 32 port 100gbe and most windows transfers max out around 40gb. Which is fine, I’m sure this switch works great as a 40g switch as well. I also hacked 120mm noctua high pressure fans into the top lid making it very much a TOR router by necessity. So something like this would be great.
However I didn’t go with iwarp compatible nics so there is a decent bit of switch compatibility required. Will they support PFC on the feature roadmap?
I wonder why so many switches go for the 1U full width form factor. You’d think if you went with 2U half width then you could get the same port density by putting two side by side, but you’d be able to use larger, more efficient fans and heatsinks, and quieter fans as well.
Hi, did you test the switch with high power transceivers?
Does it actually support modules with higher power classes. E.g. 80km 100G ZR4 modules will use 5.5W. Some devices will refuse to operate transceivers with a higher consumption than 3.5W.
Given the low power budget left for transceivers it seems like it’s designed for use with only twin ax DAC cables. Even 16 x low power 2.5 watt SR4 transceivers would be a problem. Is there any official word from mikrotik regarding transceiver power budget?
RouterOS 7.15 seemed to have added some initial support for PFC, and 7.16 seems to be refining a bit more.
Here’s the thread on QoS:
https://forum.mikrotik.com/viewtopic.php?p=1089316#p1089316
You can see I asked about clarification on RoCE towards the bottom but haven’t received an authoritative response – only a community member stating “You don’t need DCB support, just use RoCE v2” which I don’t consider to be a great answer. Maybe if yall want to ask as well and help me bring focus?
I’m sure Patrick / STH is also asking Mikrotik, as it came up in previous videos with the 4 port 100Gb CRS 5xx switch.