Qotom Q20332G9-S10 Internal Hardware Overview
This is how our first C3758 system arrived. One can see a fairly standard layout. One other item we wanted to point out is how big the SFP+ cages are. Many ask why more motherboards do not come with SFP+ networking, and this is a big reason. The cages are much larger.
We expected our system to come with off-brand memory. Instead, we got a 16GB Kingston DDR4 SODIMM. This was not an ECC SODIMM, but technically the C3758 even supports ECC RDIMMs so we were hoping to see an ECC SODIMM here.
The SSD was also really interesting. This is a Phison 512GB NVMe SSD.
The really interesting part is that it is coated. This type of coating is done in the embedded space to protect SSDs from both accidental damage, as well as from prying NAND chips off. Usually, in these systems, we get a simple off-brand SSD. Here, we get something that looks a lot more like what we would expect if we bought this as a firewall appliance from a vendor.
Once we have the components out, we can then see that the motherboard has a lot going on.
First, we get two DDR4 SODIMM slots.
We added 64GB of memory as one can see in this screenshot. We took screenshots in Windows since we figured that that would be the one we were least confident in. Still, at this point, the hardware is very well supported.
In the middle area, there are three expansion slots. Two are 2280 M.2 NVMe SSD slots. As we will show in the performance section, these are PCIe Gen3 x2 only. The last slot is for WiFi or cellular connectivity. This is used in conjunction with the nanoSIM slot on the front and a M.2 for 4G/5G (B key 3042/3052) modem.
Next, there are two SATA ports. Our system came with one SATA data and power cable that we used to connect a SSD mounted to the bottom cover.
The bottom cover 2.5″ SSD mounts we generally see are just holes, but not on the Qotom. Instead, there are actually rubber pads, possibly for hard drive mounting.
We found motherboard pictures and something we realized that was covered in ours is the ASPEED AST2400 BMC. We did not find iKVM IPMI functionality active on our systems. Also we do not have the AMI MegaRAC sticker or anything like that, the same kind of stickers that we found had incorrect company names in Dude this should NOT be in a Dell Switch or HPE Supercomputer. Still, that AST2400 is able to provide key features like the VGA output as well as introduce a watchdog feature.
There are a number of CPU options, and we purchased the 8-core C3758 and C3758R versions. Qotom also has a lower-cost C3558R version that you can see here.
These are now a generation old as we covered the Parker Ridge generation for 25GbE-era embedded systems in pieces such as our Supermicro A3SPI-4C-LN6PF review with the Intel Atom C5315. Still, Denverton or the Atom C3000(R) series will be in the market for some time.
Next, let us get to performance.
Thanks STH for actually doing real home-lab content. This is a nice system because it’s a real CPU with real NICs and fanless. I still want to see the ms-01 review, but I’m thinking this is a better value. You’ve got the makings of a 3 host Proxmox VE cluster with this for under $1000 and 100W. If each has that DAS you’ve even got enough to do Ceph.
Here is a question, If I have a Xeon-D 1540, is it worth replacing it with this? Currently running it as my firewall, but experiencing odd latency I can’t put my finger on with my knowledge of Opnsense.
I’m curious to know if the C3758R version 4 lane M.2 slots? The R version has 20 PCIE 3 lanes instead of the 16 PCIE 3 lanes on the C3758.
Hoping you guys can publish a review of it configured as a router or firewall. It’s no great shakes in the CPU department, nobody will buy it to compile C++ code…
Some of these Aspeed chips on Board, but no Mgmt-Software on it, very curios!
Maybe you can test some switching and routing performance of this board, would be nice.
It’s got the same CPU as the Netgate 8200 Max that’ll do 18Gbps firewall iperf3 in pfSense
Here’s the speed using the C3758R version in the Netgate 8200 Max specs:
L3 Forwarding
IPERF3 Traffic: 18.60 Gbps
IMIX Traffic: 11.76 Gbps
Firewall (10k ACLs)
IPERF3 Traffic: 18.55 Gbps
IMIX Traffic: 5.1 Gbps
IPsec VPN (AES-GCM-128 w/QAT)
IPERF3 Traffic: 3.24 Gbps
IMIX Traffic: 810 Mbps
That’s why you want QA.
These aren’t new processors. They’re used in so many branded firewalls and other appliances I’m shocked there’s questions on the network perf.
7 year old Atom C3758 = FAIL
Can that ancient 14nm chip route even one 10Gbps connection let alone **four**?
@Patrick – can you confirm if the system actually supports ECC?
although the “barebone” 18w idle is high, and 35w with 3 SSDs suggest some issues… perhaps some bad support in power saving features by the bios? …. unless of course the issue is with drives used – can you perhaps re-test this with some “known-to-be power-saving” drives?
(I’m interested, since I’m in search of possibilities for low idle power home server)
Uh, Carl, your question was literally answered by AldiK in the comment before yours.
It’s a cool system. I’d buy 2 barebones and use them for silent HA cluster.
Stanos I looked at their Supermicro A2SDi-8C+-HLN4F motherboard review with the same chip and it’s really close 19.1W and 32.5W https://www.servethehome.com/supermicro-a2sdi-8c_-hln4f-review/4/
Perhaps that’s what the C3758 uses?
@Patrick did you test the the SFP+ ports with any linux distribution using kernel 6.*, including Proxmox? They work in FreeBSD based systems (*sense).
I have a similar system and the intel x553 SFP+ does not work in Proxmox 8.* reported in Bug 5103 – https://bugzilla.proxmox.com/show_bug.cgi?id=5103
Whether the networking capabilities of an Atom-based firewall appliance are well known or not, I’d prefer that a technical review of such an appliance verify the expected performance and include the results in the report. For that matter, how fast it takes to compile a Linux kernel seems as relevant as using the firewall appliance to warm a cup of coffee. Even so, I found the review and hardware interesting. Thanks!
Would it be possible to introduce a testing methodology for firewalls and networking?
10 Gb ports driven by Intel X553. Last I looked Linux kernel 6.xx did not support the X553. Has this changed? If not this seems to be a rather severe limitation of the unit for any of a variety of new releases. At first glance this looks like a good candidate for an inexpensive experimental Ceph network. But if the 10G ports cannot be used . . .
If that had something like an N305… i’d be all over it.
I was also excited to use these systems as routers. Unfortunately, 4 of them failed over years, after approx 1 years of 24×7 use consistently.
I am still in favor of fanless systems, but am staying away from Qotom specifically as unreliable.
Unless you want to have a spare server fully configured on standby ready to replace a failed system, those are not reliable.
As a former Netgate staffer I found the 8200 price and rack requirement off-putting.
This week as I needed to purchase a 10G firewall for an event kit I considered the 8200 (despite the experiences I had at the company) and I absolutely could not justify the $1300 price tag under any circumstances and purchased this specific firewall on Monday for the kit.
I am looking forward to getting it on my test bench in a week or two.
Netgate needs to up their game – the 8200 price tag is way too high, but then again all their hardware prices are very high.
Does anyone know if this will run VMware?
@Curious: I don’t see a reason it could not run it. It might not be the current release, though… so you should check the VMware HCL against the known parts of this system to be sure.
buckm – the C3758R is x2 bifurcation as well, so you do not get more lanes.
SteveP – Just ran iperf3 between the C3758 and C3758R both running Proxmox VE 8.1.3 and they are passing traffic at 9-10Gbps. See https://forums.servethehome.com/index.php?threads/the-everything-fanless-home-server-firewall-router-and-nas-appliance.42718/post-406041
This would be an interesting router if it had working IPMI. I just can’t bother with a server that does not have IPMI. Not worth the time and effort to hook up a monitor, keyboard, etc. when something goes wrong and I need the console.
This isn’t bad, but as others have pointed out, the lack of iKVM isn’t particularly great.
Furthermore, that 3052 m.2 slot underneath the SSDs is a horrible idea. High-end 5G modules can easily dissipate enough heat to require a heatsink and having to install them underneath the otehr m.2 drives makes that impossible.
Hey, would the USB-c allow a USB-c DAS?
Such as https://www.qnap.com/en/product/tl-d800c
Great box – sad the QTOMO currently does not ship to Germany. Probably some misconfiguration in their Aliexpress store.
@George Salambassi – there is no USB-C on this device
Sadly I cant order this unit in germany
@George Salambassi – oh I see the SS+ on the front now… ehh. Probably?
Patrick – It is good news about Proxmox (8.1.3 maybe typo?). This is using the 6.5 kernel. Ubuntu LTS is still using 6.2, which at least originally had the problem. It looks like the X553 problem must have gotten fixed in the newer kernel. I don’t know if Canonical has backported the fix since I have avoided the X553 for this reason. I appreciate your having done the test.
Can you do a test where you put all 5 2.5g ports into a l2 bridge and then test this as if it was a switch?
I’m sure it can’t do full line rate on all ports, but how much can it do?
If I had this, I’d use it as a firewall and seriously consider just connecting 3 or 4 devices directly to it and not bothering with a separate switch. With 1 wan link, one access point, a 10g nas and 3 wired hosts there would still be one free port.
ECC works?
The motherboard has 2 SATA ports but only one SATA SSD was connected for the article. Is there internal mounting space for a second SATA SSD?
@Tim – I just got mine and, no. Just the single 2.5 screw holes but you could certainly bodge it for SSDs.
@Robert, true, the lack of IPMI or iKVM is not great, but at IPMI cost 5-9W (depending on implementation), but a console port is also not bad, you can do connect it to a RPi Zero W and have remote administration.
@Robert, missed the AST2400, then it is strange that they have not added the IPMI features, maybe a bios bug.
Also wondering if ECC SODIMMs are confirmed working, would be ideal for a proxmox host with ZFS
It also doesn’t work with alpine linux 3.19 no igxb (pretends to be there but no link ever)
ECC so dimms work, but the 10gbps thing is a MESS. I can’t reccomend these if 10gbps is your desire. I tried three different brands of twinax cables sfp+ that work fine with my ciscos, and other (brocade etc) they REFUSE to connect to ‘switches’ which include a zxytel XGS1210-12 AND an aruba 1930 24G Class4 PoE 4SFP/SFP+ 195W Switch JL683A.
The really weird thing is if I ‘cross over’ the sfp+ cable to its two interfaces it comes up and works fine. It just won’t connect to anything else 10gbps that I have with multiple twinax sfp+ cables (including the expensive cisco ones)
So if you want this for 10gbps its a bit of wash. Its kind of sad that the reviewers didn’t do a better job of explaining how tempermental the sfp+ ports are. I read TONS of people having problems with this (the authors claim to not have had any problems) myself included (the problems part).
I even used the ‘same’ version of proxmox and couldn’t really ever get things working. I might try freebsd just to see if its working. Why can’t they tell us what SFP+ modules work with the 2#$$@#%#$^%#$% thing? Or even better yet which twinax cables actually work?
Mostly its a nice PC, just don’t count on 10gbps working unless you are willing to spend tons of time on it. I was sad, since I wanted it for a nas, but its kind of useless as that. It will make a nice vm box, I just will have to live without the 10gbps until someone kindly shows me the right cable.
So if you want the 10gbps modules to talk to themselves, the 10gps works great, otherwise IMHO 10gbps is not a reason to buy this box. I tried MULTIPLE operating systems, and multiple cables with no ability to get this working.
oh and they take an eon to reboot.
I just installed truenas core and it seems the BSD drivers dont’ hate the thing. The linux ones are a disaster so alas all things linux are out unless you are a scholar and a gentleperson.
Its working with my 10gbps switch (the aruba) and the cisco twinax cable. Seems to be a dorky linux driver than. Sadness.
Final update: it seems like twin Ax is a very bad way to use 10G. This thing seems to prefer Intel SFP plus cards. They’re not too expensive on eBay. I could not recommend using TwinaX when you have multi-vendor switches.
Thomas Munn: FS.com sells DACs where they allow you to pick the vendor code for each end.
On an unrelated note, my unit came with a 12V 7.5A PSU from Channel Well Technology. CWT has designed premium ATX PSUs for consumer brands like Corsair in the past, so I’d expect it to be pretty decent, not the type of generic PSUs we often see included with these mini-PCs. It’s also UL rated.
Dear,
what is de max amount of Memmory and storage for the Everything Home Server.
Does the Everything Home Server support ProxMox or VMWare ESXi
We need for small environments a small server solution.
Many thanks and best regards,
Wim Kobben, Erasmus IT Services
Tried multiple 10GTek SFP+ and SFP28 DACs into a Mikrotik CRS310-8G+2S+IN, CRS326-24G-2S+RM and Zyxel XS1930-10 and no dice. I think it’s probably the Linux drivers, but if you can’t run newer Linux Kernels it’s pretty worthless IMO.
Some updates on my continued travails with this box: I was running it in my server closet (which does admittedly get quite hot) and the passive cooling fails massively. The system completely froze and wouldn’t restart until I unplugged it and let it ‘cool down’. Seems OK in ambient nice 23C 70’sF but in anything approaching 95F (about 30 something C) it sadly dies. To answer the other person’s question 64GB max, with ECC. I simply cannot recommend this system based upon my own testing and attempts to make it usable in a reliable fashion. The ‘fix’ is to manually updated the drivers for proxmox (which require using DKMS and all kinds of unpleasant kernel things) and ‘hopefully’ it works. You also need to use the INTEL drivers (from intel’s out of tree distro) since some genius in the ‘in tree’ kernel COMPLETELY removed the autonegotiation function of the driver!!!!
Final say on this: I got all my fancy SFP+’s only to have the unit refuse to boot (aka power on) anymore. I am sending it back to amazon. STAY AWAY.
The C3758R version sold out quick! Luckily, I got my order in. This one’s going to become my new gateway! I don’t think this one will be a heavy lifter, as the atom only has E-cores, but it’ll be nice to offload some light tasks like pihole and maybe Home Assistant.
I suggest enabling the s3 – power up, after power loss
And regarding colsole
I suggest setting it to VT100, 115200 and the same with opnsense.
Then a simple RJ-45 cisco console cable to usb-c will work in the bios.
regards
Svemd
@Svend Gundestrup Did you have to do anything special to get the console port working? I don’t get any output from it with a Cisco console cable using minicom, but it’s possible I’m doing something wrong. :)
@Svend Same question – I’m using `minicom –device /dev/ttyUSB0 -b 115200 -t vt100` and the console is dead preboot out of the box (i only pulled out the drive and that’s it).
I just picked up the C3758 (non-R) and a Noctua 5V USB powered fan to keep it cool. Picked up some large rubber band to keep it attached to the top grill.
I’m currently planning on installing OpenSense on it. I was going to go with Pfsense, but decided at the last min to change course.
I’m going to use all the ports on day one once I set it up. I’m also planning on getting the Minisforum MX-01 to have both of these units next to each other, connected together via the SFP+ ports.
After reading everything here and what i’ve seen written around online, it appears going with an Intel branded SFP+ adapter makes the most sense. Any suggestions beyond that? Any other alternatives?
So I’m not sure what all the handwringing about SFP+ DACs is. I’ve just loaded Proxmox 8.1.4 on my new rackmount unit and I’ve tested it in my company’s lab with a 10Gtek CAB-10SFP-P2M (my go-to cable for mixed-vendor applications). I’ve tested it plugged into our Mikrotik CRS326-24G-2S+ (first pic) and Unifi USW-PRO-48-POE (second pic), iperfing to our TrueNAS scale server (single Intel X722+same DAC, also plugged into the Pro-48). https://imgur.com/a/X90QcUX
It’s perfectly fine. No boot hangups either with the DAC plugged in.
I’m going to redact the above comment for reasons that should be obvious when you look at the iperf results ♂️ More testing required, will be back.
Alright so I’m back. Even after trying a bunch of different options in Proxmox 8.1.4 to enable the X553s (see this thread: https://www.servethehome.com/intel-x553-networking-and-proxmox-ve-8-1-3/), nothing worked. So I nuked it and went back to 7.4X (completely stock) and wouldn’t you know it, it works like it’s supposed to now. https://imgur.com/a/xnySI4N This leaves no question in my mind about the DAC or switches being the problem – the new driver tree for 8.X’s kernel is absolutely the culprit here.
Got mine today. Loaded Proxmox 8.1.4 in UEFI boot (systemd-boot) and updated to kernel 6.5.11-7pve. Can confirm the X553s will NOT link. I do get a syslog note saying: ixgbe 0000:0x:00.0 eno#: detected SFP+: #, but no link/communication.
The QNAP is sold out EVERYWHERE! I called the manufacture and they said their was no lead up time on the box. dude, SO many people are getting this. any alternatives that use that SFF cable?
I absolutely love my Terra master is a bit pricier but it works!
message to everyone who wanted to buy :
i live in china and they had some models are back in stock, receiving mine soon :)
try to msg them on a aliexpress to get yours
version C3758R
For the folks who have purchased this: I don’t often use SFP+ slots but will for this unit, I’ll be using this as an NG Firewall(Untangle), and need the SFP+ ports for copper 10Gbe. Can anyone recommend a compatible module?
I purchased this to find out there is no “Power on after Power Loss” feature. This makes it a nonstarter and I’ll be returning :-(
Christopher H there is the power on after AC loss feature. It’s like S3 resume state or something like that. It is in the BIOS it is just called something obscure. Some of the AliExpress listings show how to do it if you scroll through.
@AldiK
Thank you! I found the (poorly labeled) “Power on After Power Failure” feature saving me a return. I have my instance of Untangle up and running. I ended up using another Hyper-V as Untangle still doesn’t support I266 NIC’s. I know others are using Proxmox but I don’t have time to learn a new platform and futz with its nuances to get it working right now. Hyper-V just works out of the box and won’t cause issues with updates :-)
Can this box run Mikrotik OS? Can someone test if it works?
Has anyone run Sophos FW Home on this machine? I understand that the 2.5Gb ports will not work but if the BIOS is supported by Sophos, I would think the 10Gb ports will work.
Guys. Good night.
I have one of those. I installed PfSense directly.
The pfense recognizes the DAC cable, but the speed does not exceed 1.7 gbps.
Could you give me help with what’s going on?
What others have said about 6.x Linux not supporting the X533 is completely correct. No errors in dmesg, only the link never comes up. I have had to make do with running Alpine 3.17, which has a 5.x LTS kernel, and that is working, but of course will go end of life sooner than later.
Patrick, I’m curious what you think about this device:
https://www.aliexpress.us/item/3256807251648164.html
It has the same (or similar looking) 4×2.5GBE ports, 2xSFP cages but fully spec’d out it has an Intel Core i7-1265u, 32GB of DDR4 and a 512GB NVMe ssd.
The only thing I see it lacking is the SFF8087 port for the JBOD but I suspect that a USB-C JBOD enclosure would suffice.
Let me know what you think!
Would be nice to see it running Router OS from Mikrotik
Apparently, the X553 SFP+ should now also work on PVE8 with 6.x kernel: https://bugzilla.proxmox.com/show_bug.cgi?id=5103 (bug marked as fixed at the end of August).
Did anybody keep this unit and would be able to test that? Maybe Patrick?
UPDATE: I have also had success with the latest version of Proxmox. Auto-neg still does not work, but if you set it on the switch then it works just fine.
“`shell
# uname -a
Linux pve-007 6.8.12-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-2 (2024-09-05T10:03Z) x86_64 GNU/Linux
“`
“`shell
ethtool eno1
Settings for eno1:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: off
Port: FIBRE
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
“`
I saw there is a new version with the older C3808, is it the best one?