ASUS ESC4000A-E11 Internal Hardware Overview
Next, we are going to look inside the system. We are going to work from front to rear, already having covered the frontmost portion with the drives and front expansion. We will note that ASUS uses an excellent hard and transparent plastic airflow guide that is better than many in the industry. The company also keeps the area for this airflow guide clear, making it very easy to remove or install.
One of the most notable features here is cooling. Inside the system, we get an array of fans. Three fans cool the CPU, power supplies, memory, M.2, center expansion slots, and the rest of the motherboard. On either side, we have two fans that cool the main GPUs, FPGAs, or other cards. All of these fans are hot-swappable. Something different is that the connector is in the center hub of the fan.
Behind the fan partition, there is the main PCIe and power distribution area. One can see the PCIe cables leading to the front of the chassis. power is distributed to the system and the GPUs via the side edge connectors. Under the sea of black cables, one might be able to spot the M.2 connector close to the fans. There is a single onboard M.2 slot capable of up to M.2 22110 size drives.
Unlike the ASUS RS720A-E11-RS24U we reviewed, this system is a single-socket-only platform. Inside we have a single AMD EPYC 7003 “Milan” or EPYC 7002 “Rome” series CPU and SP3 socket. This system can handle up to 280W TDP CPUs thanks to the generous cooling.
The memory situation is interesting. There are a total of 8x DDR4-3200 slots. Our system came installed with two 32GB DIMMs for 64GB, but we would deploy it with 256GB or more of memory and all eight memory channels filled. Luckily we have an extensive spare parts bin.
Behind the CPU is the dual low-profile x16 riser. On one side, we can see the small heatsink for the Intel i350-am2 NIC.
On the other side we can see a microSD card slot and an ASPEED AST2600 BMC.
The riser slot itself is interesting. ASUS uses two special riser slots in a row. Each is PCIe Gen4 x16 in order to provide connectivity.
The star of the platform, however, are the two risers on either side of the platform. ASUS has a few different configuration options, but many will use this either as an 8x PCIe Gen4 x8 or 4x PCIe Gen4 x16 platform. Half of those slots are on either side. In both of our risers, we have an AMD Instinct MI200 installed, but one can see the four PCIe slots on each riser.
The AMD Instinct MI210 is the lower-power PCIe version of the MI200 family. This is the same GPU family that powers the AMD EPYC and Instinct-based Frontier supercomputer. The Fontier GPUs are OAM form factors so they can be much higher power, but the MI210 is the PCIe variant.
Here are the key specs for the MI210 PCIe GPUs that we have.
| Model | MI210 PCIe |
|---|---|
| Compute Units | 104 CU |
| Stream Processors | 6,656 |
| Peak FP64/FP32 Matrix | 45.3 TF |
| Peak FP64/FP32 Vector | 22.6 TF |
| Peak FP16/BF16 | 181.0 TF |
| Memory Size | 64GB HBM2e |
| Memory Clock | 1.6GHz |
| Memory Bandwidth | Up to 1.6 TB/sec |
| Bus Interface | PCIe Gen4 |
| Infinity Fabric Links | Up to 3 |
| Max Power | 300W TDP |
One would be correct in noting that how the system shipped to us with 2x MI210’s but also 2x 32GB DDR4 DIMMs means that we had twice the GPU memory as we had CPU memory in this system.
Next, let us get to management.
What is the SD card slot for?
@Matt H.
That’s for the Hypervisors
It’s a bit unusual to see last-gen servers getting reviewed. Is there a concrete reason why the successor to this model, the E12, wasn’t offered for review? It is already available and comes with Epyc Genoa support.
Hm … Nvidia H100 is PCIe gen 5 – what about config with actual AMD CPU which supports the gen 5 I/O ?
Tomas – We have a Gen5 AMD EPYC Genoa server with video going live in a few hours.
It’s a fantastic server, but you have to know what”s missing too:
1. There is no PCIe bifurcation option. You can install only one M.2 ssd on board or 4 M.2 ssd if you choose to not have a RAID controller.
2. You can”t do RAID on default configuration, there is nothing onboard to do that. You have to install a RAID controller.
3. There is no SATA M.2 on board.
4. The SDcard is used for BMC log only. You have to to some non supported cobfigurations to boot from SD card.
5. There is no USB port inside the server.
6. You can”t do RAID with U.2 disks.
7. If you install a RAID controller, you have only 2xU.2 disks.