The Powered JUPITER Exascale Supercomputer has started its installation with the Modular data center. For those who do not know, the JUPITER Exascale system will be unique in that it will be a European Exascale supercomputer using all NVIDIA CPUs and GPUs.
JUPITER Exascale Supercomputer Starts Installation
This morning, the Forschungszentrum Jülich account showed off some of the first pictures of the installation on X:
JUPITER kommt!
Die ersten Container für Europas neuen Exascale-Supercomputer wurden von @Evidenlive geliefert und auf der Bodenplatte installiert. Wir freuen uns, dass dieses Riesenprojekt, geleitet von @EuroHpc, nun immer mehr Gestalt annimmt. https://t.co/h2Dh1NqZdL pic.twitter.com/jxsUKASJAF— Forschungszentrum Jülich | @fzj@social.fz-juelich (@fz_juelich) September 11, 2024
When complete, the system is expected to house around 125 BullSequana XH3000 racks in 50 containers. The organizations involved have said that the finished system will be approximately 24,000 NVIDIA GH200 Superchips. At STH, we just reviewed the Supermicro ARS-111GL-NHR NVIDIA GH200 Grace Hopper 1U Server and did a big A Quick Introduction to the NVIDIA GH200 aka Grace Hopper piece. If you want to see the NVIDIA GH200 in action, those are good places to start.
For the quick summary, the NVIDIA Grace Hopper combines 72 Arm Neoverse V2 cores, plus the CPU’s memory and connects that using a high-speed NVLink-C2C interconnect to a Hopper GPU. That Hopper GPU can come in two flavors either a 94GB/ 96GB variant or a 141GB/ 144GB variant. On the memory side, the lower memory capacity SKUs can offer about twice the bandwidth of a modern Intel EPYC or AMD EPYC CPU (excluding the impending new CPU launches.)
Beyond just the NVIDIA GH200, JUPITER will use NVIDIA Quantum-2 InfiniBand networking. The test system, JEDI, that topped the Green500 list is said to be using quad-rail NVIDIA NDR200 InfiniBand.
Final Words
While 24,000 GPUs is big, it is also much smaller than modern AI clusters that use a 4:1 GPU to CPU ratio. Supermicro is now able to install ~120 liquid-cooled 64 GPU racks per day for those AI systems, but the Eviden systems used in JUPITER have more liquid cooling work done to them than the cookie cutter AI liquid cooled installations with 64x H100/ H200 GPUs and 16x x86 CPUs per rack. HPC systems typically have higher CPU to GPU ratios than the AI clusters, but there is quite a lot of variance in architectures on the HPC side.
The European Exascale folks have been focused on Arm-based supercomputers for some time. When this goes live, it will be a big win for Arm in the HPC space. NVIDIA has a clear plan for the GH200 to GB200 and beyond. It will be interesting to see if AMD, and at some point Intel, also follow suit with coupled CPU and GPU offerings beyond the MI300A.
@Cliff, which version of GH200 will be used?
Nothing says “European Exascale supercomputer” quite like Nvidia designed chips made in Taiwan.
Arm! It’s a UK based company that’s a subsidiary of a Delaware LLC (US but that is owned by SoftBank) that’s listed on the NYSE. So it’s an American chip built in Taiwan with some IP from an UK outfit whose ownership is Japanese/ NYSE.
But the systems are made by Eviden