They Let Bring a Camera Into a Top Classified US Supercomputer El Capitan

5

Seeing El Capitan Up Close

In the building, El Capitan is installed upstairs, not on the ground floor. There is normally a viewing window to the data hall, but I was allowed to get inside.

HPE Cray AMD El Capitan Down Window Side Aisle At LLNL 1
HPE Cray AMD El Capitan Down Window Side Aisle At LLNL 1

It is worth noting, that the Tuolumne system, #10 on the current Top500 list, is installed next to El Capitan. This system will support unclassified science.

HPE AMD Tuolumne At LLNL 1
HPE AMD Tuolumne At LLNL 1

Normally looking down the rows at El Capitan, it would look something like this. I asked about the lack of tiles at some points. Apparently the floor needed to be reinforced for the 9000lb (-ish) racks.

Down Closed Aisle Of HPE Cray AMD Liquid Cooled Racks For El Capitan At LLNL 1
Down Closed Aisle Of HPE Cray AMD Liquid Cooled Racks For El Capitan At LLNL 1

On this day, however, some of the doors were off, and one can see the system in its glory.

Down Aisle Of HPE Cray AMD Liquid Cooled Racks For El Capitan At LLNL 1
Down Aisle Of HPE Cray AMD Liquid Cooled Racks For El Capitan At LLNL 1

Or just take selfies in front of the liquid cooling racks.

Patrick With Liquid Cooled HPE Cray And AMD Rack CDU El Capitan At LLNL 1
Patrick With Liquid Cooled HPE Cray And AMD Rack CDU El Capitan At LLNL 1

Each rack has 128 compute blades that are completely liquid cooled. It was very quiet on this system with more noise coming from the storage and other systems on the floor.

Liquid Cooled HPE Cray And AMD Racks For El Capitan At LLNL 1
Liquid Cooled HPE Cray And AMD Racks For El Capitan At LLNL 1

On the other side of the racks, we have the HPE Slingshot interconnect cabled with both DACs and optics. There are also blue and red liquid cooling tubes here for the Slingshot interconnect side.

Down Networking Aisle Of HPE Cray AMD Liquid Cooled Racks For El Capitan At LLNL 1
Down Networking Aisle Of HPE Cray AMD Liquid Cooled Racks For El Capitan At LLNL 1

That Slingshot interconnect was not fully populated which was neat to see. We are going to show the Rabbit taking up space in the rear of these racks on the next page.

Behind the main system, was lots of storage. This is what the storage racks look like.

Shared Storage Racks For El Capitan At LLNL 1
Shared Storage Racks For El Capitan At LLNL 1

I was told these are mainly disk-based racks. That is a difference from what we are seeing in many AI data centers focusing on flash-based shared storage.

Next, let us get to the Rabbit.

5 COMMENTS

  1. I was a system administrator for a supercomputer center at Lockheed in the late 80s. We had a Cray X-MP, a Y-MP, a Connection Machine and several DEC VAXs computers as front end compute nodes. It was an exciting time with so much raw computational power that today can be had in a smart phone.

    Your visit to Aurora brought back lots of great memories.

  2. Some seriously nice kit, thanks for sharing.

    If you ever find yourself in Lugano, Switzerland then you could attempt a visit to CSCS to see their HPC/Cray machines

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.