How Liquid Cooling is Prototyped and Tested in the CoolIT Liquid Lab Tour

While up in Calgary in Canada a few weeks ago, we had the chance to do two pieces. The first was the How Liquid Cooling Servers Works with Gigabyte and CoolIT piece. While there, something that struck me was that it would be interesting to show how CoolIT is addressing the dreaded “L-word” in liquid cooling: leaks. CoolIT has an entire Liquid Lab dedicated to prototyping and testing designs as well as doing integration and compatibility testing to ensure that solutions perform well and are reliable in the field. In this piece, we are going to show off the lab.

Video Version

This is one of those pieces that probably makes more sense to watch via the video that you can find here:

Also, if you want to see a system being assembled, you can see our How Liquid Cooling Servers Works with Gigabyte and CoolIT here:

As always, we suggest opening these in a YouTube window, tab, or app for a better viewing experience.

CoolIT Liquid Lab – Making Prototypes

The CoolIT Liquid Lab has a number of machines for making prototypes. That includes CNC machines as well as skiving machines and 3D printers. We could show the CNC machine but the other machines we could not show.

In the video, you can see the CNC machine at work. This is important because it allows for rapid prototyping of designs and then being able to put them into test platforms quickly. CoolIT also gets parts from third party vendors, but having in-house prototyping is important. These machines were busy during our visit.

CoolIT Liquid Lab – Assembling Prototypes

Once parts are machined, printed, or arrive in the lab, the next step is putting them together into larger assemblies. That is the assembly floor.

Here engineers can take various parts and ensure proper fit. There are also bins of commonly used parts to aid in the quick assembly of prototypes.

CoolIT can also do things like run initial batches of liquid as well as gas like nitrogen through loops to ensure everything is properly sealed and seated.

Something that was interesting, that also makes a lot of sense is that I saw these acrylic carriers. These solutions often need to be installed across several cold plates with precisely cut tubing to fit. As a result, these carriers ensure that the entire assemblies can be transported without putting too much stress on the tubing. This makes sense, but I had no idea prior to visiting the lab.

CoolIT has a number of different nodes, fittings, and components from different vendors.

One of them I immediately recognized from our SC21 coverage as a HPE Cray Shasta node that will power some of the US’s first exascale supercomputers (more on exascale later this week on STH.)

Once the smaller assemblies are built, the next step is often the temperature chambers.

CoolIT Liquid Lab – Temperature Chambers

In the lab, there are a number of different temperature chambers.

Each of these chambers has different capabilities.

The reason these test chambers are used is for things like simulating rapid heating and cooling cycles to ensure the parts and assemblies can handle rated loads. Often, the parts are taken beyond ratings to see failure points.

This is important because of the different environments that liquid cooling is deployed in.

Beyond that, there are non-operational environments like being shipped over air freight that need to be simulated to ensure problems do not arise during shipping. These chambers all run accelerated testing cycles to get data faster than simply putting units in the field and observing.

CoolIT Liquid Lab – Integration

The integration area is where loops, CDUs, manifolds and other components come together. Here the parts are assembled. Then the secondary cooling loop (the one that goes to the racks) can be filled with different liquids and additives. This allows testing to ensure that one additive does not create corrosion, or that there are no adverse effects using a liquid and additives along with all of the components in a loop.

There is also a container-based data center simulator to test larger environments. You can see those in the video.

Putting it Together: From Small CDUs to Megawatt Scale CDUs

While a CDU (Coolant Distribution Unit) may seem like it only has a few primary components, they have evolved. Here is a 40kW CoolIT CHx40 CDU that is an old model.

Here is a CHx200, a 200kW unit that is a more modern example. One can see just how much more is going on.

Now, CDUs are much larger, and can be in the Megawatt scale. This unit is around 1MW, but CoolIT has larger units as well.

In order to achieve this scaling, CoolIT needs to test every part, and that is why the Liquid Lab is important.

Final Words

One of the biggest questions we get on liquid cooling stories is around reliability and leaks. Hopefully this tour of the Liquid Lab helps explain what is being done to test and address those challenges. The next generation of high-performance and high-density solutions will require liquid cooling, and so this is important work.

I also just wanted to say thank you to the CoolIT folks for hosting STH. There were a few places we could not show because of the prototyping going on in a working lab, but this was a great look at a lot of the process.

For those wondering, these nodes are HEAVY!