One of the crazier bits in the industry is that we have been seeing news and analysts still trumpet the benefits of 2-phase immersion cooling. There is one problem. Major hyper-scalers such as Microsoft and Meta have effectively halted two-phase immersion cooling research for a simple reason: the fluids used have been found to have such toxic side effects that companies like 3M will halt production of the chemicals in 2025. A major part of that is simply because the companies creating the immersion cooling liquids are facing many multi-billion dollar lawsuits.
What is Single-Phase Versus Two-Phase Immersion Cooling?
Immersion cooling operates on a fairly simple concept. IT equipment (for our purposes usually servers) is placed into a tank with non-conductive fluid. Since liquids generally conduct heat better than air, the idea with immersion cooling is to have heat transfer from hot components like CPUs, GPUs, AI accelerators, and more, to liquid circulating in the tank. The heat is then removed from the liquid to keep the servers running cool. What is awesome about immersion cooling versus having cold plates like one would have in consumer AIO liquid coolers, is that immersion cooling removes heat from all server components, so fans are not needed.
More efficiently removing heat means lower overall power consumption. In modern servers we review, fans can use somewhere in the 15-20% of the total system power consumption. Fans are also a potential source of failure, and companies like Alibaba have found that components are more reliable when submerged and not exposed to air.
On the subject of immersion cooling, there are generally two approaches. First is single-phase cooling. With single-phase cooling, the liquid in a tank is generally circulated using pumps between the tank with servers and a heat exchanger. There are a number of designs, but usually, this heat exchange happens with facility water. The key with single-phase cooling is that the fluid is designed to stay as a liquid. That is the single phase in the name.
Two-phase immersion cooling is similar in many ways, but there is one big difference. The two phases are liquid as well as gas. As the servers heat up fluid, it does not just warm liquid. Instead, it causes the liquid to boil and become gas. That helps circulate fluid without using a pump, a major plus. The gas is then cooled via a condenser and returned to liquid form. Usually, two-phase immersion tanks need to be sealed when operating to contain vapor. Increasing reliability is important with this because servicing equipment means, waiting for cooling, breaking the seal once the vapor has become liquid again, and then opening the tank. The industry was focused on this method not just in tanks.
We also saw individual supercomputer nodes with two-phase immersion cooling. Sometimes, like in the below Sugon supercomputer, this was simply called “phase change” referring to liquid-to-gas (and gas-to-liquid) phase changes. This approach allows for individual nodes to be serviced, without the challenge of having to shut down an entire rack. This was from SC19, but even then we were not allowed to get closer photos.
Two-phase immersion cooling showed promise because it generally could support higher-TDP parts. The challenge is that many of the fluids used are now being discontinued.
Better Cooling but More PFAS
PFAS stands for per- and polyfluoroalkyl substance. These are often referred to as “forever chemicals” because of their tendency not to break down naturally. PFAS also can cause cancer and other negative impacts on life. One of the biggest fluids used in two-phase immersion cooling but also things like semiconductor manufacturing was 3M Novec.
This class of substance was the center of 3M’s Belgium plant shutdown and Belgium working to get a PFAS ban in the EU. The US EPA has also been interested in these chemicals. These chemicals have gotten enough scrutiny, that 3M has a PFAS mini-site trying to convince folks that they are not all bad. Usually, when companies make mini-sites dedicated to saying something is not that bad, we assume it probably is.
In early June 2023, 3M announced nearing a $10B settlement with US towns over pollution from PFAS and there are other lawsuits for this class of chemicals still active. There are over 4000 lawsuits filed over PFAS, and this settlement is for a consolidated set of cases. Just this week, the trial was delayed as the settlement was near.
Companies like Meta, Microsoft, Google, and others have ESG goals, and have been looking at immersion cooling to help lower data center PUE to achieve those goals. Unfortunately using “forever chemical” substances like these is unlikely compatible with those goals. Still, we see many two-phase immersion cooling references still in the industry. For example, Dell’s Data Center Power and Cooling Solutions page still lists two-phase immersion cooling as an option despite the company’s ESG focus. Our sense is that Dell still lists that as a conversation starter. That is just one of many examples where the industry has been slow to update. Perhaps new fluids will be developed and tested for use in two-phase immersion cooling in the future.
Final Words
Many will have noticed that we stopped covering two-phase immersion cooling last year as the US EPA and Belgium started taking a bigger interest in PFAS. Hopefully, the industry finds a solution, but this is generally a challenge with phase change approaches. With the phase transition to gas, opening tanks for maintenance is a bigger challenge. That gas also has the potential to be toxic depending on the fluid used, which is what makes two-phase immersion cooling that much more challenging for data center operations.
For now, it seems like we are in a single-phase versus direct-to-chip liquid cooling world as the big US hyper-scalers are cutting back on these projects and the primary fluid will cease production sometime in the next 18-19 months. If projects pick up with new safer fluids, we will cover those on STH again.
First they take the lead out of our gas, now they’re coming for the PFAS in our datacenters.
What’s next? Soon they’ll be banning bfloat16 because it “reduces precision” too much! /s
While this is a bummer from a power efficiency standpoint, it’s probably good that they are figuring this out now and stopping before it’s more widely adopted. Last thing we need is 15 years out and homelabers are getting cancer because they got a great deal on old two phase racks and servers.
Seems that chemicals in general are a pain in the neck. We can’t be using them because they cause cancer and other health issues, but we have to have them if we want to keep these things running cool and not eat up more energy from already strained power grids and generating plants. This whole PFAS thing is a no-win for everybody. I’m with “Guy” though. It’s good that this is being dealt with NOW when the installed base of two-phase fluid cooled equipment is still very small or non-existent and not later after thousands have died or fallen ill from just running servers like this and untold billions of dollars have been sued out of the companies providing them.
From an anonymous hyper-scale reader, you’re right, we closed the project a few months ago so we’re now looking at DLC.
Agreed, now is the time. Corporations have a way of being truly awful.
https://en.wikipedia.org/wiki/Dark_Waters_(2019_film)
Whhhhaattt? No one could have seen this coming. Again, all liquid cooling does is kick the can down the short, dry, waterless road. And kill anything aquatic. There’s a cruel twist for you.
Well as someone with several years experience using GRC single-phase mineral oil based liquid cooling tanks like that shown above…I will say three things: 1) The mineral oil will get on EVERYTHING in the data hall, so one big rule: NEVER mix parts or tools from air and mineral oil cooled servers / server rooms, 2) Have a set of clothes dedicated to mineral oil infused chores, 3) Be ready to come up MacGyver solutions of your own for triaging slow cable-based leaks out of the tanks (hello Lowes Garden Center).
I’m old enough to remember the last time that a nonflammable, halogenated fluid was used as a dielectric cooling fluid – that was Polychlorinated Biphenyls (PCB), which were banned in the 1970s. Fluorinated chemicals have exactly the same set of problems – toxicity, bioaccumulation, and no natural pathways to degrade. Investing in PFAS makes as much sense as a new project using PCBs.
Flammability concerns aside, n-pentane would be good for phase-change immersion, and n-heptane would be suitable for 2-phase systems. But I guess noone wants to soak their datacenters in what is pretty close to petrol, for whatever reason.
@RGB : I have my doubts about n-Pentane beyond just flammable (Flash point 57 °F)… https://pubchem.ncbi.nlm.nih.gov/compound/pentane
GHS Hazard Statements
H225: Highly Flammable liquid and vapor [Danger Flammable liquids]
H304: May be fatal if swallowed and enters airways [Danger Aspiration hazard]
H336: May cause drowsiness or dizziness [Warning Specific target organ toxicity, single exposure; Narcotic effects]
H411: Toxic to aquatic life with long lasting effects [Hazardous to the aquatic environment, long-term hazard]
Even if “Pentane is a natural product found in Calendula officinalis and Allium ampeloprasum with data available.”
For direct immersion cooling, can’t you just use deionized water?
Just curious, I don’t really know.
It seems the electric conductivity would be low enough, and you could immerse an anode in the cooling side to control any low level ions generated.
I’ve had to work on the Denali pass servers and that’s was bad enough. I can imagine trying to service one covered in liquid.
Given the history of polyfluoro-whatever I cannot imagine why anyone thought it a good idea here.
Every data-infrastructure thermal-management option has substantial environmental impacts at scale.
It is important to look carefully at trade-offs among currently-deployed and anticipated-near-term options.
These options include two-phase NON-immersion Direct Liquid Cooling approaches that are already being deployed WITHOUT using the working fluids mentioned in this article.
Commercial offerings are already progressing toward using working fluids with more-favorable characteristics.
Here is one example:
https://blog.zutacore.com/zutacore-blog/zeroemissions
Perfluoropolyethers are usually the way to go, but rather expensive.
@Craig Dunwoody given that use of R-1336mzz(Z) is relatively limited and no study data on safety for humans has been completed I would be hesitant to praise any new refrigerant. That’s before considering that using a refrigerant that needs to be superheated and thereby is less efficient makes for a very different cost calculation.
@Michiel
This is probably the same set of warnings (or, most likely, less/fewer) than for the petrol that hundreds of millions of people put in their cars on a daily basis.
Highly Flammable liquid and vapor: Yes, for sure.
May be fatal if swallowed and enters airways: Do not put it in your mouth or try to drink it, and you will be fine.
May cause drowsiness or dizziness: Yes, huffing it will get you high(“Narcotic effects”), but it is not healthy. Avoid getting lungfuls of the vapors, same as petrol.
Toxic to aquatic life with long lasting effects: Yes, fish and frogs do not like petrol in their water.
Make decommissioned subs into data centers.
A huge thank you for this article!
The absense of news there has been bugging me for a while, but it seems a recurrent issue in IT reporting (although it could be far more general): certain failed hypes just don’t get explained for complete lack of press and good explanations are hard to come by on the open internet (while naive rumors abound).
The HP memristor and “the machine” were another one that was seriously hyped at the very highest level and then died a death so quiet, you’d never know what or who killed it.
Doug, you could use deionized water for a short time. However the complete lack of ions in it means ions want to diffuse into it from the surrounding. In the case of PC components that would be copper, tin, etc. After a short time your DI water will have corroded most of the metals and stripped ions from them…at which point it’s no longer deionized water but rather water full of metal ions.
@David appreciate your comment, and I’ll go further in not praising any working-fluid option at all. Every one has negatives that some others don’t.
Understood that there are substantial differences among options regarding amount and quality of available data on environmental and health effects throughout fluid production/deployment/recycling/disposal lifecycles.
I do believe that across years, commercial thermal-management offerings are making incremental progress toward better working-fluid options.
I also think that although details are highly implementation-specific, benefits of two-phase DLC relative to two-phase immersion could include potentially smaller total working-fluid volume; and potentially less human exposure to working fluid and vapor during routine system operation and maintenance.
The 2 phase solution was using a hammer to crack an egg, we work with GRC in Europe and this is a simple solution with huge sustainability benefits so it will be adopted in time but we are in a risk averse industry. Dell and Intel have come on board with this solution now. In ten years we will look back and say ‘did we really blow air around servers to cool them at one time?’
We collaborate with GRC in Europe, and this is a straightforward solution that has substantial sustainability advantages; as a result, it will be implemented in due course, although our industry is notoriously risk-averse. Currently, this approach has the support of Dell and Intel. Do you honestly think we blew air around servers to cool them once? We’ll ask ourselves that question again in 10 years.