Liquid cooling has turned up for brief periods ever since in mainstream products. A few years before switching processor architectures completely in the mid-2000s, for some of its high-end PowerPC Macintosh machines Apple opted for a closed-loop system based on a mixture of water and propylene glycol. The move, however, was short-lived and the company moved back to entirely air-cooled designs. Liquid cooling was something for the more esoteric end of the high-performance computing spectrum and high-power scientific instruments.
After years out in the cold, industries as diverse as automotive, renewable energy and data-centre computing have one in thing in common that has made them look at liquid cooling again as a realistic option: the growing heat density of some types of semiconductor devices. In 2016, liquid cooling re-emerged on a small scale in mobile handsets. Samsung opted to put sealed heat pipes into its top-end phones to convey heat away from the processor and avoid hotspots forming on the casing. The pipes use phase transitions between liquid and vapour near the ends to provide the cooling action.
For carmakers, the problem is the power transistors used in full-electric and hybrid vehicles. The initial approach was a cold-plate design similar to that employed in the computing sector: the fluid runs close to the electronics with thermal paste used to maintain low resistance, which is not always effective.
For suppliers such as Danfoss, Fuji Electric and Hitachi, the answer was to use direct cooling: where the sealed module is partially immersed in water. The cold plate has a recess that fits around the lower half of the module, with an O-ring seal used to prevent leakage. Water is then pumped under the module when the car is running. Hitachi claims its approach cuts the junction temperature by 25°C compared to indirect cooling even using the same cooling fluid as that used for the main engine, which can be as hot as 105°C.
Data centres are going all the way. At Alibaba, the Chinese online marketplace and a major data-centre operator, the answer is to immerse entire server racks in a liquid bath. Speaking at the recent Open Compute Project (OCP) summit in the spring, staff engineer Yangfan Zhong said the company has run complete racks immersed in fluid for over a year: long enough to work out that some components take the experience better than others and work out trade-offs in design.
Relays and optical components suffer from the cooling liquid penetrating their packages unless they are fully sealed. PVC plastic controls on the front panels tend to dissolve. Better news is that components with moving parts such hard disk drives can benefit from immersion. “The failure rate in liquid is 50 per cent lower than in air,” Zhong claims, based on a year of operation.
Less easy to predict is the effect of the liquid immersion on high-speed serial communications. Relatively high dielectric constants can be instrumental in wrecking the performance of protocols such as SATA, according to Zhong. Those with a performance much closer to that of air perform better.
Alibaba opted for a bath design that makes it relatively easy to lift blades out of a horizontal chassis, albeit with the help of a robotic crane, and so the liquid interfaces directly with air. However, the company found it needed to seal the units for regular operation because evaporation in more open tanks meant it lost some 10 per cent of the fluid during tests. With fluorocarbons, which are the most expensive but also the least flammable and highest performing fluids compared to mineral and vegetable oils, the cost of losses is significant.
Full immersion
Alibaba is not alone in pursuing full immersion, although other operators are taking a more cautious approach. Microsoft is experimenting with everything from attaching liquid-cooled cold plates to blades to partial immersion techniques that focus on heat transfer from the processor and memory heatsinks to full two-phase cooling systems with fully immersed racks. “There isn’t a commitment to any particular solution,” says Microsoft Azure director of engineering Brandon Rubenstein, adding that rack-level liquid cooling in production data centres is likely within a couple of years though it may take ten years for entire data centres to be based almost entirely on racks or shelves that rely on liquid flowing around the electronics.
Above: A Green Revolution Cooling (GRC) installation
A potential issue for immersion cooling in data centres is that the server blades are lowered into a horizontal bath rather than being slid into a vertical rack. This could have the effect of reducing server-blade density in locations where every square inch of floor space is precious. Dhruv Varma, director of product marketing at Green Revolution Cooling (GRC) argues: “While it is true that an individual horizontal rack has a slightly larger footprint than a vertical rack, when you look at the fully burdened data centre footprint, immersion cooling produces significant space savings.”
With immersion cooling, Varma says, there is no need for hot and cold aisles that separate conventional racks. The immersion tanks can be installed next to each other. The improved cooling also makes it possible to use every slot. In many cases, hot-running air-cooled servers need to be sited in racks that are not fully populated. Additionally, server designers may take advantage of the additional cooling to increase processor density on the blades themselves.
As well as core data centres, density could be the key to seeing immersion cooling be more widely used in the new generation of edge data centres, often colocated with telecom equipment to reduce communications latency. Vasanth Krishnan, energy analyst at Frost & Sullivan, says the market for edge data centres is set to more than quadruple to $3.6bn by 2022.
In addition to allowing higher compute density, Krishnan argues liquid cooling requires less maintenance than forced-air cooling of similar calibre. As colocation is a frequent deployment model, he notes: “Liquid cooling also offers the flexibility to bump up the capacity without having to change the entire cooling system. Also the cost of coolants are on the way down, which is encouraging data-centre operators to adopt immersion cooling.”
Varma says: “In our experience, if the user wants to put immersion-cooled racks in their cage, colo operators do not have any problem with it. The modular nature of the technology allows you to set up an ultra-efficient data centre, cost-effectively even in the absence of scale.
“Further, since no airflow is needed, the pods can be completely sealed. This is one of the primary reasons the US Air Force has been testing and using our ICEtank solution for more than three years.”
To try to help users pick their way around the decisions, the OCP has formed a working group to look into standards and minimum requirements for liquid cooling systems. These will take into account trade-offs such as the cost of high thermal transfer versus density as well as flammability and electrical ratings.
Rolf Brink, CEO of cloud-infrastructure specialist Asperitas, says the dielectric breakdown strength of the fluids could probably be relaxed and open the door to more choices.
Many of the current candidates were originally developed for use in cooling high-voltage switchgear and so often sport ratings of hundreds of kilovolts. After years in the wilderness, liquid cooling may be back in the mainstream thanks to the enormous thirst for power of some of the key markets for electronics.
