Immersion-1 Technical Details
In short, Immersion-1 is a massive immersion cooled FPGA cluster with 6048 Spartan®-6 LX150 FPGA chips from Xilinx. The FPGA chips sit on 1512 hot swappable boards and get power and communications from backplanes that are connected to the outside world.
While Immersion-1 is a fully functional unit that does indeed crunch numbers 24/7, it is also proof of concept that 2-phase immersion cooling is not just a dream of the future, it is a viable and elegant solution right here and right now.
Most importantly, our 2-phase immersion cooling approach cuts down engineering inefficiencies of legacy designs, lowers development and operational costs and results in a drastically lower carbon foot print. And all this while enabling us to pack the hardware in a never before seen density, keep the supporting infrastructure at a minimum and use our existing premises, instead of investing our life savings into large and remote co-location data center facility. In fact, with all that space saved, we could place our tanks at an ergonomic height for easy access - with plenty of headroom left.
With all that space saved, there is plenty of room for expansion.
Immersion-1 doesn't need fans, heatsinks or air channels.
Board to Board
Off to a good start, but current density is only scratching the surface of possibilities.
Immersion-1 Fact Sheet:
- Immersion-1 is the name of the world's largest FPGA cluster and the facility where we host it.
- Immersion-1 is a massive prototype and proof of concept for a whole new generation of computation clusters.
- Immersion-1 had to be approved by two governments before we could build it.
- Immersion-1 was built in less than half a year with a budget lower than traditional CRAC cooling equipment.
- Immersion-1 is energy efficient and saves more than 90% on electricity in comparison to traditional air cooling (even with all tricks in the book of air cooling employed, ie. cold/hot aisle containment etc).
- 6048 Xilinx Spartan®-6 LX150 FPGA cores with a total of 891 million logic cells
- 1512 hot swappable boards, connected to high density backplanes (board to board 8.5mm)
- Resulting in 64 FPGA chips organized in "logic cubes" of only 160mm (6.5") on the longest side
- In-house developed passive 2-phase immersion cooling system
- Powered by 3M™ Novec™ Engineered Fluids
- Total heat dissipation typically 70kW
- Running on only 3 variable speed outdoor fans (often our fans run on minimum)
- Reduced FPGA junction temperature, lower error rates and higher frequencies
- 96 ATX high reliability 850 Watt power supplies, capable of delivering a maximum of 81.6 Kilowatt to the cluster
- PSU to board power distribution 12V
- Measured efficiency >90%
- Industrial strength remote controlled power switches
- Real-time current measurement down to the PSU level
- 312A 230V typical load, 528A 230V maximum load
- 800A 400V TP/N Plug-In MCCB at facility level
Monitoring and Data Acquisition
- Networked modular DAQ system
- Measuring temperatures, pressure, liquid level and flow
Facility Monitoring and Control
- Temperature & Humidity
- Lights, Air Ventilation, Doors
- Flow rates, Fan & Pump frequencies
Silent and Dust Free Operation
Most of the time the only sound you hear is the hum of our high current main power cables. Once in a while, we work around the cluster and that's when it gets really "noisy": we have to turn on the air condition (Hong Kong has a hot and humid climate). Compare that with a modern data center facility where noise and dust is usually a real problem (typically 70-80 dB, using ear protectors in air cooled data centers is recommended).