HiPerGator

HiPerGator Hardware Specification Sheet

Click here to download the most recent version of the HiPerGator hardware spec sheet.

HiPerGator 1.0:

  • 16,000 cores
  • 64 TeraBytes of RAM
  • 2 PetaBytes of disk storage
  • Maximum speed of 157 Teraflops = 157 trillion floating point operations

HiPerGator 2.0 expansion adds:

  • 30,000 cores
  • 120 TeraByte of RAM
  • 1 PetaByte of disk storage
  • Maximum speed of 1,100 Teraflops – 1,100 trillion floating point operations
HiPerGator 2.0

HiPerGator 2.0

To put these numbers in context, consider the following comparisons:

  • The 1,100 Teraflops speed of HiPerGator 2.0 is the same as the combined speed of 600 PlayStation 4 and 840 Xbox One gaming stations, if they could be made to work together effectively and efficiently on large data and complex problems.
  • The human brain is estimated to have the power equivalent to 20 Petaflops.
  • The 120 TeraByte of RAM memory, not the 1 PetaByte of long-term disk storage, of HiPerGator 2.0 can hold 240 million books.

Nodes and Processors

HiPerGator consists of over 21,000 compute cores (processors). The main part of HiPerGator consists of 256 nodes that each have 4 sockets with a total of 64 cores and 256 GB RAM. Though there are several generations of processors, all adhere to the X86_64 ABI (Application Binary Interface) and are largely software compatible. The industry trend toward more processors or “cores” per die rather than increasing clock frequencies has resulted in a fairly narrow range of processor frequencies (2 to 3 GHz) among our machines but a wide range of core counts (4 to 80).

All of the processors available on HiPerGator are homogenous in terms of instruction set and support the 64-bit extensions to the Intel X86 instruction set architecture (X86_64). This instruction set architecture is referred to as “AMD 64” by AMD and “Intel 64” by Intel. However, HiPerGator has several generations of processors from both AMD and Intel implementing what we will generically refer to as the X86_64 instruction set architecture (ISA).

Currently, available processors range from older hex-core AMD Opteron 4184s to the newest 16-core AMD Opteron 6378s. In between we have several generations of Intel processors including a significant number of quad-core E5462s as well as the latest E5-2600 series. These processors range in clock speed from 2.0 GHz at the low end to 3.3 GHz at the high end. We will continue to add additional CPU resources to meet growing demand from our researchers. Older processors will be retired as they become economically inefficient to operate.

The expansion of HiPerGator consists of 30,000 cores of the latest Intel E5-2698 v3 (16 core, 2.3 GHz, 135 W) processors with 4 GB RAM per core in approximately 1,000 nodes made by Dell. The diskless nodes have 128 GB of 2133 MHz DDR4 RAM. Each node in the expansion has two sockets with a total of 32 cores per node. The total amount of RAM in HiPerGator 2.0 is 120 TeraBytes.

Electrical power and performance measures

Some researchers are interested in monitoring electrical power consumption of their computations. The SLURM scheduler generates a power summary at the of a job; by using SLURM directives you can control many details of your job, such as whether all tasks run on a single node and whether the node has the Intel processors or AMD processors. However, because a HiPerGator is a production system, we do not allow users to change the frequency of processors, or to do other research that would involve full dedication of hardware and root access.

Network

  • The nodes are connected by Mellanox 56Gbit/s FDR InfiniBand interconnect for fast data access to storage and for distributed memory parallel processing. The core switches use the 100 Gbit/s EDR InfiniBand standard.
  • There is a 10 Gbit/s Ethernet for management and for providing the operating system to the diskless nodes.
  • There is a separate 10 Gbit/s Ethernet network connecting the nodes to support the planned new services of running virtual machines and virtual cluster on HiPerGator.

Storage

The storage of all user data is on a fast parallel and highly redundant storage infrastructure that can sustain high input output rates from the many thousands of processors in HiPerGator making simultaneous requests. The parallel file system is Intel Enterprise Edition Lustre 2.5 and has a total capacity of 3 PetaBytes.