Hardware
Phase 3: Cray XE6 system
The current Phase 3 system (XE6) is contained in 30 cabinets and comprise of a total of 704 compute blades. Each blade contains four compute nodes giving a total of 2816 compute nodes, each with two 16-core AMD Opteron 2.3GHz Interlagos processors. This amounts to a total of 90,112 cores. Each 16-core socket is coupled with a Cray Gemini routing and communications chip. Each 16-core processor shares 16Gb of memory, giving a system total of around 90 Tb. The theoretical peak performance of the phase 3 system is over 800 Tflops.
There are 16 service blades on phase 3, each with two dual-core processor sockets. They act as login nodes, controllers for the I/O and for the network.
There is one Gemini router chip for every two XE node. This Gemini chip has 10 network links which are used to implement a 3D-torus of processors. The MPI point-to-point bandwidth is 5 GB/s or more. The latency between two nodes is around 1-1.5μs.
Storage systems
Shared, high-performance parallel filesystem
Over 1 PB of high-performance RAID disks are available. The disks are accessible globally from any compute node and use the Lustre distributed parallel file system.
Archive system
The archive system is based on Symantec's Enterprise NetBackup and currently consists of 1300 800GB tapes, with a maximum capacity of approximately 1.02 PetaBytes.
Backup system
The backup system includes 70 TB of disk space, known as NAS (Network Attached Storage) space, which holds the users' home directory space. Files are backed up initially to a 168 TB MAID (Massive Array of Idle Disks) disk space, from which they are staged to a tape system as required.
The NAS storage is held on BlueArc Titan 2200 servers. MAID storage is held on a COPAN Systems Revolution 220TX MAID VTL. The tape subsystem is a Quantum i2000 tape library with 4 LTO-4 FC tape drives. The backup system is controlled by 3 net nodes through 12 Gigabit Ethernet connections. The system includes 2 redundant Windows servers running Veritas NetBackup software.
Previous hardware
You can also find information on the previous phases of HECToR:
