Hardware
HECToR (Phase 2a) is an integrated system which includes a scalar MPP XT4 system, a vector X2 system and storage systems.
Cray XT4 scalar supercomputer
The XT4 comprises 1416 compute blades, each of which has 4 quad-core processor sockets. This amounts to a total of 22,656 cores, each of which acts as a single CPU. The processor is an AMD 2.3 GHz Opteron. Each quad-core socket shares 8 GB of memory, giving a total of 45.3 TB over the whole XT4 system. The theoretical peak performance of the system is 208 Tflops.
There are 24 service blades, each with 2 dual-core processor sockets. They act as login nodes and controllers for I/O and for the network.
Each quad-core socket controls a Cray SeaStar2 chip router. This has 6 links which are used to implement a 3D-torus of processors. The point-to-point bandwidth is 2.17 GB/s, and the minimum bi-section bandwidth is 4.1 TB/s. The latency between two nodes is around 6μs.
The system is held in 60 cabinets.
Cray X2 vector system
The X2 part of the system includes 28 vector compute nodes; each node has 4 Cray vector processors, making 112 processors in all. Each processor is capable of 25.6 Gflops, giving a peak performance of 2.87 Tflops. Each 4-processor node shares 32 GB of memory.
The X2 interconnection network has a point-to-point bandwidth of 16 GB/s and a bi-section bandwidth of 254 GB/s. The average ping-pong MPI latency ~4.6 microsec.
Storage systems
The storage systems are accessible both by the XT4 and the X2 systems.
Direct attached storage
934 TB of high-performance RAID disks are controlled by 3 controllers through 12 IO nodes. The disks are accessible globally from any compute node and use the Lustre distributed parallel file system.
Backup system
The backup system includes 70 TB of disk space, known as NAS (Network Attached Storage) space, which holds the users' home directory space. Files are backed up initially to a 112 TB MAID (Massive Array of Idle Disks) disk space, from which they are staged to a tape system as required.
The NAS storage is held on BlueArc Titan 2200 servers. MAID storage is held on a COPAN Systems Revolution 220TX MAID VTL. The tape subsystem is a Quantum i2000 tape library with 4 LTO-4 FC tape drives. The backup system is controlled by 3 net nodes through 12 Gigabit Ethernet connections. The system includes 2 redundant Windows servers running Veritas NetBackup software.
Archive system
The archive system is based on Symantec's Enterprise NetBackup and currently consists of 1300 800GB tapes, with a maximum capacity of approximately 1.02 PetaBytes.
You can also find information on the HECToR Phase 1 hardware.
