Hardware
Phase 2b: Cray XE6 system
The Phase 2b (XE6) is contained in 20 cabinets and comprises a total of 464 compute blades. Each blade contains four compute nodes, each with two 12-core AMD Opteron 2.1GHz Magny Cours processors. This amounts to a total of 44,544 cores. Each 12-core socket is coupled with a Cray Gemini routing and communications chip. Each 12-core processor shares 16Gb of memory, giving a system total of 59.4 Tb. The theoretical peak performance of the phase 2b system is over 360 Tflops.
There are 16 service blades on phase 2b, each with two dual-core processor sockets. They act as login nodes, controllers for the I/O and for the network.
There is one Gemini router chip for every two XE node. This Gemini chip has 10 network links which are used to implement a 3D-torus of processors. The MPI point-to-point bandwidth is 5 GB/s or more. The latency between two nodes is around 1-1.5μs.
Storage systems
Shared, high-performance parallel filesystem
596 TB of high-performance RAID disks are available. The disks are accessible globally from any compute node and use the Lustre distributed parallel file system.
Archive system
The archive system is based on Symantec's Enterprise NetBackup and currently consists of 1300 800GB tapes, with a maximum capacity of approximately 1.02 PetaBytes.
Backup system
The backup system includes 70 TB of disk space, known as NAS (Network Attached Storage) space, which holds the users' home directory space. Files are backed up initially to a 168 TB MAID (Massive Array of Idle Disks) disk space, from which they are staged to a tape system as required.
The NAS storage is held on BlueArc Titan 2200 servers. MAID storage is held on a COPAN Systems Revolution 220TX MAID VTL. The tape subsystem is a Quantum i2000 tape library with 4 LTO-4 FC tape drives. The backup system is controlled by 3 net nodes through 12 Gigabit Ethernet connections. The system includes 2 redundant Windows servers running Veritas NetBackup software.
