Changes to HECToR for Phase 3
There will be a number of changes to the system during the switch to Phase 3 next week. We outline those that will impact on users below. If you are unsure of any of the details or have any further questions please contact the HECToR Helpdesk at helpdesk@hector.ac.uk.
You can find the timetable for the Phase 3 upgrade on the HECToR website:
Machine Name
The alias login.hector.ac.uk will continue to work through the upgrade process and also for the final Phase 3 service and we recommend that all users use this alias when accessing HECToR to avoid confusion. The underlying name of the machine will change to phase3.hector.ac.uk when the machine returns with the Phase 3 configuration (Interlagos processors) at 09:00 11 November 2011.
Queues
Any jobs that remain in the queues at 09:00 on Wednesday 9 November 2011 will be deleted as when the system returns it will be at the Phase 3 configuration (Interlagos processors) and jobs using binaries compiled on the Phase 2b system (Magny-Cour processors) are not guaranteed to work.
There will be a number of changes to the queue layout throughout the upgrade process. These are summarised below:
- Throughout the upgrade process the Low Priority Access (LPA) queues will be disabled. Any jobs sent to the LPA queues will queue until such time as they are re-enabled.
- During the first part of the upgrade (17:00 7 November 2011 - 09:00 9 November 2011, when HECToR is 896 Phase 2b (Magny-Cours) nodes, (24 cores per node, 21504 cores) the maximum queue size available will be 512 nodes (12288 cores).
- During the second part of the upgrade (09:00 11 November 2011 - 09:00 16 November 2011, when HECToR is 1856 Phase 3 (Interlagos) nodes, (32 cores per node, 59392 cores) the maximum queue size available will be 1024 nodes (32768 cores).
Once the machine returns as the full-size Phase 3 (Interlagos) system at 09:00 17 November 2011 it will consist of 2816 nodes (32 cores per node, 90,112 cores) and the maximum queue size will be 2048 nodes (65,536 cores).
Changes to job submission scripts
With the change of processor architecture to Interlagos there will be 32 cores available per node. This means that the maximum values for the -l mppnppn PBS option and the -N aprun option will change to 32. The majority of users use fully-populated compute nodes and will wish to change there job submission scripts to include the line:
#PBS -l mppnppn=32
and change the corresponding value of the -N option to aprun to 32.
The Running Jobs chapter of the HECToR User Guide will be updated to provide more details and example job submission scripts for the Phase 3 system. You can preview the version of this document for the Phase 3 system at:
This may help you in understanding what you must do to switch to running on the Phase 3 system.
Recompiling for Interlagos
As the Phase 3 system uses a new processor architecture all software should be recompiled prior to running on the system. If you run into any difficulty in recompiling your code please contact the HECToR Helpdesk at helpdesk@hector.ac.uk.
Changes to system software and compilers
As part of the upgrade of HECToR to Phase 3 there will be some changes to the system software and compilers available on HECToR. We will also be updating the default versions of many of the system software to support the new Interlagos Opteron processors.
Default compiler suite change
The default compiler suite on the system will change from PGI to Cray. If you still want to use the PGI compiler you can do so with the
module switch PrgEnv-cray PrgEnv-pgi
command. To make the change permanent you should add this line to your $HOME/.bashrc.
xt-mpt parallel environment module depreciated
As part of the upgrade to the HECToR operating system in preparation for the phase 3 upgrade the xt-mpt module (which contains the MPI and SHMEM libraries) has been depreciated in favour of two separate modules:
- xt-mpich2 - Contains the MPI library (now loaded by default).
- xt-shmem - Contains the SHMEM library.
This change was implemented on 2 November 2011.
Pathscale compiler suite no longer fully supported
The Pathscale compiler suite continues to be available on Phase 3 although, as it is no longer officially supported by Cray, none of the system numerical libraries (for example, xt-libsci which include BLAS and LAPACK) support the compiler suite.
We strongly encourage all users to switch to a different compiler suite on Phase 3 as the future of this compiler on the Cray architecture is uncertain.
If you still wish to continue using the Pathscale compiler suite you will need to either compile your own versions of the numerical libraries you need or try to manually link to the GNU versions on HECToR.
Changes to module defaults
Accompanying the change of architecture to Interlagos processors there will be a number of changes to the default versions of modules on the system to allow codes to take advantages of the new processor architecture.
The xtpe-interlagos module will be loaded by default instead of the xtpe-mc12 module. This module ensures that the compilers are passed the correct compilation flags to compile for the new processors and that the optimal versions of libraries are picked up.
Other planned changes to module defaults are documented on the HECToR website at:
We plan to implement these changes on the 11 November 2011
Charging
Charging for jobs will be disabled from when the system returns with Interlago processors (09:00 11 November 2011) until the acceptance and availability testing is completed (estimated early December).
The final charging rate for the Phase 3 system is still to be determined and will be communicated to users at a later date.
