Welcome to HECToRNews 6, November 2009
- Training Course Timetable
- HECToR Phase2B Upgrade Summary
- New Face for the HECToR website
- Technical Reports
- Access to HECToR
- HECToR Distributed CSE Support Technical Meeting
- Programming Environment issues
- Distributed Support
This is the sixth Newsletter for HECToR users from the Computational Science and Engineering support (CSE) team of NAG Ltd. The HECToR newsletter intends to keep users updated with useful information on the national supercomputing service, for the previous issues please see here.
In this issue we have information on HECToR related training courses, details of the forthcoming phase 2B hardware upgrade, technical reports on code optimizations for multi-core, a new streamlined method of access to significant HECToR resources, general points regarding the HECToR programming environment and information on the distributed CSE support service.
Training Course Timetable
The current schedule is below:
- November 9-10, 2009 NAG Manchester - Quad Core Training
- November 11, 2009 NAG Manchester - DL_POLY
- January, 2010(tba) Southern venue - Quad Core Training
- January 4-6, 2010 Imperial College London - Parallel Programming with MPI
- January 7-8, 2010 Imperial College London - OpenMP
- January 11-15, 2010 NAG Oxford - Best Practice in HPC Software Development
- January 26-28, 2010 University of Bath - Parallel Programming with MPI
These training courses run by NAG Ltd. are provided free of charge to HECToR users and UK academics whose work is covered by the remit of one of the participating research councils (EPSRC, NERC and BBSRC).
HECToR Phase2B Upgrade Summary
The plan for the Phase2B upgrade was confirmed mid September. The rationale behind the upgrade path as discussed at the Town Meeting in April are available online.
The Phase2B upgrade will take place in two stages:
- A 20 cabinet 'Baker' system will be installed, with an early user service available in March 2010. The system will comprise 44,544 cores (464 blades x 8 chips x 12 cores), delivering an estimated peak performance of 338 Tflops. Approximately 30 cabinets of XT4 will also be retained. The XT4 will provide the main service at this stage.
- This will be followed by an upgrade to the current Seastar 2+ interconnect by replacing it with the newer Gemini interconnect by the end of 2010.
The combination of Baker+Gemini will then provide the main service. Exact details of the upgrade will be confirmed nearer the time. The transition to 12 core is an exciting prospect and if a user suspects that their code may need performance investigation prior to the upgrade, then they are encouraged to seek the help of the NAG CSE team by contacting the HECToR helpdesk. Help and advice can then be given on how the code might be able to get full benefit from the hardware upgrade.
New Face for the HECToR website
The HECToR Website has recently been simplified to convey a high level public view of the service.
This new view includes:
- example case studies from users
- an introduction to the service
- guidance notes for applying for access
The original site with all the detailed information for users is now in a separate 'User Site' area. This is also accessible by the 'User Site' links on the new home page. You can also access this directly here. Note - all other bookmarks you have will still work. As always your comments on the website are welcome. Please contact the HECToR helpdesk if there is any additional content you would like to see on the site.
There are two new technical reports written by the NAG CSE team which are now available:
- optimization of a direct numerical simulation code for turbulence and combustion SoFTaR
- quad core benchmarking and optimisation for CASTEP
Both reports are the result of researchers contacting the NAG CSE team and requesting to have their code's performance investigated. Site visits were arranged to facilitate in depth discussion with the research staffs involved. The CSE team were then able to provide more in depth support. Performance bottlenecks were identified and new communication routines were implemented improving the code performance.
In particular, the shared memory optimizations for multi-core which were implemented in CASTEP are well understood by the CSE team and they are widely applicable to other codes so if you think this very effective optimization may be beneficial to your code please contact the NAG CSE team. Please also note that the improved shared memory version of CASTEP, (CASTEP 4.4-shm-xt) is available on HECToR now. See the Applications section below for further details on how to use this module.
The NAG CSE team have also produced a Fortran compiler performance comparison review page. This provides a comparison of different compiler performance for a collection of Fortran codes currently being used on HECToR. The codes are from different scientific areas and use a selection of computational algorithms which are common to most high performance numerical codes.
Access to HECToR
HECToR Direct Access, Call for applicants
In response to the 2009 Gateway review of the HECToR project, EPSRC are piloting a new access class for the HECToR service called 'Direct Access' (Class 1B) where users can request significant amounts ( greater than 1,000,000 AUs) of computing resource only for a period of up to six months.
The Direct Access mechanism will provide successful applicants with discretionary access to the HECToR system in order to provide resource to cover activities such as:
- Bridging access between grant applications,
- Resource to trial application developments at scale,
- rovide preliminary results in aid of grant applications via Responsive Mode or to a Call for Proposals;
There is no deadline for submission, however: the proposals will be assessed by a Resource Allocation Panel (RAP) which will meet every four months (February 2010, June 2010 and October 2010), the deadline for applications being submitted to the Panel will be three weeks before the Panel meeting.
How To Apply
You should complete a Technical Assessment Form. The technical assessment form should then be submitted to the helpdesk. The completed technical assessment will normally be returned to the PI within two weeks of submission, although if the reviewer has concerns about the project, it may take extra time to address these with the PI. Applicants must submit their request for a technical assessment at least two weeks before they intend to submit their application to EPSRC.
If the technical assessment endorses the proposal, the applicant is required to email a completed RAP Application Form and the completed technical assessment directly to EPSRC. Applications should not be submitted through Je-S.
Proposals will not be subject to postal peer review; instead they will be assessed by the Resource Allocation Panel (RAP) who will meet every four months. The HECToR RAP inbox will be checked weekly and the submitted proposals will be considered at the next RAP meeting provided the accompanying documentation is complete and correct.
|Closing date||RAP meeting||Deadline for start of access|
|12th January 2010||2nd February 2010||2nd March 2010|
|27th May 2010||17th June 2010||17th July 2010|
|21st September 2010||12th October 2010||12th November 2010|
Applicants will be notified of the decision within one week of the Panel meeting. Successful applicants should then contact firstname.lastname@example.org to confirm the start date of their access. The start date must be within one month of the panel date and resources must be used within six months of the start date.
These details and all the necessary forms are available at:
- http://www.hector.ac.uk/howcan/admin/apply/HECToRRAP.php and
Increase in Computing Resource for Class 2 Access
Since the start of HECToR new users wishing to explore the capability and usage of the service are allowed to apply for Class2 project access. This can be either with a Class 2A application for access without requested distributed (dCSE) support or Class 2B access which accompanies the submission of a proposal for dCSE support. Under this mechanism up to 200,000 AU can now be requested for Class 2A access and up to 400,000 AU for Class 2B access. This is double the previous amount which was allowed for Class 2 projects. For further details please see here
Suspension of Charging on the X2
In February, charging was suspended on the X2 for an initial trial period of 6 months. It is confirmed that charging will remain suspended on the X2 through to the end of 2009. The situation will then be revisited.
Any new or current HECToR PIs wishing to the use the X2 should submit a Class 2 technical assessment form to the Helpdesk. This form should clearly state the project details. Please note that it is not necessary to calculate an AU requirement and this open access is not restricted to EPSRC users. All new X2 HECToR users will be required to attend a specific X2 training course
For more information on the training course in using the X2 please see the X2 course description.
Usage of the X2 will continue to be tracked, but you will not be charged. Users are encouraged to take this opportunity to explore ways of employing the vector capability in their research. Application forms are available online. If you have any questions, please do not hesitate to contact the HECToR helpdesk.
New Capability Incentive Scheme on HECToR
A Capability Incentive Scheme is now in operation on HECToR. The principal purpose of the Research Councils in funding the HECToR service is to provide a system with capabilities (both in hardware and user software support) which are unmatched by computing services at the University or department level. HECToR is a high-end supercomputing platform, and as such the Capability Incentive Scheme is targeted at large-scale simulations which can effectively utilise a significant fraction of the machine.
The Capability Incentive Scheme is an encouragement to users to broaden their computational science and to exploit the capabilities of the service. Under the scheme, jobs will be discounted at three levels depending on how well they scale. The three levels of incentives are:
|Level||Min Number of Cores||AU Discount|
For example, a job running in a "par:n512c4" queue (2048 cores) would receive a 15% discount, regardless of job duration.
Users of HPCx may already be familiar with a similar scheme which requires the use of a token. In the case of HECToR Capability Incentives, you do not need to apply to receive the discount, nor do you need to specify a token in your batch script.
ALL jobs on HECToR using more than 1024 cores will now be automatically discounted at the appropriate level. Users wishing assistance in scaling their codes should contact the HECToR helpdesk.
HECToR Distributed CSE Support Technical Meeting
- Focus on Computational Science and Engineering;
- Share experiences;
- Identify common issues and solutions;
- Strengthen HPC in the UK.
The slides from presentations given at the meeting are available here. The attendees were all involved with past or present dCSE projects. Further technical meetings are planned and will also include contributions from the wider HECToR user community.
Programming Environment issues
Compiling for quad core
Please remember when compiling your code for quad core you should load the module xtpe-quadcore. This will add the appropriate compiler flags for targeting quad core to the ftn and cc wrappers. It will also link in the quad core enabled version of libsci. These instructions along with the full explanation of what the quad core upgrade entails are available here.
However, the following message will appear
W-A-R-N-I-N-G **** xtpe-quadcore module is removed in the next release. Unload xtpe-quadcore and use xtpe-barcelona, xtpe-istanbul or xtpe-shanghai. To unload, use this commandline: module unload xtpe-quadcore
Currently, module load xtpe-quadcore will invoke the xtpe-barcelona module. The message can therefore be avoided by using module load xtpe-barcelona instead of module load xtpe-quadcore.
HECToR Archive Solution
The ability to archive data is critical for many groups on HECToR. The development of the HECToR archive solution is progressing well, and the archive solution should be in place for acceptance testing early November.
Users of HPCx will be familiar with the solution. There will be a simple command line interface for users, as well as a scheduled nightly archive from pre-defined directories. Both onsite and offsite archive storage will be available.
CASTEP 5.0 / CASTEP 4.4-shm-xt
The latest version of CASTEP (5.0) has now been installed on HECToR (for the XT system).You can select this version by using:
module add castep/5.0/xt
There is also a newly installed version of CASTEP 4.4 with shared memory enhancements implemented by the NAG HECToR CSE team which improves the scaling and performance of the code, for more details please see here. You can select this version with:
module add castep/4.4/shm-xt
More information on running CASTEP jobs (along with sample job submission scripts) is available on the HECToR User Wiki. (You will need your HECToR SAFE login details to access the Wiki).
To gain access to the CASTEP code on HECToR you must have a valid UKCP licence, available from http://www.castep.org/. Once you have a valid licence please contact the HECToR helpdesk to enable access to the CASTEP code for your account.
VASP 5.2 is now available on HECToR.
To gain access to the VASP 5.2 code on HECToR you must have a valid VASP 5 licence. Note - existing users of VASP on HECToR will need to apply for a separate license in order to use VASP 5. Once you have a valid VASP 5 licence please contact the HECToR helpdesk.
There are two binaries: vasp5 (default) and vasp5_gamma. You can select this version using
module load vasp5
More information on running VASP jobs (along with sample job submission scripts) is available on the HECToR User Wiki. (You will need your HECToR SAFE login details to access the Wiki).
Cray have acquired the Pathscale Compiler Suite from SiCortex who had recently ceased operating, please see here for more information. This means that it is once again a supported application on HECToR. If you have any questions on this please contact the helpdesk.
This is also referred to as dCSE support and funding is available to provide extended help with improving the performance of existing HECToR codes and developing high-performance algorithmic improvements for them. Support is also available to port new codes from other systems to HECToR. Awards to support proposed projects are assessed via an independent panel review which is held roughly every three months.
There are twenty four current projects that are underway. There is a new report now available on a recently completed project which has improved the performance of the Quantum Monte Carlo code CASINO. This dCSE work was performed by an HPC expert from the NAG CSE team. Performance gains show a speedup of up to 100% for certain large scale models. The completed report is available here.
Work currently being carried out for the "Massive Remote Batch Visualizer" dCSE project was presented at the recent vizNET workshop in Loughborough by George Leaver of the University of Manchester. He discussed his experiences in porting the AVS/Express visualisation software to HECToR, as well as mentioning other visualisation packages, such as ParaView, which are available on the machine.
Further information on the dCSE support service can be found here. The next application deadline is the 4th January 2010. Priority will be given to projects that propose specialist support to address any improvements for the phase 2A/B multi-core transitions and also those which can justify a reasonable AU saving impact for the wider HECToR community.
Applicants are advised to contact email@example.com with a brief description of their proposed work well before the application deadline. All applicants for the current round will be informed of the outcome of their proposals early February 2010. NAG staff are available to visit institutions to talk about this service. If you are interested in a visit please contact us at firstname.lastname@example.org