The HECToR Service is now closed and has been superceded by ARCHER.

Welcome to HECToRNews 6, November 2009



This is the sixth Newsletter for HECToR users from the Computational Science and Engineering support (CSE) team of NAG Ltd. The HECToR newsletter intends to keep users updated with useful information on the national supercomputing service, for the previous issues please see here.

In this issue we have information on HECToR related training courses, details of the forthcoming phase 2B hardware upgrade, technical reports on code optimizations for multi-core, a new streamlined method of access to significant HECToR resources, general points regarding the HECToR programming environment and information on the distributed CSE support service.

Training Course Timetable

The current schedule is below:

These training courses run by NAG Ltd. are provided free of charge to HECToR users and UK academics whose work is covered by the remit of one of the participating research councils (EPSRC, NERC and BBSRC).

For more information on HECToR training, including the most up to date schedule, please see here, or contact [Email address deleted]

HECToR Phase2B Upgrade Summary

The plan for the Phase2B upgrade was confirmed mid September. The rationale behind the upgrade path as discussed at the Town Meeting in April are available online.

The Phase2B upgrade will take place in two stages:

  • A 20 cabinet 'Baker' system will be installed, with an early user service available in March 2010. The system will comprise 44,544 cores (464 blades x 8 chips x 12 cores), delivering an estimated peak performance of 338 Tflops. Approximately 30 cabinets of XT4 will also be retained. The XT4 will provide the main service at this stage.
  • This will be followed by an upgrade to the current Seastar 2+ interconnect by replacing it with the newer Gemini interconnect by the end of 2010.

The combination of Baker+Gemini will then provide the main service. Exact details of the upgrade will be confirmed nearer the time. The transition to 12 core is an exciting prospect and if a user suspects that their code may need performance investigation prior to the upgrade, then they are encouraged to seek the help of the NAG CSE team by contacting the HECToR helpdesk. Help and advice can then be given on how the code might be able to get full benefit from the hardware upgrade.

New Face for the HECToR website

The HECToR Website has recently been simplified to convey a high level public view of the service.

This new view includes:

  • example case studies from users
  • an introduction to the service
  • guidance notes for applying for access

The original site with all the detailed information for users is now in a separate 'User Site' area. This is also accessible by the 'User Site' links on the new home page. You can also access this directly here. Note - all other bookmarks you have will still work. As always your comments on the website are welcome. Please contact the HECToR helpdesk if there is any additional content you would like to see on the site.

Technical Reports

There are two new technical reports written by the NAG CSE team which are now available:

  • optimization of a direct numerical simulation code for turbulence and combustion SoFTaR
  • quad core benchmarking and optimisation for CASTEP

Both reports are the result of researchers contacting the NAG CSE team and requesting to have their code's performance investigated. Site visits were arranged to facilitate in depth discussion with the research staffs involved. The CSE team were then able to provide more in depth support. Performance bottlenecks were identified and new communication routines were implemented improving the code performance.

In particular, the shared memory optimizations for multi-core which were implemented in CASTEP are well understood by the CSE team and they are widely applicable to other codes so if you think this very effective optimization may be beneficial to your code please contact the [Email address deleted]. Please also note that the improved shared memory version of CASTEP, (CASTEP 4.4-shm-xt) is available on HECToR now. See the Applications section below for further details on how to use this module.

The NAG CSE team have also produced a Fortran compiler performance comparison review page. This provides a comparison of different compiler performance for a collection of Fortran codes currently being used on HECToR. The codes are from different scientific areas and use a selection of computational algorithms which are common to most high performance numerical codes.

Access to HECToR

HECToR Direct Access, Call for applicants

In response to the 2009 Gateway review of the HECToR project, EPSRC are piloting a new access class for the HECToR service called 'Direct Access' (Class 1B) where users can request significant amounts ( greater than 1,000,000 AUs) of computing resource only for a period of up to six months.

The Direct Access mechanism will provide successful applicants with discretionary access to the HECToR system in order to provide resource to cover activities such as:

  • Bridging access between grant applications,
  • Resource to trial application developments at scale,
  • rovide preliminary results in aid of grant applications via Responsive Mode or to a Call for Proposals;

There is no deadline for submission, however: the proposals will be assessed by a Resource Allocation Panel (RAP) which will meet every four months (February 2010, June 2010 and October 2010), the deadline for applications being submitted to the Panel will be three weeks before the Panel meeting.

How To Apply

You should complete a Technical Assessment Form. The technical assessment form should then be submitted to the helpdesk. The completed technical assessment will normally be returned to the PI within two weeks of submission, although if the reviewer has concerns about the project, it may take extra time to address these with the PI. Applicants must submit their request for a technical assessment at least two weeks before they intend to submit their application to EPSRC.

If the technical assessment endorses the proposal, the applicant is required to email a completed RAP Application Form and the completed technical assessment directly to EPSRC. Applications should not be submitted through Je-S.

Proposals will not be subject to postal peer review; instead they will be assessed by the Resource Allocation Panel (RAP) who will meet every four months. The HECToR RAP inbox will be checked weekly and the submitted proposals will be considered at the next RAP meeting provided the accompanying documentation is complete and correct.

Closing date RAP meeting Deadline for start of access
12th January 2010 2nd February 2010 2nd March 2010
27th May 2010 17th June 2010 17th July 2010
21st September 2010 12th October 2010 12th November 2010

Applicants will be notified of the decision within one week of the Panel meeting. Successful applicants should then contact to confirm the start date of their access. The start date must be within one month of the panel date and resources must be used within six months of the start date.

These details and all the necessary forms are available at:

Increase in Computing Resource for Class 2 Access

Since the start of HECToR new users wishing to explore the capability and usage of the service are allowed to apply for Class2 project access. This can be either with a Class 2A application for access without requested distributed (dCSE) support or Class 2B access which accompanies the submission of a proposal for dCSE support. Under this mechanism up to 200,000 AU can now be requested for Class 2A access and up to 400,000 AU for Class 2B access. This is double the previous amount which was allowed for Class 2 projects. For further details please see here

Suspension of Charging on the X2

In February, charging was suspended on the X2 for an initial trial period of 6 months. It is confirmed that charging will remain suspended on the X2 through to the end of 2009. The situation will then be revisited.

Any new or current HECToR PIs wishing to the use the X2 should submit a Class 2 technical assessment form to the Helpdesk. This form should clearly state the project details. Please note that it is not necessary to calculate an AU requirement and this open access is not restricted to EPSRC users. All new X2 HECToR users will be required to attend a specific X2 training course

For more information on the training course in using the X2 please see the X2 course description.

Usage of the X2 will continue to be tracked, but you will not be charged. Users are encouraged to take this opportunity to explore ways of employing the vector capability in their research. Application forms are available online. If you have any questions, please do not hesitate to contact the HECToR helpdesk.

New Capability Incentive Scheme on HECToR

A Capability Incentive Scheme is now in operation on HECToR. The principal purpose of the Research Councils in funding the HECToR service is to provide a system with capabilities (both in hardware and user software support) which are unmatched by computing services at the University or department level. HECToR is a high-end supercomputing platform, and as such the Capability Incentive Scheme is targeted at large-scale simulations which can effectively utilise a significant fraction of the machine.

The Capability Incentive Scheme is an encouragement to users to broaden their computational science and to exploit the capabilities of the service. Under the scheme, jobs will be discounted at three levels depending on how well they scale. The three levels of incentives are:

Level Min Number of Cores AU Discount
Bronze 1024 5%
Silver 2048 15%
Gold 4096 30%

For example, a job running in a "par:n512c4" queue (2048 cores) would receive a 15% discount, regardless of job duration.

Users of HPCx may already be familiar with a similar scheme which requires the use of a token. In the case of HECToR Capability Incentives, you do not need to apply to receive the discount, nor do you need to specify a token in your batch script.

ALL jobs on HECToR using more than 1024 cores will now be automatically discounted at the appropriate level. Users wishing assistance in scaling their codes should contact the HECToR helpdesk.

HECToR Distributed CSE Support Technical Meeting

The first dCSE technical meeting took place at the Holiday Inn Hotel, Oxford on 23-24 September. This two day event was an opportunity for those involved with existing dCSE projects to:

  • Focus on Computational Science and Engineering;
  • Share experiences;
  • Identify common issues and solutions;
  • Strengthen HPC in the UK.

The slides from presentations given at the meeting are available here. The attendees were all involved with past or present dCSE projects. Further technical meetings are planned and will also include contributions from the wider HECToR user community.

Programming Environment issues

Compiling for quad core

Please remember when compiling your code for quad core you should load the module xtpe-quadcore. This will add the appropriate compiler flags for targeting quad core to the ftn and cc wrappers. It will also link in the quad core enabled version of libsci. These instructions along with the full explanation of what the quad core upgrade entails are available here.

However, the following message will appear

W-A-R-N-I-N-G **** xtpe-quadcore module is removed in the next release. Unload xtpe-quadcore and use xtpe-barcelona, xtpe-istanbul or xtpe-shanghai. To unload, use this commandline: module unload xtpe-quadcore

Currently, module load xtpe-quadcore will invoke the xtpe-barcelona module. The message can therefore be avoided by using module load xtpe-barcelona instead of module load xtpe-quadcore.

HECToR Archive Solution

The ability to archive data is critical for many groups on HECToR. The development of the HECToR archive solution is progressing well, and the archive solution should be in place for acceptance testing early November.

Users of HPCx will be familiar with the solution. There will be a simple command line interface for users, as well as a scheduled nightly archive from pre-defined directories. Both onsite and offsite archive storage will be available.


CASTEP 5.0 / CASTEP 4.4-shm-xt

The latest version of CASTEP (5.0) has now been installed on HECToR (for the XT system).You can select this version by using:

module add castep/5.0/xt

There is also a newly installed version of CASTEP 4.4 with shared memory enhancements implemented by the NAG HECToR CSE team which improves the scaling and performance of the code, for more details please see here. You can select this version with:

module add castep/4.4/shm-xt

More information on running CASTEP jobs (along with sample job submission scripts) is available on the HECToR User Wiki. (You will need your HECToR SAFE login details to access the Wiki).

To gain access to the CASTEP code on HECToR you must have a valid UKCP licence, available from Once you have a valid licence please contact the HECToR helpdesk to enable access to the CASTEP code for your account.

VASP 5.2

VASP 5.2 is now available on HECToR.

To gain access to the VASP 5.2 code on HECToR you must have a valid VASP 5 licence. Note - existing users of VASP on HECToR will need to apply for a separate license in order to use VASP 5. Once you have a valid VASP 5 licence please contact the HECToR helpdesk.

There are two binaries: vasp5 (default) and vasp5_gamma. You can select this version using

module load vasp5

More information on running VASP jobs (along with sample job submission scripts) is available on the HECToR User Wiki. (You will need your HECToR SAFE login details to access the Wiki).

Pathscale Compiler

Cray have acquired the Pathscale Compiler Suite from SiCortex who had recently ceased operating, please see here for more information. This means that it is once again a supported application on HECToR. If you have any questions on this please contact the helpdesk.

Distributed Support

This is also referred to as dCSE support and funding is available to provide extended help with improving the performance of existing HECToR codes and developing high-performance algorithmic improvements for them. Support is also available to port new codes from other systems to HECToR. Awards to support proposed projects are assessed via an independent panel review which is held roughly every three months.

There are twenty four current projects that are underway. There is a new report now available on a recently completed project which has improved the performance of the Quantum Monte Carlo code CASINO. This dCSE work was performed by an HPC expert from the NAG CSE team. Performance gains show a speedup of up to 100% for certain large scale models. The completed report is available here.

Work currently being carried out for the "Massive Remote Batch Visualizer" dCSE project was presented at the recent vizNET workshop in Loughborough by George Leaver of the University of Manchester. He discussed his experiences in porting the AVS/Express visualisation software to HECToR, as well as mentioning other visualisation packages, such as ParaView, which are available on the machine.

Further information on the dCSE support service can be found here. The next application deadline is the 4th January 2010. Priority will be given to projects that propose specialist support to address any improvements for the phase 2A/B multi-core transitions and also those which can justify a reasonable AU saving impact for the wider HECToR community.

Applicants are advised to contact [Email address deleted] with a brief description of their proposed work well before the application deadline. All applicants for the current round will be informed of the outcome of their proposals early February 2010. NAG staff are available to visit institutions to talk about this service. If you are interested in a visit please contact us at [Email address deleted]