The HECToR Service is now closed and has been superceded by ARCHER.

Improved Global High Resolution Chemistry-Aerosol Modelling for Climate and Air Quality

TOMCAT is a global atmospheric chemical transport model (CTM) which hosts the GLOMAP aerosol and microphysics module. TOMCAT-GLOMAP simulates past and future changes in global atmospheric chemistry and aerosol, including potential impacts on air quality, climate and ecosystems with a high-complexity high realism approach. It implements evolving state-of-the-art scientific understanding of each controlling process and has become one of the most sophisticated models of its type in the world.

TOMCAT (and related SLIMCAT) have been used in the UK since the 1990s. GLOMAP development and use on national HPC facilities began in 2001, two successful dCSE projects have already been completed to help improve the performance of the code on HECToR, please see here and here. Recently, model development funded by the NERC National Centre for Atmospheric Science (NCAS) has begun to make TOMCAT-GLOMAP available as a core community model. The codes operate in distributed memory mode with an option to activate shared memory parallelism.

The aim of this Distributed Computational Science and Engineering (dCSE) project is to improve TOMCAT-GLOMAP performance on HECToR for high resolution global simulations and thus remove the restriction of moderate spatial resolutions (2.8 degree latitude x 2.8 degree longitude) over just a few years. This will be achieved by refactoring the code which will facilitate a new approach to the parallel I/O to be developed. The concern is that the file access pattern will affect the performance due to the increase in data volume being read-from and written-to disk storage. The following work will be performed:

  • Detailed analysis of the high resolution model with the detailed aerosol-chemistry routines. This is large task which will involve different numbers and combinations of MPI/OMP cores, domain decompositions, compilers etc. The outcome will be a recommendation for performing high resolution simulations while achieving best scalability.
  • Refactored I/O sections of TOMCAT-GLOMAP. This will develop the I/O sections of the code to use a new parallel read structure and enable high resolution simulations to be performed. This will also involve rewriting all input files to the model so that they have a standard format.

The overall outcome of this work may be summarised as follows.

  • The raw performance of the code has been shown to be the best it can get, with one routine dominating the calculation phase in using up to 50% of the time per iteration (without file interaction).
  • The areas of weakness lay in the file access patterns. Analysis showed that although the actual cost of writing and reading data is small, the supporting features of data collection prior to writing and data distribution post reading are not efficient.
  • Four functions were chosen for improvement and two of these demonstrated dramatic improvements in performance.
  • The function which reports the altitude profiles of a calculated field added an overhead of as much as 40 seconds in its original form. The revised version is less than 0.05 seconds overhead on the iteration of 0.5 seconds. This has been confirmed for higher processor counts and the overhead does not vary much from 10% of the iteration time. Already the new method has been re-used within a function that deals with satellite data. In that situation there are 15 samples a day at differing longitude and latitude, so a low-cost algorithm is important.
  • The function which calculates zonal means was also indentified. This has the same characteristic as the function described above and the revised version of that has provided similar benefits. The improvements to the function have made the feature less expensive and thus a researcher can consider increasing the number of times that they ask for this data to be logged. A time series plot of such data could reveal trends that might have been missed when using a lower frequency output.
  • The third and fourth functions are not currently widely used, so did not result in any net performance gain. However, from a software engineering point of view the modified design is such that they should be re-used wherever there are functions that deal with the distribution of data after a read from file and the collection of data from other MPI tasks before writing to disk. There are several places in the code where this can be done.
  • In all four replacement algorithms the underlying serial file access method has been retained for compatibility and thus makes them immediately available for use in production work.
  • The developments are now in use on HECToR within the NCAS group. Although the developments were performed for TOMCAT, the two functions (gather and a scatter) are also used by many functions in GLOMAP.
  • This work was presented at the Cray User Group Meeting in May 2013.

Please see PDF or HTML for a report which summarises this project.